JP7527580B1 - Program, method, information processing device, and system - Google Patents

Abstract

[Problem] Debugging processing using a trained model is customized for each client while reducing the effort. [Solution] A program to be executed by a computer having a processor and a memory. The program causes the processor to execute the steps of acquiring test knowledge related to a client, adjusting, based on the acquired test knowledge, processing using one of a plurality of trained models used when creating test cases based on specifications, the trained models being trained to output predetermined elements that constitute test cases in response to input of data related to the specifications, and providing the processing using the adjusted trained model as processing for the client to which the test knowledge was provided. [Selected Figure] Figure 3

Description

Application of Article 30, Paragraph 2 of the Patent Act Announced at the 89 Matsuri 2023 held on August 9, 2023 (https://topics.shiftinc.jp/89matsuri2023/)

This disclosure relates to a program, a method, an information processing device, and a system.

Software is functionally tested before launch to ensure its quality. Tests are designed based on the contents of the software specifications to ensure the desired quality within the development schedule and budget set by the software development company.

Patent document 1 describes a technology that makes it easy to perform software testing.

JP 2022-129520 A

The test support system described in Patent Document 1 includes a model storage unit that stores a learning model trained using screen definition information of a system as input data and at least one of test cases, test programs, and test data as teacher data, a screen definition input unit that accepts input of the screen definition information of the system to be tested, a generation unit that provides the screen definition information to the learning model to generate at least one of the test cases, the test programs, and the test data, and an output unit that outputs the generated test cases, the test programs, and at least one of the test data.

However, in Patent Document 1, "a learning model is used in which the screen definition information of the system is used as input data and at least one of test cases, test programs, and test data is used as training data," and creating a trained model for each client is time-consuming.

The objective of this disclosure is to customize debugging processes using trained models for each client while reducing the amount of work required.

A program to be executed by a computer having a processor and a memory. The program causes the processor to execute the steps of acquiring test knowledge related to a customer, adjusting, based on the acquired test knowledge, a process using one of a plurality of trained models used when creating test cases based on specifications, the trained models being trained to output predetermined elements that constitute test cases in response to input of data related to the specifications, and providing the process using the adjusted trained model as a process for the customer who provided the test knowledge.

According to this disclosure, debugging processes using trained models can be customized for each client while reducing the amount of work required.

1 is a block diagram showing an example of the overall configuration of a system 1. FIG. 2 is a block diagram illustrating an example of the configuration of a terminal device 10 illustrated in FIG. 1. FIG. 2 is a diagram illustrating an example of a functional configuration of a server 20. FIG. 2 is a diagram showing the data structure of a user information table 2021. FIG. 2 shows the data structure of a specification table 2022. FIG. 23 is a diagram showing the data structure of a prompt table 2023. 13 is a diagram showing the data structure of a test design document table 20241. FIG. FIG. 23 is a diagram showing the data structure of a pattern table 20242. FIG. 2 is a diagram showing the data structure of a test viewpoint table 20243. FIG. 2 is a schematic diagram illustrating the operation of the server 20. 2 is a diagram for explaining the operations of the terminal device 10 and the server 20 when the server 20 assists in creating a test design document. FIG. 13 is a schematic diagram showing an example of a test design document creation screen displayed on a display 141. FIG. 13 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. 13 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. 13 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. 13 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. A flowchart showing an example of the operation of the server 20 when adjusting processing using a trained model. FIG. 13 is a conceptual diagram for explaining parameter tuning by the adjustment module 2035. FIG. 13 is a conceptual diagram for explaining parameter tuning by the adjustment module 2035. FIG. 13 is a diagram showing another example of the functional configuration of the server 20 when the RAG is adopted. FIG. 13 is a block diagram showing another example of the overall configuration of the system 1. FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90.

Below, an embodiment of the present disclosure will be described with reference to the drawings. In the following description, identical parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

<Summary>
The system according to this embodiment uses a trained model in each step of creating a test case based on a specification. The system acquires test knowledge related to the customer. The system adjusts, based on the acquired test knowledge, a process using a trained model that has been trained to output predetermined elements that constitute a test case, out of multiple trained models used when creating a test case based on a specification. The system provides the adjusted process as a process for the customer that provided the test knowledge.

Specifically, for example, in this embodiment, in the step of identifying test subjects, the system uses the trained model to output a test category. In the step of selecting test viewpoints, the system uses the trained model to output test viewpoints and check items. In the step of determining the specific content of the test, the system uses the trained model to output a test pattern. In the step of determining the specific content of the test, the system uses the trained model to output an expected value of the test. For example, a large language model (LLM: Large Language Models) may be used as the trained model.

Based on the acquired test knowledge, the system may, for example, adjust processing using a trained model that outputs a test category. Based on the acquired test knowledge, the system may also adjust processing using a trained model that outputs a test perspective (and check items). Based on the acquired test knowledge, the system may, for example, adjust processing using a trained model that outputs a test pattern (factors/levels). Based on the acquired test knowledge, the system may also adjust processing using a trained model that outputs an expected value.

Note that in this embodiment, the test section may be expressed by other expressions. For example, the test section may use expressions that represent the target of the test, such as test target items or test target functions. The test perspective may be expressed by other expressions. For example, the test perspective may use expressions that represent the content of the test, such as the viewpoint (way of thinking) on which the test is based. The check items may be expressed by other expressions. For example, the check items may use expressions that represent information for identifying the test to be performed.

<1 Overall system configuration>
Fig. 1 is a block diagram showing an example of the overall configuration of a system 1. The system 1 shown in Fig. 1 includes, for example, a terminal device 10, a server 20, and an LLM system 30. The terminal device 10, the server 20, and the LLM system 30 are communicatively connected via, for example, a network 80.

In FIG. 1, an example is shown in which the system 1 includes two terminal devices 10, but the number of terminal devices 10 included in the system 1 is not limited to two. The number of terminal devices 10 included in the system 1 may be one, or three or more.

In FIG. 1, an example is shown in which the system 1 includes one LLM system 30, but the number of LLM systems 30 included in the system 1 is not limited to one. The number of LLM systems 30 included in the system 1 may be two or more.

In this embodiment, a collection of multiple devices may be one server. The method of allocating multiple functions required to realize the server 20 according to this embodiment to one or more pieces of hardware can be appropriately determined in consideration of the processing capacity of each piece of hardware and/or the specifications required for the server 20. Specifically, for example, the server having the function of creating test cases based on the specifications and the server on which the trained model is built may be different devices.

The terminal device 10 shown in FIG. 1 is an information processing device used by a user who uses the service provided by the server 20. For example, the terminal device 10 is an information processing device operated by a test designer who creates a test design document using the test design document creation support provided by the server 20. The terminal device 10 is realized, for example, by a stationary PC (Personal Computer), a laptop PC, a head-mounted display, or the like. The terminal device 10 may also be a portable computer such as a smartphone or a tablet terminal.

The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, storage 16, and a processor 19. The input device 13 is a device for receiving input operations from a user (e.g., a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to the user (a display, a speaker, etc.).

The server 20 is an information processing device that provides a service that supports the creation of test design documents. For example, the server 20 uses multiple trained models to support the user in creating test design documents.

The server 20 is, for example, an information processing device realized by a computer connected to the network 80. As shown in FIG. 1, the server 20 includes a communication IF 22, an input/output IF 23, a memory 25, a storage 26, and a processor 29. The input/output IF 23 functions as an input device for accepting input operations from a user, and as an interface with an output device for outputting information to the user.

The LLM system 30 is a system in which a large-scale artificial intelligence model (LLM) is constructed for use in the field of natural language processing (NLP), for example. By studying large amounts of text data (web pages, books, articles, etc.), the LLM can understand patterns of language used by humans and effectively perform natural language generation (NLG) tasks.

LLMs are used in many NLP tasks, such as generating responses to specific questions, automatic sentence generation, text summarization, translation, sentiment analysis, etc. They can also be used in a variety of applications, such as education, entertainment, customer service, and product development. Examples of LLMs include the following:
・GPT-4 (registered trademark) (OpenAI, Inc.)
PaLM2 (registered trademark) (Google Inc.)
・StableLM (StableAI)
・Llama2 (Meta)

The LLM system 30 inputs test-related information and prompts output from the server 20 to the LLM, and causes the LLM to output answers based on the input text data and prompts. The LLM system 30 transmits the answers output from the LLM to the server 20.

Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

<1.1 Configuration of terminal device>
Fig. 2 is a block diagram showing an example of the configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected to each other, for example, by a bus or the like.

The communication unit 120 performs processes such as modulation and demodulation processes for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on signals generated by the control unit 190 and transmits the signals to the outside (e.g., server 20). The communication unit 120 performs reception processing on signals received from the outside and outputs the signals to the control unit 190.

The input device 13 is a device for inputting instructions or information by the user operating the terminal device 10. The input device 13 is realized, for example, by a touch-sensitive device 131 or the like where instructions are input by touching the operation surface. When the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, keyboard, mouse, or the like. The input device 13 converts instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may include, for example, a receiving port that receives electrical signals input from an external input device.

The output device 14 is a device for presenting information to a user who operates the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

The audio processing unit 17 performs, for example, digital-analog conversion processing of an audio signal. The audio processing unit 17 converts the signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

The camera 160 is a device that receives light using a light receiving element and outputs the light as a photographic signal.

The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10 equipped with the GPS module is detected based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

The storage unit 180 is realized, for example, by the memory 15 and the storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

User information 181 includes, for example, information about the user who uses the terminal device 10. Information about the user includes, for example, the user's name, age, address, date of birth, contact information, etc.

The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 performs the functions of an operation reception unit 191, a transmission/reception unit 192, and a presentation control unit 193 by operating according to the program.

The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131, etc.

The operation reception unit 191 also receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 17. The operation reception unit 191 acquires instructions from the user, for example, by analyzing the received voice signal and extracting a specific noun.

The transmission/reception unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmission/reception unit 192 transmits information input by the user or instructions from the user to the server 20. The transmission/reception unit 192 also receives information provided by the server 20.

The presentation control unit 193 controls the output device 14 to present the information provided by the server 20 to the user. Specifically, for example, the presentation control unit 194 causes the information transmitted from the server 20 to be displayed on the display 141. The presentation control unit 194 also causes the information transmitted from the server 20 to be output from the speaker 172.

<1.2 Functional configuration of the server>
Fig. 3 is a diagram showing an example of the functional configuration of the server 20. As shown in Fig. 3, the server 20 fulfills the functions of a communication unit 201, a storage unit 202, and a control unit 203.

The communication unit 201 performs processing for the server 20 to communicate with external devices.

The storage unit 202 includes, for example, a user information table 2021, a specification table 2022, a prompt table 2023, a test information table 2024, a client test information table 2027, and multiple trained models. The multiple trained models include, for example, trained models used in at least one of the steps when designing a test.

Specifically, for example, the multiple trained models include a first trained model 2025 that is used when selecting a test perspective. The multiple trained models also include a second trained model 2026 that is used when determining the specific content of a test. The multiple trained models also include a first trained model for customer 20251 that is a first trained model constructed for a specific customer. In this embodiment, for example, when identifying test subjects and when determining the specific content of a test, the LLM constructed in the LLM system 30 is used.

Note that the information stored in the memory unit 202 is not limited to these.

The user information table 2021 is a table that stores information about users. Details will be described later.

The specification table 2022 is a table that stores information related to specifications. Details will be described later. In this embodiment, the specification is data that describes the specifications for the functions of the test target. The test designer creates a test design document, for example, while referring to the specification. The specification includes, for example, an area in which detailed specifications of the function are stored, and an area in which message definitions are stored. If the specification is created using a spreadsheet software, these areas are realized, for example, on different sheets. The detailed specifications of the function include, for example, the name of the specification, object type, requirements, and control.

The prompt table 2023 is a table that stores information about prompts (instructions). Details will be described later.

The test information table 2024 is a table that stores information accumulated in various past debugging processes. For example, the test information table 2024 is a table that stores test design documents created in various past debugging processes and elements decomposed from the test design documents. In other words, the test information table 2024 is a table for generalizing accumulated test-related information and sharing it as shared knowledge. The elements decomposed from the test design documents represent, for example, information for specifying necessary tests and information in which specific test cases are set. In the test information table 2024, for example, identifiers representing attributes are assigned to the stored test design documents and elements. Multiple types of attributes can be set, for example, types such as business type/industry, system type, and process type. By using identifiers representing attributes, it is possible to classify test design documents or elements according to the attributes.

In this embodiment, the test design document includes, for example, meta information, information about the tests to be performed, and a pattern table. The meta information includes, for example, the title of the test design document, project name, product name, management number, version, creation date, creator, last update date, updater, and other information. Note that the information included in the meta information is not limited to these. Any of this information may be absent, or other information may be included.

Information about the tests to be performed plays the role of, for example, a so-called basic design document that specifies the broad framework of the tests. Information about the tests includes, for example, "test target items/target functions." "Test target items/target functions" represent, for example, the items/functions that are the subject of the tests. "Test target items/target functions" may also be referred to as test categories. Test categories may include sub-concepts of major functions (category 1), medium functions (category 2), and minor functions (category 3).

Information about a test includes, for example, a test perspective. A test perspective represents, for example, a viewpoint (way of thinking) on which a test is based. A test perspective is associated with a test section. The sequence of test perspectives, for example, specifies the order of tests. A test perspective is associated with a check item. A check item represents information for identifying the test to be performed.

The information about the test includes, for example, an expected value. The expected value represents, for example, a behavior or state that is expected to occur when checking. Note that the information about the test is not limited to this. Any of this information may be absent, or other information may be included.

The pattern table specifies, for example, the contents to be confirmed, the conditions for the behavior manifested by the confirmation process, and specific test cases that combine the confirmation contents and conditions as patterns. In the pattern table, for example, at least one pattern is set according to the function to be inspected. In the pattern table, the contents to be confirmed are specified, for example, as factors and levels. Factors represent, for example, factors that can affect changes in the state of a computer. Levels represent, for example, specific elements of factors. In the pattern table, the conditions for the behavior manifested by the confirmation process are specified, for example, by conditions stored as expected values. In the pattern table, specific test cases are specified, for example, using a matrix associated with factors and levels.

From the test design document, for example, test perspectives are broken down into information for specifying the necessary tests. Also, from the test design document, for example, patterns are broken down into information for which specific test cases are set.

The client test information table 2027 is a table that stores information accumulated in the debugging process performed by the customer. For example, the client test information table 2027 is a table that stores a test design document created in the debugging process performed by the customer, elements decomposed from this test design document, and test results. In other words, the client test information table 2027 is a table for accumulating information about tests performed by the customer as the customer's test knowledge. The elements decomposed from the test design document represent, for example, information for specifying the required tests and information in which specific test cases are set. The data stored in the client test information table 2027 may have the same data structure as the test information table 2024, or may have a data structure specific to the customer.

The client test information table 2027 is stored, for example, in a storage area (customer-dedicated environment) set up for the corresponding customer. In other words, the client test information table 2027 is stored in a storage area different from the test information table 2024. In other words, information regarding tests performed by the customer is stored in a table (database) different from the test information table 2024.

The client test information table 2027 stores, for example, information provided by a customer and accumulated in debugging processes previously performed by the customer. The client test information table 2027 also stores information accumulated by debugging processes based on a test design document created using the services provided by the control unit 203. The client test information table 2027 may store only information output by processing using the first trained model for customer 20251 out of the information output by the debugging processes of the customer. In other words, at least a portion of the information output by the debugging processes of the customer is not stored in the test information table 2024, but is stored only in the client test information table 2027.

The first trained model 2025 is a model for outputting a test perspective estimated to be suitable based on a test section and at least one check item associated with the test perspective. Specifically, the first trained model 2025 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has been subjected to additional learning. In the additional learning, the correct answer for each task is taught, that is, fine tuning is performed, so that the model is specialized for creating a test design document. Specifically, for example, the first trained model 2025 is trained to output a test perspective estimated to be suitable for the input test section and at least one check item associated with the test perspective. At this time, the learning data for training the first trained model 2025 is, for example, a test section stored in the test information table 2024 as input data, and a test perspective and check item stored in the test information table 2024 corresponding to the input test section as correct answer output data. The information entered as input data may be major functions (category 1), medium functions (category 2), and minor functions (category 3) as sub-concepts contained in the test category.

Context tuning may also be performed on the first trained model 2025. In context tuning, for example, example answers and answer formats are included in the instruction text so that information is output from the first trained model 2025 in a format desired by the user.

The first trained model 2025 may be retrained at a specified time based on the test category, test perspective, and confirmation items contained in the test design document newly added to the test information table 2024.

The first trained model 2025 may be trained to output a test perspective estimated to be suitable for an input test category and at least one check item associated with the test perspective, as well as to output an index value representing the suitability of the output test perspective. The first trained model 2025 may also be trained to output a test perspective and check items, as well as to output an index value representing the suitability of the output test perspective and check item. Depending on the test design document, different test perspectives and check items may be set even for the same test category. The index value is output, for example, by training the first trained model 2025 with learning data that has variation in the test perspectives and check items associated with the test category.

The first trained model 2025 may be trained to output a test perspective estimated to be suitable for an input test section and at least one check item associated with the test perspective, and to output an index value representing the risk of a malfunction occurring in the output test perspective. The first trained model 2025 may also be trained to output a test perspective and check item, and to output an index value representing the risk of a malfunction occurring in the output test perspective and check item. For example, in the test information table 2024, test results such as a malfunction occurring are stored for each test perspective, etc. The learning data for training the first trained model 2025 is, for example, the test section stored in the test information table 2024 as input data, and the test perspective, check item, and test result stored in the test information table 2024 corresponding to the input test section are set as correct output data. As a result, an index value representing a malfunction risk is output together with the test perspective and check item.

The first trained model for customer 20251 is a first trained model constructed for a customer. The first trained model for customer 20251 may be the same model as the first trained model 2025, or may be adjusted for the customer by the adjustment module 2035. The first trained model for customer 20251 is stored, for example, in a memory area set for the corresponding customer. For example, when a customer for whom a dedicated trained model has been constructed uses a service, the control unit 203 uses the first trained model for customer 20251 instead of the first trained model 2025.

The second trained model 2026 is a model for outputting expected values that are estimated to be appropriate based on the specification data, test section, test viewpoint, check item, and test pattern. Specifically, the second trained model 2026 is a model that has been subjected to additional learning on a large-scale artificial intelligence model used in the field of natural language processing (NLP). In the additional learning, the correct answer for each task is taught, that is, fine tuning is performed, so that the model is specialized for creating a test design document. Specifically, for example, the second trained model 2026 is trained to output expected values that are estimated to be appropriate for the input specification data, test section, test viewpoint, check item, and test pattern. At this time, the learning data for training the second trained model 2026 is, for example, the test section, test viewpoint, check item, and test pattern stored in the test information table 2024 as input data, and the expected value stored in the test information table 2024 corresponding to the input information is used as the correct output data. In addition, depending on the test viewpoint and check item, a pattern table may not exist. In such a case, the second trained model 2026 is trained to output expected values that are estimated to be appropriate for, for example, the input specification data, test category, test viewpoint, and check item. At this time, the learning data for training the second trained model 2026 is, for example, the test category, test viewpoint, and check item stored in the test information table 2024 as input data, and the expected value stored in the test information table 2024 that corresponds to the input information is set as the correct output data.

Context tuning may also be performed on the second trained model 2026. In context tuning, for example, example answers and answer formats are included in the instruction text so that information is output from the second trained model 2026 in a format desired by the user.

When the test category, test perspective, check items, and test patterns are used as input data and the expected value is used as correct output data, information that can identify the company, etc. that conducted the test may be masked. The second trained model 2026 may be retrained at any time based on the test category, test perspective, check items, test patterns, and expected value contained in a newly added test design document.

The additional learning may also include instruction tuning and reinforcement learning from human feedback (RLHF). In instruction tuning, for example, a data set consisting of various instructions and their corresponding desirable responses, and a data set consisting of various instructions and their corresponding undesirable responses, etc., are used as learning data, making it possible to output information that is in line with the test engineer's intentions.

The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions contained in the program. The program includes an application such as a web browser application. The program includes a programming language such as JavaScript (registered trademark) that is executed on a web browser application stored in the terminal device 10. The control unit 203 operates according to the program to perform functions shown as a reception control module 2031, a transmission control module 2032, a creation assistance module 2033, a presentation control module 2034, and an adjustment module 2035.

The reception control module 2031 controls the process in which the server 20 receives signals from external devices according to a communication protocol.

The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices according to a communication protocol.

The creation assistance module 2033 assists the user in creating a test design document. Specifically, for example, the creation assistance module 2033 uses a trained model at each step in test design to suggest effective options for test design to the user.

Specifically, for example, the creation support module 2033 uses the LLM system 30 when identifying test targets. For example, the creation support module 2033 transmits to the LLM system 30 the specification data selected by the user and an instruction statement for outputting a test division suitable for the specification data. The LLM system 30 inputs the specification data and the instruction statement transmitted from the server 20 to the LLM, and causes the LLM to output a test division suitable for the specification data. The test division output from the LLM may be one or more. The test division output from the LLM may be assigned division 1, division 2, or division 3 as a sub-concept of the test division.

The creation assistance module 2033 issues an instruction statement for outputting a test section in response to, for example, a user pressing a predetermined button. The creation assistance module 2033 may also receive text information from the user regarding an instruction for outputting a test section, and use the received text information as the instruction statement. The instruction statement may include content for explaining the structure of the specification data. The instruction statement may also include content for specifying the output format of the test section. The creation assistance module 2033 may convert the specification data into data in a predetermined format and transmit it to the LLM system 30. The predetermined format includes, for example, the Markdown format.

For example, the creation assistance module 2033 uses the first trained model 2025 when selecting a test perspective. For example, the creation assistance module 2033 inputs a test section related to a test design document and an instruction for outputting a test perspective and check items corresponding to the test section to the first trained model 2025, and causes the first trained model 2025 to output the test perspective and check items corresponding to the input test section. The test perspectives output from the first trained model 2025 may be one or more. The check items associated with the test perspectives output from the first trained model 2025 may be one or more.

When a test perspective, a check item, and an index value for the test perspective are output from the first trained model 2025, the creation assistance module 2033 may determine whether to output the test perspective and the check item to a later stage based on the index value. For example, when the index value of a test perspective output from the first trained model 2025 is less than a preset threshold, the creation assistance module 2033 may not output the test perspective and the check item associated with the test perspective to a later stage.

When there are multiple types of index values, such as compatibility and defect risk, test perspectives and check items may be output when either one of the index values exceeds a threshold, or test perspectives and check items may be output when both index values exceed a threshold. Alternatively, one index value may be given priority.

For example, the creation assistance module 2033 uses the first trained model for customer 20251 when selecting a test perspective for a specific customer. For example, the creation assistance module 2033 inputs a test category related to a test design document and an instruction for outputting a test perspective and check items corresponding to the test category to the first trained model for customer 20251, and causes the first trained model for customer 20251 to output the test perspective and check items corresponding to the input test category. The test perspectives output from the first trained model for customer 20251 may be one or more. The check items associated with the test perspectives output from the first trained model for customer 20251 may be one or more.

When a test perspective, a check item, and an index value for the test perspective are output from the first trained model for customer 20251, the creation assistance module 2033 may determine whether to output the test perspective and the check item to a later stage based on the index value. For example, when the index value of a test perspective output from the first trained model for customer 20251 is less than a preset threshold value, the creation assistance module 2033 may not output the test perspective and the check item associated with the test perspective to a later stage.

The creation assistance module 2033 issues an instruction statement for outputting a test perspective and check items, for example, in response to a user pressing a specific button. The creation assistance module 2033 may also receive text information from the user regarding an instruction for outputting a test perspective and check items, and use the received text information as the instruction statement. The instruction statement may include content for explaining the structure of the test section. The instruction statement may also include content for specifying the output format of the test perspective and check items.

In addition, the creation support module 2033 uses the LLM system 30 when determining the specific content of the test. For example, the creation support module 2033 transmits to the LLM system 30 the specification data, test category, test viewpoint, check item, and instructions for outputting appropriate factors and levels based on this information selected by the user. The LLM system 30 inputs the specification data, test category, test viewpoint, check item, and instructions transmitted from the server 20 to the LLM, and causes the LLM to output factors and levels appropriate for the specification data, test category, test viewpoint, and check item selected by the user. The factors and levels output from the LLM may be one or multiple.

The creation support module 2033 issues an instruction statement for outputting factors and levels in response to, for example, pressing a specific button by the user. The creation support module 2033 may also receive text information from the user regarding instructions for outputting factors and levels, and use the received text information as the instruction statement. The instruction statement may include content for explaining what each piece of information to be input is. The instruction statement may also include content for specifying the output format of the factors and levels. The creation support module 2033 may convert the specification data into data in a specific format and transmit it to the LLM system 30. The specific format includes, for example, Markdown format.

In addition, for example, when generating a test pattern, the creation assistance module 2033 uses rules for generating patterns from factors and levels that are pre-stored in the storage unit 202. For example, the creation assistance module 2033 applies rules to factors and levels to generate a pattern.

For example, the creation assistance module 2033 uses the second trained model 2026 when generating expected values. For example, the creation assistance module 2033 inputs the specification data, test category, test viewpoint, check item, and test pattern selected by the user, as well as an instruction statement for outputting an expected value appropriate for this information, to the second trained model 2026, and causes the second trained model 2026 to output an expected value appropriate for the input information.

Also, for example, when a pattern table is not associated with a test perspective and a check item, the creation assistance module 2033 inputs the specification data, test category, test perspective, and check item selected by the user, as well as an instruction statement for outputting an expected value appropriate for this information, into the second trained model 2026, and causes the second trained model 2026 to output an expected value appropriate for the input information.

The creation assistance module 2033 issues an instruction statement for outputting the expected value in response to, for example, a user pressing a specific button. The creation assistance module 2033 may also receive text information from the user regarding an instruction for outputting the expected value, and use the received text information as the instruction statement. The instruction statement may include content for explaining what each piece of information to be input is. The instruction statement may also include content for specifying the output format of the expected value.

The presentation control module 2034 controls the process of presenting information to the user. For example, the presentation control module 2034 presents information output from the trained model to the user. Specifically, for example, the presentation control module 2034 presents the test category output from the LLM system 30 to the user. The presentation control module 2034 also presents the test perspective and check items output from the first trained model 2025 to the user. The presentation control module 2034 also presents the test perspective and check items output from the customer-use first trained model 20251 to the user. The presentation control module 2034 also presents the factors and levels output from the LLM system 30 to the user. The presentation control module 2034 also presents to the user a pattern generated by applying a predetermined rule to the factors and levels. The presentation control module 2034 also presents to the user an expected value output from the second trained model 2026.

The adjustment module 2035 adjusts the trained model and/or the creation support module 2033 for the customer. Specifically, for example, when a test perspective, confirmation items, and an index value for the test perspective are output from the first trained model for customer 20251, the adjustment module 2035 adjusts a threshold for the index value output from the first trained model for customer 20251. For example, the adjustment module 2035 acquires attributes to which the customer belongs. The attributes include, for example, business type/industry, system type, and process type, and are stored in the client test information table 2027. The adjustment module 2035 adjusts a threshold in the creation support module 2033 based on, for example, the acquired attributes.

For example, the adjustment module 2035 sets a low suitability threshold for test perspectives and check items that are likely to be adopted for a specified attribute. Also, for example, the adjustment module 2035 sets a low suitability threshold for test perspectives and check items that are unlikely to be adopted for a specified attribute. Also, for example, the adjustment module 2035 sets a low defect risk threshold for test perspectives and check items that are likely to detect defects for a specified attribute. Also, for example, the adjustment module 2035 sets a high defect risk threshold for test perspectives and check items that are unlikely to detect defects for a specified attribute.

The adjustment module 2035 may also set the threshold value by analyzing the information stored in the client test information table 2027. For example, the adjustment module 2035 may extract test perspectives and check items that are frequently adopted by analyzing the information stored in the client test information table 2027, and set the threshold value based on the extracted results. For example, the adjustment module 2035 may also extract test perspectives and check items that frequently detect defects by analyzing the information stored in the client test information table 2027, and set the threshold value based on the extracted results. As a result, test perspectives and check items that are likely to be adopted by customers are output from the first trained model for customer 20251.

The adjustment module 2035 may also adjust the first trained model 20251 for the customer by performing additional learning on the first trained model 2025. Specifically, for example, the adjustment module 2035 uses the test category stored in the client test information table 2027 as input data, and the test perspective and confirmation items stored in the client test information table 2027 corresponding to the input test category as correct answer output data to fine-tune the first trained model 20251 for the customer. The information input as input data may be large function (category 1), medium function (category 2), and small function (category 3) as sub-concepts included in the test category. As a result, the first trained model 20251 for the customer outputs customer-specific test perspectives and confirmation items in addition to the test perspectives and confirmation items based on the test information table 2024, or in place of the test perspectives and confirmation items based on the test information table 2024, for the input test category. Furthermore, when adding a customer-specific test perspective, several dozen actual test cases linked to the test perspective to be added are required in training the first customer-use trained model 20251.

The adjustment module 2035 may also adjust the learned model and the creation support module 2033 for the customer by adjusting the searcher. That is, the adjustment module 2035 may adjust the learned model and the creation support module 2033 for the customer by adopting a Retrieval-based Language Model (RAG). Specifically, for example, the adjustment module 2035 constructs a searcher in the memory unit 202 that searches the client test information table 2027 for information similar to the input information. The searcher may be constructed in a memory area for the customer. For example, the searcher calculates the similarity between the input test division vector and the test division vector stored in the client test information table 2027, and obtains the test division with the large similarity as a search result. For example, the creation support module 2033 inputs the test division related to the test design document into the searcher. The searcher obtains the test division similar to the input test division from the client test information table 2027. The creation assistance module 2033 inputs the test category related to the test design document, the test category acquired from the client test information table 2027, and an instruction statement for outputting the test viewpoint and check items corresponding to the test category to the first trained model for customer 20251. The first trained model for customer 20251 outputs the test viewpoint and check items corresponding to the input test category.

<2 Data Structure>
4 to 9 are diagrams showing the data structures of tables stored in the server 20. Note that Fig. 4 to 9 are merely examples and do not exclude data that is not listed. In addition, data that is listed in the same table may be stored in separate storage areas in the storage unit 202.

Figure 4 is a diagram showing the data structure of the user information table 2021. The user information table 2021 shown in Figure 4 is a table that uses the user ID as a key and has columns for name, age, sex, date of birth, and contact information.

The user ID is an item that stores an identifier for uniquely identifying a user. The name is an item that stores the name of the user. The age is an item that stores the age of the user. The gender is an item that stores the gender of the user. The date of birth is an item that stores the date of birth of the user. The contact information is an item that stores the contact information (e.g., a telephone number, an email address, etc.) of the terminal device 10 owned by the user.

Figure 5 is a diagram showing the data structure of the specification table 2022. The specification table 2022 shown in Figure 5 is a table having columns such as client ID, project ID, creation date and time, specification name, specification data, and latest update, with the specification ID as a key. Note that the information stored in Figure 5 is not limited to these. For example, any of these may not be stored, or further information may be stored.

The specification ID is an item that stores an identifier for uniquely identifying the specification. The client ID is an item that stores an identifier for uniquely identifying the client who requires a test based on the specification. The project ID is an item that stores an identifier for uniquely identifying the project to which the function to be tested based on the specification belongs. The creation date and time is an item that stores the date and time when the specification was created. The specification name is an item that stores the name of the specification. The specification name may be given by extracting a part of the content of the specification, or may be given by the creator of the specification. The specification data includes data on the detailed specifications of the function, data on message definitions, and the like. The item "specification data" may store reference information (path) for a specification data file located in another location. The last update is an item that stores the date and time when the specification data was last updated.

Figure 6 is a diagram showing the data structure of the prompt table 2023. The prompt table 2023 shown in Figure 6 is a table having columns such as prompt data, with a prompt ID as a key. Note that the information stored in Figure 6 is not limited to this. For example, further information may be stored. For example, there may be a column that stores an ID that identifies an industry. This allows the creation assistance module 2033 to select instructions appropriate for each industry, for example.

The prompt ID is an item that stores an identifier for uniquely identifying a prompt. The prompt data is an item that stores data related to the prompt, which is an instruction text to be input to the LLM. Data related to the prompt is, for example, text information related to the prompt. In the text information related to the prompt, for example, the content that identifies the information is left blank, and the specified content is inserted when the prompt is used.

The text information related to the prompt is, for example, an instruction sentence for outputting a test section based on the specification data, and includes, for example, the following content:
"The input document is a specification {...}. Please output the test category for testing the functions related to this specification in the format...."

The text information related to the prompt is, for example, an instruction for outputting a test viewpoint and confirmation items corresponding to the test category, and includes, for example, the following content.
"The input document is in the test category {...}. Please output the test category and the corresponding test viewpoints and confirmation items in the format of...."

Furthermore, the text information related to the prompt is, for example, an instruction for outputting factors and levels based on specification data, test category, test viewpoint, and confirmation item, and includes, for example, the following content:
"The documents you have entered are the specification {...}, the test category {...} for which you would like to test this specification, the test viewpoint {...}, and the confirmation item {...}. Please output the factors and levels for the test based on this information in the format of ...."

Furthermore, the text information related to the prompt is, for example, an instruction for outputting an expected value based on specification data, test category, test viewpoint, check item, and test pattern, and includes, for example, the following content:
"The documents you have entered are the specification {...}, the test category {...} for which you would like to test this specification, the test viewpoint {...}, the confirmation items {...}, and the test pattern {...}. Please output the expected value for conducting the test based on this information in the format of ...."

The "prompt data" item may store reference information (path) to a prompt data file located elsewhere.

Figures 7 to 9 are diagrams showing the data structure of the test information table 2024. The test information table 2024 includes, for example, a test design document table 20241, a pattern table 20242, and a test viewpoint table 20243. The pattern table 20242 and the test viewpoint table 20243 are tables that store elements decomposed from the test design document table 20241.

Figure 7 is a diagram showing the data structure of the test design document table 20241. Each record of the test design document table 20241 shown in Figure 7 has columns for file name, tag, designer ID, project ID, service ID, attribute, link, and latest update. The file name is an item that stores the name of the test design document. The tag is an item that stores information that represents the characteristics of the test. Information in the "tag" item is stored, for example, based on information set for the project related to the specification. In addition, information in the "tag" item may be set arbitrarily by the user.

The designer ID is an item that stores an identifier for uniquely identifying the person who created the test design document. The project ID is an item that stores an identifier for uniquely identifying the project for which the test design document was created. The service ID is an item that stores an identifier for uniquely identifying the service in which the test target is used. The attribute is an item that stores information indicating the type to which the test design document belongs. Multiple types of attributes can be set in the "attribute" item, and for example, one or more attributes can be set from the business type/industry, system type, process type, etc. The information in the "attribute" item is set, for example, based on the information stored in the "tag" item. In addition, the information in the "attribute" item may be set arbitrarily by the user.

A link is an item that stores specification data associated with a test design document. The specification data itself may be associated with the item "link", but the relevant portion of the specification data may also be associated. For example, if the specification data is a spreadsheet program, the relevant portion of the specification data may be stored as an expression such as a slide, row, or cell. Last updated is an item that stores the most recent date and time the test design document was updated.

Figure 8 is a diagram showing the data structure of pattern table 20242. Each record of pattern table 20242 shown in Figure 8 has columns for pattern name, pattern number, tag, file name, project ID, service ID, attribute, and latest update. "Pattern name" is an item that stores the name of the pattern. "Pattern number" is an item that stores a unique number assigned to the pattern. "Tag" is an item that stores information that is attached as a tag to the test design document that is the source of decomposition.

The file name is an item that stores the name of the test design document that was the source of the decomposition. The project ID is an item that stores an identifier for uniquely identifying the project in which the test design document that was the source of the decomposition was created. The service ID is an item that stores an identifier for uniquely identifying the service in which the test target is used. The attribute is an item that stores information indicating the type to which the test design document that was the source of the decomposition belongs. The last update is an item that stores the date and time when the pattern was most recently updated.

Figure 9 is a diagram showing the data structure of the test perspective table 20243. Each record of the test perspective table 20243 shown in Figure 9 has columns for test perspective, check item, tag, category 1, category 2, attribute, and latest update. The test perspective is an item that stores the viewpoint of the test. The check item is an item that stores information for identifying the test to be performed. The tag is an item that stores information attached as a tag to the original test design document. Category 1 is an item that stores information about the first category attached to the test perspective. The information stored in the item "category 1" is set by the user when inputting the test perspective, for example, at the stage of creating the test design document. The information stored in the item "category 1" is set by the user, for example, from a plurality of items that are set in advance. The information stored in the item "category 1" is, for example, information related to the test category.

Classification 2 is an item that stores information about the second classification assigned to the test perspective. The information stored in the item "Classification 2" is set by the user when inputting the test perspective, for example, during the creation stage of the test design document. The information stored in the item "Classification 2" is set by the user, for example, by selecting from a plurality of items that are set in advance. The information stored in the item "Classification 2" is, for example, information related to the test section. The number of classifications stored in the test perspective table 20243 is not limited to two, and may be three or more. The attribute is an item that stores information indicating the type to which the test design document that is the source of the decomposition belongs. The last update is an item that stores the most recent date and time when the test perspective was updated.

<3 Operation>
The operation of the server 20 when assisting in the creation of a test design document will be described.

(Test design document creation support 1)
FIG. 10 is a schematic diagram showing the operation of the server 20. As shown in FIG.
FIG. 11 is a diagram for explaining the operations of the terminal device 10 and the server 20 when the server 20 assists in creating a test design document.

First, a user (test engineer) who wants to create a test design document accesses the server 20. The user logs in to a service that assists in the creation of a test design document, provided by the server 20, for example, by entering his or her user ID and password.

In step S11, the terminal device 10 accepts specification data designation from the user. Specifically, for example, the display 141 displays a list of specifications stored in the specification table 2022. The user selects a specification for which a test design document needs to be created from the specifications displayed in the list. After selecting a specification, the user inputs an instruction to the terminal device 10 to output a test section corresponding to the specification. The transmission/reception unit 192 transmits information related to the specified specification data and an instruction to output the test section to the server 20.

In step S12, the server 20 obtains an appropriate test division for the specification data based on the specified specification data. Specifically, for example, the creation support module 2033 reads an instruction statement for outputting a test division suitable for the specification data from the prompt table 2023. The creation support module 2033 outputs information about the specification data specified by the user and the read instruction statement to the LLM system 30. The LLM system 30 inputs the information about the specification data transmitted from the server 20 and the instruction statement to the LLM, and causes the LLM to output a test division suitable for the specification data. At this time, the LLM outputs, for example, text information indicating the items/functions to be tested. The LLM assigns, for example, hierarchical information such as "test division," "division 1," "division 2," "division 3," or any combination of these to the text information. The LLM system 30 transmits the information output from the LLM to the server 20.

The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test section transmitted from the LLM system 30. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the test section transmitted from the LLM system 30, to the terminal device 10.

In step S13, the terminal device 10 displays to the user a test design document creation screen including the test section sent from the LLM system 30.

Figure 12 is a schematic diagram showing an example of a test design document creation screen displayed on display 141. The creation screen shown in Figure 12 includes a display area 1411, an operation object 1412, a display area 1413, and an operation object 1414.

The display area 1411 is an area where a list of specification data is displayed. In FIG. 12, the presentation control unit 193 displays one piece of specification data in the display area 1411. In FIG. 12, the presentation control unit 193 indicates that the specification data has been specified by the user by placing a mark in the box to the left of the specification data. Note that when the test design document creation screen is created for a single piece of specification data, the display area 1411 does not necessarily have to be provided.

Operation object 1412 is an object for inputting an instruction to request a test section proposal.

The display area 1413 is an area where a list of test sections is displayed. In FIG. 12, the presentation control unit 193 displays text information in the display area 1413 based on the hierarchical information of the test section, section 1, section 2, and section 3.

The operation object 1414 is an object for inputting an instruction to request the proposal of a test perspective and confirmation items. The presentation control unit 193 displays the operation object 1414, for example, in an area associated with a specific test section. Specifically, for example, the presentation control unit 193 displays the operation object 1414 in association with a line in which text information with section 3, which indicates that the test section is the lowest level, is displayed.

In the creation screen shown in FIG. 12, the user specifies specification data in the display area 1411 and presses the operation object 1412. The terminal device 10 then transmits an instruction to the server 20 requesting a test section proposal. In response to the instruction, the creation support module 2033 causes the LLM system 30 to output a test section appropriate for the specification data specified by the user. The presentation control unit 193 displays text information related to the test section output for the specified specification data in the display area 1413.

In step S14, the terminal device 10 accepts the selection of the test subject. Specifically, for example, the display 141 displays a list of multiple test sections provided by the server 20. The user selects a test section for which the user wants to check the test perspective and check items from the test sections displayed in the list. The user inputs an instruction to the terminal device 10 to output the test perspective and check items corresponding to the selected test section. The transmission/reception unit 192 transmits information about the selected test section and an instruction to output the test perspective and check items to the server 20.

In step S15, the server 20 outputs appropriate test perspectives and check items for the test section based on the selected test section. Specifically, for example, the creation assistance module 2033 reads an instruction statement for outputting a test perspective and check items corresponding to the test section from the prompt table 2023. The creation assistance module 2033 inputs the selected test section and the read instruction statement to the first trained model 2025, and causes the first trained model 2025 to output test perspectives and check items appropriate for the test section. At this time, the first trained model 2025 outputs, for example, text information representing the test perspective and text information representing the check items.

The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test viewpoints and check items output from the first trained model 2025. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the test viewpoints and check items output from the first trained model 2025, to the terminal device 10.

In step S16, the terminal device 10 displays to the user a test design document creation screen that includes the output test perspectives and confirmation items.

Figure 13 is a schematic diagram showing another example of a test design document creation screen displayed on the display 141. The creation screen shown in Figure 13 includes an operation object 1414 and a display area 1415.

The display area 1415 is an area where the test perspectives and check items for the selected test section are displayed. In FIG. 13, the presentation control unit 193 displays the test perspectives and check items output for two test sections in the display area 1415. In FIG. 13, the presentation control unit 193 displays a test section for which one set of test perspectives and check items are proposed, and a test section for which two sets of test perspectives and check items are proposed.

In the creation screen shown in FIG. 12, the user presses the operation object 1414 corresponding to the test section for which the user desires a test perspective and check items. The terminal device 10 then transmits an instruction to the server 20 requesting a proposal of a test perspective and check items. In response to the instruction, the creation assistance module 2033 causes the first trained model 2025 to output a test perspective and check items appropriate for the test section selected by the user. The presentation control unit 193 displays text information related to the test perspective and check items output for the selected test section in the display area 1415.

In step S17, the terminal device 10 accepts the selection of a test perspective and a check item. Specifically, for example, the display 141 displays the test perspective and check items provided by the server 20. The user selects a test perspective and check item for which the user wants to check factors and levels from the displayed test perspectives and check items. The user inputs an instruction to the terminal device 10 to output factors and levels corresponding to the selected test perspective and check item. The transmission/reception unit 192 transmits information related to the selected test perspective and check item and an instruction to output the factors and levels to the server 20.

In step S18, the server 20 acquires factors and levels appropriate for the information based on the specification data specified by the user, the test division, test perspective, and check item selected by the user. Specifically, for example, the creation support module 2033 reads out an instruction statement for outputting factors and levels appropriate for the specification data, test division, test perspective, and check item from the prompt table 2023. The creation support module 2033 outputs the specification data specified by the user, information on the test division, test perspective, and check item selected by the user, and the read instruction statement to the LLM system 30. The LLM system 30 inputs the specification data, information on the test division, test perspective, and check item transmitted from the server 20, and the instruction statement to the LLM, and causes the LLM to output factors and levels appropriate for the selected test perspective and check item. At this time, the LLM outputs, for example, text information representing the factors and text information representing the levels. The LLM may output a remark for the level. The LLM system 30 transmits the information output from the LLM to the server 20.

The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the factors and levels transmitted from the LLM system 30. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the factors and levels transmitted from the LLM system 30, to the terminal device 10.

In step S19, the terminal device 10 displays to the user a test design document creation screen including the factors and levels transmitted from the LLM system 30.

Figure 14 is a schematic diagram showing another example of a test design document creation screen displayed on the display 141. The creation screen shown in Figure 14 includes an operation object 1416 and a display area 1417.

Operation object 1416 is an object for inputting instructions requesting suggestions for factors and levels.

The display area 1417 is an area where the factors and levels for the selected test perspective and check item are displayed. In FIG. 14, the presentation control unit 193 displays the factors and levels for the test perspective: "display/hide" and the check item: "text is displayed according to the pattern" in the display area 1417.

In the creation screen shown in FIG. 14, the user selects the test perspective and check item for which the factors and levels are desired, and presses the operation object 1416. The terminal device 10 then transmits an instruction to the server 20 requesting a proposal of factors and levels. In response to the instruction, the creation support module 2033 causes the LLM system 30 to output factors and levels appropriate for the specification data specified by the user, the test category selected by the user, the test perspective, and the check item. The presentation control unit 193 displays text information related to the factors and levels output from the LLM system 30 in the display area 1417.

In step S110, the terminal device 10 accepts an instruction to create a test pattern. Specifically, for example, the display 141 displays a screen for creating a pattern table based on factors and levels provided by the server 20 in response to an instruction from the user. The user selects a rule for creating a test pattern that is displayed on the creation screen. The rule for creating a test pattern is, for example, "two-factor coverage," "full coverage," "constraint + full coverage," etc. When a rule is selected, the transmission/reception unit 192 transmits information on the factors and levels, information on the selected rule, and an instruction to create a test pattern to the server 20.

In step S111, the server 20 creates a test pattern from the factors and levels based on the selected rule. Specifically, for example, the creation support module 2033 applies the selected rule to the factors and levels to create a test pattern. At this time, the creation support module 2033 expands predetermined marks in accordance with the selected rule in a matrix with multiple columns, with factors and levels as rows. The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the created test pattern. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the created test pattern, to the terminal device 10.

In step S112, the terminal device 10 displays a test design document creation screen including the test pattern to the user.

Figure 15 is a schematic diagram showing another example of a test design document creation screen displayed on the display 141. The creation screen shown in Figure 15 includes an operation object 1418 and a display area 1419.

Operation object 1418 is an object for selecting a rule for creating a test pattern. In FIG. 15, "bifactor coverage" is selected as the rule.

The display area 1419 is an area where the created test pattern is displayed. In FIG. 15, the presentation control unit 193 displays a matrix containing "circles" as a test pattern in the display area 1419.

In the creation screen shown in FIG. 15, the user presses an operation object 1418 for selecting a rule for creating a test pattern. The terminal device 10 then transmits an instruction to the server 20 requesting the creation of a test pattern. The creation assistance module 2033 creates a test pattern in accordance with the instruction and the selected rule. The presentation control unit 193 displays the created test pattern in the display area 1419.

In step S113, the terminal device 10 accepts an instruction to create an expected value. Specifically, for example, the display 141 displays factors and levels provided by the server 20, and a test pattern created by the server 20. The user inputs an instruction to the terminal device 10 to output an expected value corresponding to a case defined by the test pattern. The transmission/reception unit 192 transmits information about the case defined by the test pattern and an instruction to output the expected value to the server 20.

In step S114, the server 20 outputs an appropriate expected value for the selected case based on the specification data specified by the user, the test division, test viewpoint, check item, and case selected by the user. Specifically, for example, the creation support module 2033 reads an instruction statement for outputting an expected value appropriate for the specification data, test division, test viewpoint, check item, and test pattern from the prompt table 2023. The creation support module 2033 inputs the specification data specified by the user, the test division, test viewpoint, check item, and test pattern selected by the user, and the read instruction statement to the second trained model 2026, and causes the second trained model 2026 to output an expected value appropriate for the input information. At this time, the second trained model 2026 outputs, for example, text information related to the expected value.

The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the expected value output from the second trained model 2026. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the expected value output from the second trained model 2026, to the terminal device 10.

In step S115, the terminal device 10 displays to the user a test design document creation screen that includes the output expected values.

Figure 16 is a schematic diagram showing another example of a test design document creation screen displayed on the display 141. The creation screen shown in Figure 16 includes an operation object 14110, a display area 14111, and a display area 14112.

Operation object 14110 is an object for inputting an instruction to request a proposal for expected values.

Display area 14111 is an area where the cases for which expected values are to be created are displayed. In FIG. 16, the presentation control unit 193 displays a matrix in which factors and levels are rows and cases are columns in the display area 14111 as the cases for which expected values are to be created. In FIG. 16, for example, the presentation control unit 193 displays a matrix of the factor: "product type" and level: "regular products only" as the first case of the test pattern, with the factor: "total product price" and level: "less than 2000 yen".

The display area 14112 is an area where the expected value and the case for which the expected value was created are displayed. In FIG. 16, the presentation control unit 193 displays a black circle representing the target for which the expected value is to be created, and the created expected value in the display area 14112. In FIG. 16, the presentation control unit 193 displays, for example, the expected value created for the first case, but expected values for the other cases may also be created in bulk.

In the creation screen shown in FIG. 16, the user presses the operation object 14110 in the display area 14112. The terminal device 10 then transmits an instruction to the server 20 requesting a proposal of an expected value. In response to the instruction, the creation assistance module 2033 causes the second trained model 2026 to output an expected value for a case defined by the test pattern. The presentation control unit 193 displays text information related to the output expected value in the display area 14112.

(Test design document creation support 2)
The following describes the operation when at least one of a plurality of processes using a trained model when creating a test case based on a specification is adjusted for a customer. This operation will also be described with reference to FIG. 11 between the terminal device 10 and the server 20.

In step S15 of FIG. 11, the server 20 outputs appropriate test perspectives and check items for the test section based on the selected test section. Specifically, for example, the creation assistance module 2033 reads an instruction statement for outputting a test perspective and check items corresponding to the test section from the prompt table 2023. The creation assistance module 2033 inputs the selected test section and the read instruction statement into the first customer-use trained model 20251, and causes the first customer-use trained model 20251 to output test perspectives and check items appropriate for the test section. At this time, the first customer-use trained model 20251 outputs, for example, text information representing the test perspective and text information representing the check items.

The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test viewpoints and check items output from the first trained model for customer 20251. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the test viewpoints and check items output from the first trained model for customer 20251, to the terminal device 10.

In step S16, the terminal device 10 displays to the user a test design document creation screen including the output test perspectives and check items, for example as shown in FIG. 13.

(Adjusting processing using trained models)
17 is a flowchart showing an example of the operation of the server 20 when adjusting a process using a trained model. In the following, an example will be described in which a process using a first trained model for customer 20251 that outputs a test viewpoint (and a check item) is adjusted based on the acquired test knowledge.

First, the client test information table 2027 stores information provided by the customer that has been accumulated during debugging processes previously performed by the customer. The control unit 203 stores, for example, information provided by the customer in the client test information table 2027.

An administrator of a service that provides assistance in creating a test design document operates the server 20 and adjusts the process using the trained model for the client. The administrator selects, for example, an adjustment method. Examples of the adjustment methods are as follows. Note that the following methods may be adopted individually or in combination.
・Parameter tuning ・Fine tuning ・RAG

The method selected by the administrator is not limited to one. The administrator may select a combination of two or more of the above adjustments. The adjustment method may be a method other than the above, such as instruction tuning. When the administrator selects a tuning method, the adjustment module 2035 starts the adjustment.

In step S21, the adjustment module 2035 reads information from the client test information table 2027.

In step S22, the adjustment module 2035 adjusts the processing using the first trained model for customer 20251 for the customer. When parameter tuning is selected by the administrator, the adjustment module 2035 adjusts the threshold for the index value output from the first trained model for customer 20251 based on, for example, information read from the client test information table 2027.

Figures 18 and 19 are conceptual diagrams for explaining parameter tuning by the adjustment module 2035. The adjustment module 2035 adjusts the threshold for the index value output from the first trained model for customer 20251.

In addition, when fine tuning is selected by the administrator, the adjustment module 2035 trains the first trained model for the customer 20251, for example, using the test category stored in the client test information table 2027 as input data and the test perspectives and confirmation items stored in the client test information table 2027 that correspond to the input test category as correct answer output data.

In addition, when an RAG is selected by the administrator, the adjustment module 2035 constructs a searcher in the memory unit 202 that searches the client test information table 2027 for a test section similar to the input test section, for example.

Figure 20 is a diagram showing another example of the functional configuration of the server 20 when RAG is adopted. As shown in Figure 20, a searcher 2028 is constructed in the memory unit 202. The searcher 2028 is stored, for example, in a memory area set for the corresponding customer.

In step S23, the adjustment module 2035 provides processing using the first trained model 20251 for the customer in a memory area for the customer.

The adjustment module 2035 may suggest to the user an adjustment of the processing using the first trained model for customer 20251 at a predetermined timing. For example, the adjustment module 2035 suggests to the user an adjustment of the processing using the first trained model for customer 20251 at a predetermined period, for example, every six months. Also, for example, the adjustment module 2035 suggests to the user an adjustment of the processing using the first trained model for customer 20251 when a predetermined amount of data is accumulated in the client test information table 2027.

As described above, in the above embodiment, the adjustment module 2035 acquires test knowledge related to the customer. The adjustment module 2035 adjusts, based on the acquired test knowledge, a process using one of a plurality of trained models used when creating a test case based on a specification, the trained models having been trained to output predetermined elements that constitute a test case in response to input of data related to the specification. The adjustment module 2035 provides the process using the adjusted trained model as a process for the customer to which the test knowledge was provided. In this way, the adjustment module 2035 can customize the support for creating a test design document for the customer by adjusting a process using one of a plurality of trained models used when creating a test case.

Therefore, according to the program, method, information processing device, and system of this embodiment, the process using the trained model used in the debugging process can be customized for each client while reducing the effort.

In addition, in the above embodiment, the trained model is a large-scale language model. This enables the creation assistance module 2033 to provide the user with information that corresponds to the input information with a high degree of accuracy.

In the above embodiment, the trained model to be adjusted is trained model 2025 trained to output a test perspective in response to input of a test section related to the specification. Since a test perspective is information that represents the content of a test, it is likely to include content specific to a customer and content specific to a business type/industry. Therefore, when a process using the trained model is customized for a customer, the effect of the customization is likely to be reflected in the output. Therefore, by adjusting a process using a trained model that has been trained to output a test perspective in response to input of a test section related to the specification, it is possible to more effectively customize support for creating test design documents for customers.

In the above embodiment, the trained model is trained to output predetermined elements constituting a test case and index values related to the elements in response to input of data related to the specifications. The adjustment module 2035 adjusts the threshold value of the index value used as a condition for presenting the output element based on the acquired test knowledge. This enables the adjustment module 2035 to more easily adjust the processing using the trained model for the customer.

In addition, in the above embodiment, the trained model is trained to output predetermined elements constituting a test case and multiple types of index values related to the elements in response to input of data related to the specifications. The adjustment module 2035 adjusts the threshold for each type of index value based on the acquired test knowledge. This enables the adjustment module 2035 to adjust the processing using the trained model for the customer in a simple but highly accurate manner.

In the above embodiment, the adjustment module 2035 uses the acquired test knowledge as learning data to additionally train the trained model. This enables the adjustment module 2035 to adjust the trained model so that customer-specific test target items, test target functions, test contents, etc. are output from the trained model. The adjustment module 2035 also enables the trained model to adjust the trained model so that business/industry-specific test target items, test target functions, test contents, etc. are output from the trained model.

In the above embodiment, in processing using the trained model, the creation assistance module 2033 inputs data related to the specifications to a retriever 2028, and among the information stored in a predetermined database 2027, data similar to the input data related to the specifications is output from the retriever 2028 and input to the trained model. The adjustment module 2035 stores the acquired test knowledge in the database 2027. This enables the adjustment module 2035 to adjust processing using the trained model for the customer with high accuracy while reducing processing man-hours.

In addition, in the above embodiment, the adjustment module 2035 acquires the test results of the test cases created using multiple trained models as test knowledge. This enables the adjustment module 2035 to adjust the processing using the trained models based on the test results of the test cases created using the support for creating a test design document. At least a portion of the results of the debugging processing performed by the customer are not stored in the test information table 2024, but are stored in the client test information table 2027, making it possible to prevent information regarding the debugging processing performed by the customer from leaking.

<Modification>
In the above embodiment, a case where a test case is created by selecting information output by a trained model by a user has been described as an example. However, the information output by the trained model does not necessarily have to be adopted as is. For example, the user may modify the information output from the trained model. That is, the user may modify and adopt the test division output from the LLM system 30. The user may also add a new test division and adopt the added test division. The user may also modify and adopt the test perspective and check item output from the first trained model 2025. The user may also add a new test perspective and check item and adopt the added test perspective and check item. The user may also modify and adopt the factor/level output from the LLM system 30. The user may also add a new factor/level and adopt the added factor/level. The user may also modify the test pattern created based on the factor/level. The user may also add a new test pattern and adopt the added test pattern. The user may also modify and adopt the expected value output from the second trained model 2026. Also, the user may create a new expected value and adopt the created expected value.

In addition, in the above embodiment, a case is described in which information of a type expected to be input is input to a trained model, and the trained model is caused to output specified information. However, the information input to the trained model may be part of the information expected to be output. In other words, the user may be responsible for part of the creation of the specified elements, and the trained model may carry out the rest.

For example, when setting a test division, the user creates an upper-level test division, such as test division 1, and instructs the server 20 to output a lower-level test division. The creation support module 2033 outputs the specification data, the test division created by the user, and an instruction statement for outputting the lower-level test division to the LLM system 30. The LLM system 30 causes the LLM to output the lower-level test division and transmits the output information to the server 20.

For example, when setting test perspectives and check items, the user creates at least one test perspective and check item, and instructs the server 20 to output other test perspectives and check items. The creation assistance module 2033 outputs the test category, the test perspectives and check items created by the user, and an instruction statement for outputting the other test perspectives and check items to the first trained model 2025. The first trained model 2025 outputs the other test perspectives and check items.

For example, when setting factors and levels, the user creates at least one factor and level, and instructs the server 20 to output other factors and levels. The creation support module 2033 outputs to the LLM system 30 the specification data, test category, test viewpoint, check items, factors and levels created by the user, and instructions for outputting other appropriate factors and levels based on this information. The LLM system 30 causes the LLM to output the factors and levels, and transmits the output information to the server 20.

Also, for example, when setting expected values, the user creates at least one expected value and instructs the server 20 to output other expected values. The creation assistance module 2033 outputs to the second trained model 2026 the specification data, test category, test perspective, check items, test pattern, expected values created by the user, and instructions for outputting other expected values appropriate for this information. The second trained model 2026 outputs the other test perspectives and check items.

In the above embodiment, the LLM system 30 is configured to output the test division and the factors/levels. However, the test division and the factors/levels are not limited to being output from the LLM system 30, and may be output from a trained model that has been subjected to additional training on the LLM, such as the first trained model 2025 and the second trained model 2026. In the additional training, for example, the correct answer for each task is taught, so-called fine tuning is performed, to specialize in the creation of a test design document. Specifically, for example, by training a test division suitable for the specification as the correct answer, a test division suitable for the input specification data is output in response to input of a command (prompt) for outputting the specification data and a test division suitable for the specification data. Also, for example, by training a factor and level corresponding to the specification data, test division, test perspective, and check item as the correct answer, a factor and level suitable for the input information are output in response to input of a command for outputting the specification data, test division, test perspective, check item, and factors/levels corresponding to these information.

In addition, in the above embodiment, a case has been described in which various information is input to the trained model. However, the information input to the trained model is not necessarily limited to the above. For example, in the above embodiment, the check items are input to the trained model along with the test viewpoints. However, if the check items are not included in the training data, the check items do not necessarily have to be input to the trained model.

In the above embodiment, for example, the creation assistance module 2033 outputs test perspectives and check items that are plausibly set among the test perspectives and check items output from the first trained model 2025. However, the test perspectives and check items output from the first trained model 2025 are not limited to test perspectives and check items that are plausibly set. For example, the test information table 2024 stores the results of tests performed for the set test perspectives and check items. Then, as learning data for training the first trained model 2025, for example, the test category stored in the test information table 2024 is used as input data, and the test perspectives, check items, and information on defects as test results stored in the test information table 2024 corresponding to the input test category are used as correct output data. As a result, the first trained model 2025 outputs a test perspective that is estimated to be suitable for the input test category, at least one check item associated with the test perspective, and information on defects. This makes it possible to identify test perspectives and check items in which many defects are likely to occur, improving test efficiency. Note that the first trained model 2025 may preferentially output test perspectives and check items in which many defects are likely to occur from among test perspectives estimated to be suitable for an input test category and at least one check item associated with the test perspectives.

The first trained model 2025 may also be trained to output a test perspective that is estimated to be appropriate for an input test category, and at least one check item that is associated with the test perspective, and to output an index value that represents the defect occurrence rate for the output test perspective. The first trained model 2025 may also be trained to output a test perspective and check items, and to output an index value that represents the defect occurrence rate for the output test perspective and check item.

In the above embodiment, the user can select, modify, or add to the suggestions output by the trained model. The creation assistance module 2033 stores, for example, in the test information table 2024, items that have been modified or added by the user. The creation assistance module 2033 also stores, for example, in the test information table 2024, items that have been proposed by the trained model and selected by the user. When fine-tuning the trained model, the server 20 includes in the training data the history of the adoption of the item (whether it is a selected item, a modified item, or an added item). This allows the trained model to prioritize output of content created by the user, making it possible to output the information desired by the user with high accuracy.

In the above embodiment, a case has been described in which a test category is input to the first trained model 2025 when proposing test perspectives and check items. However, when specification data is included in the learning data, the specification data may be input to the first trained model 2025 in addition to the test category.

In the above embodiment, the server 20 has the prompt table 2023. However, the server 20 does not necessarily have to have the prompt table 2023. For example, the server 20 may have a predetermined collaboration module, and the collaboration module may include a prompt template. The collaboration module is, for example, an API (Application Programming Interface). Note that the collaboration module may not be provided within the server 20, but may be provided in a server different from the server 20.

Specifically, for example, in step S12, the creation assistance module 2033 outputs information about the specification data specified by the user to the linkage module. The linkage module creates an instruction statement by combining the information it has as a template with information about the specification data. The linkage module outputs the information about the specification data specified by the user and the created instruction statement to the LLM system 30.

Also, for example, in step S15, the creation assistance module 2033 outputs information about the test category selected by the user to the collaboration module. The collaboration module creates an instruction sentence by combining information held as a template with information about the test category. The collaboration module inputs information about the test category selected by the user and the created instruction sentence to the first trained model 2025, and causes the first trained model 2025 to output test perspectives and confirmation items appropriate for the test category.

Also, for example, in step S18, the creation support module 2033 outputs to the linkage module information related to the specification data specified by the user, and the test division, test perspective, and check items selected by the user. The linkage module creates an instruction statement by combining the information held as a template with information related to the specification data, test division, test perspective, and check items. The linkage module outputs to the LLM system 30 the specification data specified by the user, and information related to the test division, test perspective, and check items selected by the user, and the created instruction statement.

Also, for example, in step S114, the server 20 outputs an appropriate expected value for the selected case based on the specification data specified by the user, the test division, test perspective, check items, and case selected by the user. Specifically, for example, the creation support module 2033 outputs the case specified by the specification data specified by the user, the test division, test perspective, check items, and test pattern selected by the user to the linkage module. The linkage module creates an instruction statement by combining information held as a template with the specification data specified by the user, and the case specified by the test division, test perspective, check items, and test pattern selected by the user. The linkage module inputs the specification data specified by the user, the test division, test perspective, check items, and test pattern selected by the user, and the created instruction statement to the second trained model 2026, and causes the second trained model 2026 to output an expected value appropriate for the input information.

In addition, in the above embodiment, the LLM of the LLM system 30 outputs the test category and also outputs the factors and levels, but this is not limited to this. The LLM that outputs the test category and the LLM that outputs the factors and levels may be different LLMs.

In addition, in the above embodiment, a case where the multiple trained models are LLMs is described as an example. However, the multiple trained models according to this embodiment may be trained models that are not LLMs.

In addition, in the above embodiment, an example is described in which a process using the first trained model for customer 20251 that outputs test viewpoints (and confirmation items) is adjusted. However, the process to be adjusted is not limited to the process using the first trained model for customer 20251. For example, the adjustment module 2035 may customize a process using the second trained model 2026 for a customer.

Specifically, for example, when an expected value and an index value for the expected value are output from the second trained model 2026, the adjustment module 2035 adjusts the threshold for the index value output from the second trained model 2026 based on the information stored in the client test information table 2027.

The adjustment module 2035 also fine-tunes the second trained model 2026 using, for example, specification data, test category, test viewpoint, check items, and test pattern stored in the client test information table 2027 as input data, and the expected value stored in the client test information table 2027 corresponding to the input information as correct output data.

The adjustment module 2035 also employs, for example, RAG and constructs in the storage unit 202 a searcher that searches the client test information table 2027 for information similar to the input specification data, test division, test viewpoint, check item, and test pattern. The creation support module 2033 inputs, for example, the specification data, test division, test viewpoint, check item, and test pattern to the searcher. The searcher acquires information similar to the input information from the client test information table 2027. The creation support module 2033 inputs, to the second trained model 2026, the specification data, test division, test viewpoint, check item, and test pattern, the information acquired from the client test information table 2027, and an instruction statement for outputting an expected value corresponding to the input information. The second trained model 2026 outputs an expected value corresponding to the input information.

Also, for example, if the test section is output from a trained model that has undergone additional training in an LLM, the adjustment module 2035 may customize processing using the trained model for the customer.

Specifically, for example, when a test section and an index value for the test section are output from the trained model, the adjustment module 2035 adjusts the threshold for the index value output from the trained model based on the information stored in the client test information table 2027.

The adjustment module 2035 also fine-tunes the trained model, for example, using specification data stored in the client test information table 2027 as input data and a test category stored in the client test information table 2027 that corresponds to the input specification data as correct output data.

The adjustment module 2035 also employs, for example, RAG and constructs in the memory unit 202 a searcher that searches the client test information table 2027 for specification data similar to the input specification data. The creation assistance module 2033, for example, inputs specification data to the searcher. The searcher acquires specification data similar to the input specification data from the client test information table 2027. The creation assistance module 2033 inputs the specification data, the specification data acquired from the client test information table 2027, and an instruction statement for outputting a test category corresponding to the input specification data to the trained model. The trained model outputs a test category corresponding to the input information.

Also, for example, when factors and levels are output from a trained model that has undergone additional training in an LLM, the adjustment module 2035 may customize processing using the trained model for the customer.

Specifically, for example, when a factor/level and an index value for the factor/level are output from the trained model, the adjustment module 2035 adjusts the threshold for the index value output from the trained model based on the information stored in the client test information table 2027.

The adjustment module 2035 also fine-tunes the trained model using, for example, specification data, test category, test perspective, and confirmation items stored in the client test information table 2027 as input data, and factors and levels stored in the client test information table 2027 that correspond to the input information as correct output data.

The adjustment module 2035 also employs, for example, RAG and constructs in the storage unit 202 a searcher that searches the client test information table 2027 for information similar to the input specification data, test division, test viewpoint, and check item. The creation support module 2033 inputs, for example, the specification data, test division, test viewpoint, and check item to the searcher. The searcher acquires information similar to the input information from the client test information table 2027. The creation support module 2033 inputs, to the trained model, the specification data, test division, test viewpoint, and check item, the information acquired from the client test information table 2027, and an instruction statement for outputting appropriate factors and levels based on the input information. The trained model outputs factors and levels corresponding to the input information.

In the above embodiment, parameter tuning, fine tuning, and RAG have been described as adjustment methods for processing using a trained model. However, the adjustment methods are not limited to these. Although it requires more labor than parameter tuning, fine tuning, and RAG, a trained model for a customer may be created by new learning. Specifically, for example, the first trained model for customer 20251 is created by new learning. In this embodiment, since multiple trained models are used, even if the first trained model for customer 20251 is created by new learning, the labor costs can be significantly reduced compared to creating all trained models by new learning. Since the test perspective is information that represents the content of the test, it is easy for content specific to the customer and content specific to the business type/industry to appear. Therefore, by creating the first trained model for customer 20251 by new learning, it is possible to effectively customize the support for creating test design documents for the customer.

Customization of the trained model for a customer may be applied to one of the trained models used in multiple steps, or to multiple trained models. In this case, the method used to customize the trained model may be, for example, any one of parameter tuning, fine tuning, and RAG, or a combination of any of these. Furthermore, when multiple trained models are customized, the method used to customize each trained model may be different.

In the above embodiment, an example has been described in which the first trained model for customer 20251 and the client test information table 2027 are constructed in a customer-dedicated environment in the memory unit 202. However, the environment in which the first trained model for customer 20251 and the client test information table 2027 are constructed is not limited to within the server 20. The first trained model for customer 20251 and the client test information table 2027 may be stored on the customer's server.

Figure 21 is a block diagram showing another example of the overall configuration of system 1. In Figure 21, system 1 includes a terminal device 10, a server 20, a client server 40, and an LLM system 30. The terminal device 10, the server 20, the client server 40, and the LLM system 30 are connected for communication via, for example, a network 80.

The client server 40 functions as a memory unit 402. The memory unit 402 includes, for example, a client test information table 4027 and a first trained model for customer 40251.

In addition, in the above embodiment, the creation assistance module 2033 performs the support for creating the test design document, but this is not limited to being performed by the creation assistance module 2033. For example, if the client test information table 4027 and the first trained model for customer 40251 are stored in the client server 40, the function of the creation assistance module may be incorporated in the client server 40.

In addition, in the above embodiment, an example of a mode in which the creation assistance module 2033 is provided in the server 20 has been described, but the present invention is not limited to this mode, and the functions of the creation assistance module 2033 may be provided in the terminal device 10. In this case, for example, some or all of the information stored in the storage unit 202 may be stored in the terminal device 10.

<4. Basic hardware configuration of computer>
22 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 91, a main storage device 92, an auxiliary storage device 93, and a communication IF (interface) 99. These are electrically connected to each other by a bus.

The processor 91 is hardware for executing a set of instructions written in a program. The processor 91 is composed of an arithmetic unit, registers, peripheral circuits, etc.

The main memory device 92 is used to temporarily store programs and data processed by the programs. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

The auxiliary storage device 93 is a storage device for saving data and programs. For example, it is a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc.

The communication IF 99 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.
The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet through a predetermined access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network directly connected by a USB (Universal Serial Bus) cable, etc.

The computer 90 can be realized virtually by distributing all or part of each hardware configuration across multiple computers 90 and connecting them together via a network. In this way, the concept of computer 90 includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

<Basic Functional Configuration of Computer 90>
A description will be given of the functional configuration of a computer realized by the basic hardware configuration of a computer 90 shown in Fig. 22. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

The functional units of the computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 connected to each other via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

The control unit is realized by the processor 91 reading out various programs stored in the auxiliary storage device 93, expanding them in the main storage device 92, and executing processing according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

The storage unit is realized by a main storage device 92 and an auxiliary storage device 93. The storage unit stores data, various programs, and various databases. Furthermore, the processor 91 can secure a storage area corresponding to the storage unit in the main storage device 92 or the auxiliary storage device 93 in accordance with a program. Furthermore, the control unit can cause the processor 91 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

A database refers to a relational database, which is used to manage sets of data called tables, which are structured according to rows and columns, by relating them to each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, it is possible to set relationships between tables and associate them.
Usually, a column is set in each table as a key for uniquely identifying a record, but setting a key in the column is not essential. The control unit can cause the processor 91 to add, delete, or update records in a specific table stored in the storage unit according to various programs.

The communication unit is realized by the communication IF 99. The communication unit realizes the function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 91 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general purpose processors, application specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and/or combinations thereof, programmed to perform the functions described. Processors include transistors and other circuits and are considered to be circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in a memory.
In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed in this specification or any hardware known to be programmed to realize or perform the described functions.
If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and/or processor.

Although several embodiments of the present disclosure have been described above, these embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are considered to be within the scope of the invention and its equivalents as set forth in the claims, as well as within the scope and spirit of the invention.

<Additional Notes>
The matters described in the above embodiments will be supplemented below.
(Appendix 1)
A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the steps of: acquiring test knowledge related to a customer; adjusting, based on the acquired test knowledge, a process using one of a plurality of trained models used when creating test cases based on specifications, the trained models being trained to output specified elements that constitute a test case in response to input of data related to the specifications; and providing the process using the adjusted trained model as a process for the customer who provided the test knowledge.
(Appendix 2)
The program according to claim 1, wherein the trained model is a large-scale language model.
(Appendix 3)
In the adjustment step, the trained model to be adjusted is a trained model trained to output a test viewpoint in response to input of a test category related to the specification.
(Appendix 4)
A program described in any one of (Appendix 1) to (Appendix 3), in which in the adjustment step, the trained model is trained to output, in response to input of data related to a specification, predetermined elements constituting a test case and index values related to the elements, and the program adjusts a threshold value of the index value used as a condition for presenting the output element based on the acquired test knowledge.
(Appendix 5)
A program described in claim 4, wherein in the adjusting step, the trained model is trained to output multiple types of index values, and the threshold is adjusted for each type of index value.
(Appendix 6)
A program described in any one of (Appendix 1) to (Appendix 5), in which, in the adjusting step, the acquired test knowledge is used as learning data to additionally learn a trained model.
(Appendix 7)
A program described in any one of (Appendix 1) to (Appendix 6), in which, in a processing using a trained model, data relating to a specification is input into a searcher, and among information stored in a specified database, data similar to the input data relating to the specification is output from the searcher and input into the trained model, and in the adjustment step, the acquired test knowledge is stored in the database.
(Appendix 8)
A program described in any one of (Appendix 1) to (Appendix 7), in which, in the acquiring step, test results of test cases created using multiple trained models are acquired as test knowledge.
(Appendix 9)
A method executed by a computer having a processor and a memory, wherein the processor executes all of the steps performed in any of the inventions according to (Appendix 1) to (Appendix 8).
(Appendix 10)
An information processing device comprising a processor and a memory, wherein the processor executes all of the steps executed in any of the inventions relating to (Appendix 1) to (Appendix 8).
(Appendix 11)
A system comprising means for executing all the steps performed in any of the inventions according to (Appendix 1) to (Appendix 8).

Reference Signs List 1... System 10... Terminal device 120... Communication unit 13... Input device 131... Touch-sensitive device 14... Output device 15... Memory 16... Storage 19... Processor 20... Server 22... Communication IF
23... Input/output IF
25... memory 26... storage 29... processor 30... LLM system

Claims (11)

A program for causing a computer having a processor and a memory to execute the program, the program causing the processor to:
acquiring test knowledge relating to a customer;
A step of adjusting, based on the acquired test knowledge, a process using any one of a plurality of trained models used when creating a test case based on a specification, the trained models being trained to output predetermined elements constituting a test case in response to input of data related to the specification;
and providing a process using the adjusted trained model as a process for a customer who has provided the test knowledge. The program according to claim 1, wherein the trained model is a large-scale language model. The program according to claim 1, wherein the trained model to be adjusted in the adjustment step is a trained model trained to output a test viewpoint in response to an input of a test category related to the specification. The program according to claim 1, wherein in the adjustment step, the trained model is trained to output predetermined elements constituting a test case and index values related to the elements in response to input of data related to the specification, and the threshold value of the index value used as a condition for presenting the output element is adjusted based on the acquired test knowledge. The program according to claim 4, wherein in the adjustment step, the trained model is trained to output multiple types of index values, and the threshold is adjusted for each type of index value. The program according to claim 1, wherein in the adjustment step, the acquired test knowledge is used as training data to additionally train the trained model. In a process using the trained model, data related to the specification is input to a searcher, and data similar to the input data related to the specification among information stored in a predetermined database is output from the searcher and input to the trained model;
2. The program according to claim 1, wherein the acquired test knowledge is stored in the database in the adjusting step. The program according to claim 1, wherein in the acquiring step, test results of test cases created using multiple trained models are acquired as the test knowledge. A method executed by a computer having a processor and a memory, the processor executing all of the steps executed in the invention according to any one of claims 1 to 8. An information processing device comprising a processor and a memory, the processor executing all steps executed in the invention according to any one of claims 1 to 8. A system having means for executing all steps performed in any one of claims 1 to 8.