JP7388785B1 - Information processing device, method, program, and system - Google Patents

Abstract

[Problem] To provide technology for efficiently and objectively evaluating the quality of development work by engineers. [Solution] A program according to one aspect of the present disclosure allows a first engineer to add or add a computer to a local repository of an issue management service based on information stored in the issue management service when the engineer uses the issue management service. A method for obtaining the contents of a review request for requesting a review of a changed file and additional information related to the review request, an evaluation of the review request in model input information based on the contents of the review request and additional information related to the review request. By applying a trained model that performs inference for the purpose, it functions as a means for evaluating review requests and a means for outputting evaluation results of review requests. [Selection diagram] Figure 4

Description

The present disclosure relates to an information processing device, method, program, and system.

In an organization, it is important to evaluate the productivity and quality of work performed by its members.

Patent Document 1 describes a technical idea intended to evaluate a business from a qualitative perspective and accurately evaluate a business related to the business.

Japanese Patent Application Publication No. 2011-164715

In the technical idea of Patent Document 1, the user (the activity subject) needs to subjectively evaluate the activity. Therefore, in addition to the problem of forcing users to perform complicated tasks when implementing this technical concept, there is also the problem that even the same activity (task) can be evaluated differently depending on the subject (in other words, the evaluation is not objective). be.

The purpose of the present disclosure is to provide a technique for efficiently and objectively evaluating the quality of development work by engineers.

A program according to one aspect of the present disclosure is configured to control a computer from information stored in the issue management service by the engineer using the issue management service, to determine whether the first engineer has added or modified a file to a local repository of the issue management service. A means for acquiring the content of the review request for which a review is requested and additional information related to the review request, and inferences for evaluating the review request based on the model input information based on the content of the review request and additional information related to the review request. By applying a trained model that performs this, it functions as a means to evaluate review requests and a means to output evaluation results of review requests.

FIG. 1 is a block diagram showing the configuration of an information processing system according to the present embodiment. FIG. 2 is a block diagram showing the configuration of a client device according to the present embodiment. FIG. 2 is a block diagram showing the configuration of a server according to the present embodiment. FIG. 2 is an explanatory diagram of one aspect of the present embodiment. FIG. 3 is a diagram showing the data structure of a repository database according to the present embodiment. FIG. 2 is a diagram showing a data structure of a development activity database according to the present embodiment. It is a flowchart of information processing of this embodiment. FIG. 3 is a diagram illustrating an example of a screen displayed during information processing according to the present embodiment.

Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In addition, in the drawings for explaining the embodiments, the same components are generally designated by the same reference numerals, and repeated explanations thereof will be omitted.

In this specification, a team means a group composed of multiple members. In this specification, a team may refer to an engineer team that is a group composed of multiple engineers. Teams are responsible for developing specific products. A product is typically software, and may be, for example, a website for providing a web service such as SaaS (Software as a Service), or an application installed on a computer. Within the team, each engineer divides their roles by function, output, etc. to proceed with development. One or more teams may belong to an organization, such as a company, for example. Furthermore, the engineers in the team are not limited to employees employed by one specific company (organization), but may be employees of any company involved in a development project that is jointly developed by multiple companies. Furthermore, the engineers in the team may include temporary employees and engineers who are self-employed (so-called freelancers).

(1) Configuration of information processing system The configuration of the information processing system will be explained. FIG. 1 is a block diagram showing the configuration of an information processing system according to this embodiment.

As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30.
The client device 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW. Furthermore, the server 30 is connected to an external system 50 via a network NW.

The client device 10 is an example of an information processing device that sends a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer. The client device 10 can be used to check the evaluation results provided by the information processing system 1 in response to a review request. Here, a review request refers to an activity in which an engineer requests another person to review a file added or modified in a local repository, and corresponds to, for example, a pull request on GitHub.

Users of client device 10 may include, for example, at least one of the following:
・The engineer who made the review request to be evaluated ・The reviewer who was assigned the review request to be evaluated ・Other members or leaders of the team to which the above engineer or reviewer belongs ・The person responsible for the development of the product that the above engineer or reviewer is in charge of・The above engineer or reviewer's superior ・The person in charge of human resources in the organization to which the above engineer or reviewer belongs ・The executive officer in the organization to which the above engineer or reviewer belongs ・The person who views job information for the organization to which the above engineer or reviewer belongs ・Persons who are considering requesting work from or employing the above engineers or reviewers.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. Server 30 is, for example, a server computer. The server 30 provides evaluation results of the review request in response to a request from the client device 10.

The external system 50 provides information regarding development activities by engineers in response to requests from the server 30. External system 50 is typically a software development platform, such as GitHub (registered trademark). GitHub manages source code that is a product of development by engineers, reviews of source code, and the like.

Note that the external system 50 is not limited to GitHub, and may be another software development platform such as GitLab (registered trademark) or Bitbucket (registered trademark). Alternatively, the external system 50 may be an issue management tool (eg, Jira (registered trademark)) or a task management tool (eg, Trello (registered trademark), backlog (registered trademark), etc.) in software development. These software development platforms, issue management tools, or task management tools can also be collectively referred to as issue management services. In the following description, it is assumed that GitHub is used as the external system 50.

GitHub is a source code hosting service. GitHub has a repository (management means) for version control of source code, which is a product of development. GitHub manages a list of source code at a certain point in time. GitHub has various functions such as issue (creation), push, commit, pull request, and merge.

Issues is a function for managing projects and source code issues. Push is a function that uploads source code created or modified in an engineer's local environment (local repository) to a remote environment (remote repository). Commit is a function for registering changes to source code. A pull request is a function that notifies you that a push has been made and requests a review. Merge is a function that finalizes review results and integrates history. Here, source code is a type of programming language such as JavaScript (registered trademark) in addition to code (character string) written in a programming language such as Python or a markup language such as HTML or CSS. This is a concept that includes scripts (character strings) written in a certain scripting language.

A programming language is a language used for software development, and specifically includes JavaScript (registered trademark), Java (registered trademark), Scala, PHP, Ruby, Python, Go, C#, and the like. Also, a software framework is a software platform used for software development, specifically jQuery, React, Vue.js, Angular, Nuxt.js, Next.js, ReactNative, Spring Framework, Play Framework, Larav. el, Examples include CakePHP, RubyonRails, Django, Flask, TensorFlow, gin, Unity, and Express.

(1-1) Configuration of client device The configuration of the client device will be explained. FIG. 2 is a block diagram showing the configuration of the client device of this embodiment.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, and a communication interface 14. Client device 10 is connected to display 21 .

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of ROM (Read Only Memory), RAM (Random Access Memory), and storage (for example, flash memory or hard disk).

The programs include, for example, the following programs.
・OS (Operating System) program ・Application program that executes information processing (e.g. web browser)

The data includes, for example, the following data.
・Databases referenced in information processing ・Data obtained by executing information processing (that is, execution results of information processing)

The processor 12 is a computer that implements the functions of the client device 10 by activating a program stored in the storage device 11. The processor 12 is, for example, at least one of the following.
・CPU (Central Processing Unit)
・GPU (Graphic Processing Unit)
・ASIC (Application Specific Integrated Circuit)
・FPGA (Field Programmable Array)

The input/output interface 13 acquires information (for example, user instructions) from an input device connected to the client device 10 and outputs information (for example, an image signal) to an output device connected to the client device 10. configured.
The input device is, for example, a keyboard, pointing device, touch panel, or a combination thereof.
The output device is, for example, a display 21, a speaker, or a combination thereof.

Communication interface 14 is configured to control communication between client device 10 and an external device (eg, server 30).

The display 21 is configured to display images (still images or moving images). The display 21 is, for example, a liquid crystal display or an organic EL display.

(1-2) Server configuration The server configuration will be explained. FIG. 3 is a block diagram showing the configuration of the server of this embodiment.

As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of ROM, RAM, and storage (eg, flash memory or hard disk).

The programs include, for example, the following programs.
・OS program ・Application program that executes information processing

The data includes, for example, the following data.
・Databases referenced in information processing ・Execution results of information processing

The processor 32 is a computer that implements the functions of the server 30 by activating a program stored in the storage device 31. The processor 32 is, for example, at least one of the following.
・CPU
・GPU
・ASIC
・FPGA

The input/output interface 33 is configured to obtain user instructions from an input device connected to the server 30 and output information to an output device connected to the server 30.
The input device is, for example, a keyboard, pointing device, touch panel, or a combination thereof.
The output device is, for example, a display.

Communication interface 34 is configured to control communications between server 30 and an external device (eg, client device 10 or external system 50).

(2) One aspect of the embodiment One aspect of the present embodiment will be described. FIG. 4 is an explanatory diagram of one aspect of this embodiment.

As shown in FIG. 4, various engineers utilize the external system 50 to perform various development activities. The external system 50 stores information regarding logs of such development activities (hereinafter referred to as "development activity information"). The server 30 acquires, from the external system 50, information on development activities, particularly information regarding review requests. The server 30 acquires at least one of the content of the review request or additional information (described later) related to the review request. The server 30 may periodically collect development activity information regarding review requests, or may collect such development activity information in response to some trigger (for example, receiving an evaluation request from the client device 10).

The server 30 determines a review request to be evaluated from among the review requests stored in the external system 50. The server 30 generates model input information based on at least one of the content of the determined evaluation target (review request) or additional information related to the evaluation target.

The server 30 applies the review request evaluation model to the generated model input information to evaluate the review request corresponding to the model input information. The review request evaluation model is a trained model that performs inference for evaluation of review requests, and can be constructed based on the results of machine learning (supervised learning) described below.

The server 30 transmits the evaluation results of the review request to the client device 10. The client device 10 presents information based on the evaluation results of the review request to the user US1.

User US1 may be a person different from the engineer who made the review request to be evaluated. For example, if user US1 is a reviewer assigned to a review request, user US1 can estimate the man-hours required for the review in advance by checking the evaluation results for the review request before starting the review. It becomes easier. Alternatively, if user US1 is in a position to manage or evaluate the personnel of engineers who have made review requests, the evaluation results of review requests made by each engineer may be used to assign tasks within the team or to evaluate the personnel of engineers. It can be used as a material for judgment.

On the other hand, the user US1 may be the same person as the engineer who made the review request to be evaluated. In this case, by presenting information based on the evaluation results of the review request to the user US1 regarding the review request made by the user US1, it is possible to encourage the user to reflect or to encourage the user.

In this way, according to the present embodiment, it is possible to efficiently and objectively evaluate the quality of development work (specifically, review requests) by engineers. As a result, the quality of review requests can be improved, and low-quality review requests can be detected before the start of subsequent processes, improving development efficiency for the entire team (for example, by shortening lead times or reducing failures). Improvements in development speed) and quality (for example, software with fewer bugs) can be expected.

(3) Database The database of this embodiment will be explained. The following database is stored in the storage device 31.

(3-1) Repository database The repository database of this embodiment will be explained. FIG. 5 is a diagram showing the data structure of the repository database of this embodiment.

The repository database stores repository information. The repository information is information regarding the repository created in the external system 50.

As shown in FIG. 5, the repository database includes a "repository ID" field, a "team ID" field, and an "address" field. Each field is associated with each other.

A repository ID is stored in the "repository ID" field. The repository ID is information that identifies a repository. The server 30 can acquire the repository ID from the external system 50.

A team ID is stored in the "team ID" field. The team ID is information that identifies the team that created the repository specified by the corresponding repository ID. The server 30 can acquire the team ID from the external system 50. Note that the team ID is useful when collecting development activity information by engineers belonging to the same team, for example, to build a review request evaluation model described later on a team basis, but it is useful in cases where the collection of such information is not necessary. In this case, it is not necessary to store the team ID in the repository database.

The team ID may be associated with the organization ID in a database (not shown). The organization ID associated with the team ID identifies the organization to which the team corresponding to the team ID belongs. Further, the team ID may be associated with the engineer ID in a database (not shown). The engineer ID associated with the team ID identifies members belonging to the team corresponding to the team ID.

Address information is stored in the "address" field. The address information is information (eg, URL (Uniform Resource Locator)) for accessing the repository specified by the corresponding repository ID.

(3-2) Development activity database The development activity database of this embodiment will be explained. FIG. 6 is a diagram showing the data structure of the development activity database of this embodiment.

The development activity database stores development activity information. The development activity information is information regarding activities performed by engineers regarding source codes stored in a repository created in the external system 50.

For example, when a user (that is, an engineer) pushes source code from a local repository, the external system 50 records information that can identify the user who made the push and the date and time when the push was made as log information. Similarly, the external system records log information for at least one of the following activities:
・Activity of recording added or changed files in the local repository of the issue management service (e.g., corresponds to a commit on GitHub, and similar activities are hereinafter referred to as "commit")
・Activities to create issues regarding source code ・Activities to close issues ・Activities to request a review of files added or modified in the local repository (e.g., equivalent to a pull request on GitHub, hereinafter similar activities will be referred to as "pull requests") )
・Activities to review files in response to the above requests ・Activities to approve the integration of files requested for review into the main project (for example, equivalent to merging on GitHub, hereinafter similar activities are referred to as "merging") ・Comments Activities to create responses to comments/Activities to create responses to comments

The server 30 updates the development activity database based on the development activity information acquired from the external system 50. The server 30 may acquire development activity information periodically, or may acquire development activity information in response to some trigger (for example, receiving an evaluation request from the client device 10).

As shown in Figure 6, the development activity database includes an "activity ID" field, a "repository ID" field, an "action ID" field, an "engineer ID" field, a "date and time" field, and a "source code" field. field.

The activity ID is stored in the "activity ID" field. The activity ID is information that identifies an activity performed by an engineer regarding source code stored in a repository created in the external system 50.

A repository ID is stored in the "repository ID" field. The repository ID is information that identifies the repository in which the source code in which the activity specified by the corresponding activity ID was performed is stored.

Action information is stored in the "action" field. The action information is information regarding the type of activity specified by the corresponding activity ID.

The "Engineer ID" field stores an engineer ID. The engineer ID is information that identifies the engineer (that is, the user of the external system 50) who performed the activity specified by the corresponding activity ID.

The "date and time" field stores date and time information. The date and time information is information regarding the date and time when the activity specified by the corresponding activity ID was performed.

Source code information is stored in the "source code" field. The source code information is information regarding the source code in which the activity specified by the corresponding activity ID was performed (for example, file name information, information on changed files).

(4) Information processing Information processing of this embodiment will be explained. FIG. 7 is a flowchart of information processing according to this embodiment. FIG. 8 is a diagram showing an example of a screen displayed in the information processing of this embodiment.

The information processing in FIG. 7 may be executed periodically or in response to a request from the client device 10.

As shown in FIG. 7, the server 30 executes evaluation target determination (S130).
Specifically, the server 30 determines a review request to be evaluated from among the review requests stored in the external system 50.

As a first example of determining an evaluation target (S130), the server 30 determines a review request designated by a specific user as an evaluation target. The user may specify review requests to be evaluated individually, or may specify them for each repository. When the user specifies a repository, the server 30 can unconditionally determine all review requests for the repository to be evaluated.

As a second example of determining an evaluation target (S130), the server 30 determines a review request to which a specific user is assigned as a reviewer as an evaluation target.

As a third example of determining the evaluation target (S130), the server 30 determines as the evaluation target a review request made by an engineer designated by a specific user, or a review request to which the engineer is assigned as a reviewer.

As a fourth example of determining an evaluation target (S130), the server 30 determines as an evaluation target a review request generated in a team designated by a specific user.

As a fifth example of determining an evaluation target (S130), the server 30 determines a review request made by a specific user as an evaluation target.

As a sixth example of determining the evaluation target (S130), the server 30 selects review requests (in short, new review requests) that have occurred in the most recent predetermined period (for example, the period since the previous execution of this information processing), or A review request whose status has changed (for example, became open) within a period of time is determined as an evaluation target.

The server 30 can also combine two or more of the first to sixth examples.
Further, prior to step S130, the user of the client device 10 can give a user instruction regarding designation of an evaluation target, an engineer, or a team via the client device 10. The client device 10 may then generate an evaluation request based on the user's instruction and transmit it to the server 30.

Note that the server 30 may perform user authentication at the same time as the evaluation request, or before or after the evaluation request.
Specifically, the client device 10 acquires information used for user authentication and transmits it to the server 30. Based on this information, the server 30 confirms whether the operator of the client device 10 is a registered user. The server 30 executes subsequent processing if the user authentication is successful. However, if user authentication is performed at the same time as or after the evaluation request, and the user is not assigned the authority to view the evaluation results of the evaluation target, the server 30 rejects the evaluation request.

The information used for user authentication includes information that can identify the user (for example, a user ID) and information provided by the operator of the client device 10 to prove that the operator is the user himself/herself (Example 7). For example, passwords, biometric information, etc.).

After step S130, the server 30 executes information acquisition (S131).
Specifically, the server 30 refers to the development activity database (FIG. 6), for example, and extracts information regarding the evaluation object according to the evaluation object (review request) determined in step S130. The information to be extracted may be the review request itself to be evaluated, additional information related to the review request, or a combination of both. Note that the server 30 may not only directly acquire information from the development activity database, but may also process or analyze the information as necessary.

As a first example of information acquisition (S131), the server 30 acquires volume information. The volume information is information regarding the volume of the review request (evaluation target). The volume information can include information regarding at least one of the following, for example.
・Number of changed lines in review request ・Number of changed files in review request

It is thought that the greater the number of changed lines in a review request, the greater the risk of failure caused by the review request (described later) or the lead time between processes related to the review request (described later). It is considered that the greater the number of changed files in a review request, the greater the risk of failure due to the review request or the longer the lead time between processes related to the review request.

As a second example of acquiring information (S131), the server 30 acquires character information. The character information is information regarding characters constituting a review request (evaluation target) or additional information of the review request. The character information can include, for example, information regarding at least one of the following.
・Extension of the modified file in the review request ・Character strings that appear in the modified file name ・Character strings that appear in the modified file ・Feature vectors extracted from the modified file

The information on the extension of the name of the changed file in the review request is used to identify the programming language or scripting language used to write the changed file, and to determine whether the changed file corresponds to a program change. It can be. That is, it is possible to determine the type of review request from the information on the extension of the changed file name of the review request. It is thought that there is a correlation between the type of review request and the risk of failure occurrence or lead time. As a first example, if the extension is "cpp", "rb", or "py", the server 30 can determine that the review request is a program change. As a second example, if the extension is "yml", "json", or "xml", the server 30 can determine that the review request is probably a configuration file change. As a third example, if the extension is "lock", the server 30 can determine that the review request is a change to a dependent library.

Information on the character string appearing in the changed file name of the review request can also be used as material for determining the type of the review request. As a first example, if the character string "Pipfile", "pyproject", or "pom" appears in the modified file name, the server 30 can determine that the review request is a modification of a dependent library. For example, if a user changes their Pipfile, the dependency management tool will automatically update Pipfile.lock in addition to the user's manually updated Pipfile. Although PiPefile.lock is an update in the scale of hundreds to thousands of lines, it is an automatically generated result and does not need to be confirmed by review, so it is less likely to cause problems compared to code of the same size updated manually. It can be determined that the risk of occurrence is low. As a second example, if the character string "Client" or "network" appears in the changed file name, the server 30 determines that the review request is a change related to an element that communicates with the outside, that is, it is difficult to test and may cause bugs. It can be determined that there is a possibility that the As a third example, if the character string "util" appears in the changed file name, the server 30 determines that the review request is a change related to an element that is referenced from various places, that is, the scope of the change is large and there is a bug. It can be determined that there is a possibility that this is likely to occur.

Information on character strings that appear in a changed file requested for review can be used as material for determining signs of bugs caused by the changed file. As a first example, if the number of occurrences of "def" or "func" in the modified file is small (for example, below a threshold value depending on the number of lines in the file), the server 30 determines whether the bloated function is a bug. It can be determined that there is a possibility of becoming a hotbed. As a second example, if the variable name in the change file is "tmp", "t", "a", or "b", the server 30 is aware that the variable name that is difficult to understand may become a breeding ground for bugs. It can be determined that there is.

When extracting a feature vector from a changed file of a review request, the server 30 can use a neural network such as BERT (Bidirectional Encoder Representations from Transformers), for example.

As a third example of information acquisition (S131), the server 30 acquires time information. The time information is information regarding the time when a process related to the review request (evaluation target) was performed or the lead time between processes related to the review request. The time information can include information regarding at least one of the following, for example.
・Actual or predicted value of the time required from the creation of a review request to the completion of a review ・The time when the file corresponding to the review request was recorded (committed) in the local repository ・The time when the review request was created ・The review request changed from draft status Actual or predicted value of the time required to transition to the open state ・Actual or predicted value of the time required from the time a review request transitions to the open state until a reviewer is assigned Here, the above predicted value is based on the past Even if it is a representative value (e.g., mean value, median value, mode, maximum value, minimum value, first quartile, or third quartile) of multiple actual values measured in good.

The commit time of the file corresponding to the review request or the creation time of the review request can be used to determine the condition of the engineer at that time. For example, the server 30 determines that review requests committed or created late at night are more likely to contain defects due to careless mistakes than review requests committed or created at other times, and are more likely to cause failures. It may be determined that the risk of occurrence is high or the lead time is long.

The time required for a review request to transition from a draft state to an open state can be used to determine whether the specification design is well-organized or whether development is possible with a small patch size. The server 30 can evaluate that the shorter the actual measured value or predicted value of the required time, the lower the risk of failure occurrence or the shorter the lead time.

The time required from when a review request transitions to an open state until a reviewer is assigned can be used to determine whether reviewer assignment is appropriately organized. The server 30 can evaluate that the shorter the actual measured value or predicted value of the required time, the lower the risk of failure occurrence or the shorter the lead time.

The predicted value of the above-mentioned required time may be predicted based on the information of the worker in charge of each process, for example, or based on a trained model (for example, the second example of the review request evaluation model described below) based on other information. It may be predicted by

As a fourth example of information acquisition (S131), the server 30 acquires worker information. The worker information is information about the worker in each process related to the review request (evaluation target). The worker information can include, for example, information regarding at least one of the following.
・Engineer who made the review request ・Reviewer assigned to the review request

Note that before extracting information from the development activity database, the server 30 may acquire development activity information from the external system 50 and update the database to reflect the latest information held by the external system 50. .

After step S131, the server 30 executes generation of model input information (S132).
Specifically, the server 30 generates model input information based on the information regarding the evaluation target acquired in step S131. As an example, the server 30 may convert the information acquired in step S131 into feature quantities, and generate a vector in which each feature quantity is arranged as model input information.

After step S132, the server 30 evaluates the review request (S133).
Specifically, the server 30 applies the review request evaluation model to the model input information generated in step S132. Thereby, the server 30 evaluates the review request determined as the evaluation target in step S130.

The first example of the review request evaluation model makes inferences about the risk of failure caused by the review request (evaluation target). Here, the failure occurrence risk may be, for example, a modified failure rate. The change failure rate means the rate of failures that occur due to changes. As an example, the change failure rate in GitHub can be expressed as the ratio of the number of pull requests merged into the main branch from a branch tagged as failure to the number of pull requests merged into the main branch. A branch tagged as faulty is, for example, a branch (in any uppercase or lowercase) that includes a string that indicates a modification (hotfix) or a string that indicates reverting a change (revert).

The first example of the review request evaluation model can be constructed by supervised learning. The learning data used for supervised learning includes model input information for learning and corresponding correct answer information. The model input information for learning is generated in the same format as the model input information (during operation) in step S132 based on review requests made in the past. The correct answer information indicates whether or not a failure actually occurred due to the corresponding review request. By learning a large amount of such learning data, it is possible to create a trained model that has a function of statistically predicting the risk of occurrence of a failure inherent in a review request corresponding to given model input information.

The second example of the review request evaluation model makes inferences about the lead time between processes regarding the review request (evaluation target). The lead time to be inferred can include, for example, at least one of the following.
・Time required from reviewer assignment to the first review ・Time required from review until the file corresponding to the review request is approved for integration into the main project (that is, merged in GitHub) ・Review request (evaluation target) is opened The time required from transition to the state until the file corresponding to the review request is approved (merged) for integration into the project body.

The second example of the review request evaluation model can be constructed by supervised learning. The learning data used for supervised learning includes model input information for learning and corresponding correct answer information. The model input information for learning is generated in the same format as the model input information (during operation) in step S132 based on review requests made in the past. The correct answer information indicates the actual value of the lead time between processes regarding the corresponding review request. By learning a large amount of such training data, it is possible to create a trained model that has a function of statistically predicting the lead time between processes regarding review requests corresponding to given model input information.

Review request evaluation models can be constructed for various units. Specifically, the review request evaluation model may be constructed for each individual engineer, team including multiple engineers, organizational unit including multiple teams, or industry unit including multiple organizations. Although building a review request evaluation model in small group units is disadvantageous for collecting learning data, it has the advantage of easily increasing the degree of fit. Further, a review request evaluation model may be constructed for each programming language or markup language.

After step S133, the server 30 outputs the evaluation results (S134).
Specifically, the server 30 generates information for presenting the evaluation result of the review request in step S133 to the user of the client device 10. The server 30 transmits the generated information to the client device 10. Here, the generated information may be, for example, information for the client device 10 to generate a screen to be displayed on the display 21.

After step S134, the client device 10 executes information presentation (S110).
Specifically, the client device 10 receives the information transmitted in step S134, and causes the display 21 to display a screen based on the information. Thereby, the user can confirm the evaluation result (for example, the risk of occurrence of failure due to the review request or the lead time between processes regarding the review request) for the review request (evaluation target).

As an example, the client device 10 causes the display 21 to display a screen shown in FIG. The screen in FIG. 8 includes an object J20.

Object J20 displays review request information. The review request information includes information regarding the evaluation results of the review request. In the example of FIG. 8, the evaluation result of the review request includes the evaluation result of the risk of occurrence of a failure due to the review request, and the evaluation result of the lead time between processes regarding the review request. In the example of FIG. 8, the evaluation result is a three-level evaluation indicated by the number of displayed icons, but is not limited to this and may be expressed by a score or text. By checking these evaluation results, users can perform careful reviews to prevent failures, start reviews early to avoid long lead times, and provide feedback to engineers who requested reviews. It becomes easier to deal with things like going to work. Although not shown in FIG. 8, for example, when the evaluation result of the occurrence risk or lead time is below the threshold, the server 30 uses the information given to the evaluation of the occurrence risk or lead time among the information related to the evaluation target. An element that is estimated to have a large adverse effect may be identified, and information indicating the element may be presented to the user via the client device 10.

The review request information displayed in the object J20 can further include at least one of the following.
・Title of the review request ・Creator of the review request ・Status of the review request ・Date and time of creation of the regulation request

Further, the information displayed on the screen of FIG. 8 may be sorted or filtered according to user instructions. For example, the user can instruct via the client device 10 to sort the review requests in descending order of risk of occurrence of a failure, or to display only review requests by a specific creator.

(5) Summary As explained above, when an engineer uses the issue management service (in other words, the external system 50), the server 30 uses the information stored in the issue management service to enable the engineer to use the issue management service's local Obtain the contents of a review request for reviewing files added to or modified in the repository and additional information related to the review request. The server 30 applies a learned model (that is, a review request evaluation model) that performs inference for evaluating a review request to model input information based on the content of the acquired review request and additional information related to the review request. By doing so, the review request is evaluated and the evaluation result of the review request is output. This makes it possible to efficiently and objectively evaluate the quality of review requests, and allows users to utilize the evaluation results in their own work. As a result, the quality of review requests can be improved, and low-quality review requests can be detected before the start of subsequent processes, improving development efficiency for the entire team (for example, by shortening lead times or reducing failures). Improvements in development speed) and quality (for example, software with fewer bugs) can be expected.

The model input information may include volume information regarding the review request. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between the volume of the review request and the quality of the review request.

The volume information may relate to at least one of the number of changed lines or the number of changed files in the review request. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between the quality of the review request and at least one of the number of changed lines or the number of changed files in the review request.

The model input information may include character information regarding the review request. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between the characters (strings) related to the review request and the quality of the review request.

The character information is related to at least one of information regarding the extension of the modified file of the review request, information regarding the character string appearing in the modified file name, information regarding the character string appearing in the modified file, or a feature vector extracted from the modified file. It can be anything. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between each character information illustrated here and the quality of the review request.

The model input information may include time information regarding the review request. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between the time related to the review request and the quality of the review request.

The time information includes the actual or predicted value of the time required from the creation of the review request until the review is completed, the time when the file corresponding to the review request was recorded in the local repository, the time when the review request was created, and whether the review request is in draft status. It relates to at least one of the actual measured value or predicted value of the time required to transition to the open state, or the actual measured value or predicted value of the required time from the time the review request transitions to the open state until a reviewer is assigned. It's fine. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between each piece of time information illustrated here and the quality of the review request.

The model input information may include worker information regarding the review request. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between the worker regarding the review request and the quality of the review request.

The worker information may relate to at least one of the engineer who made the review request to be evaluated, or the reviewer assigned to the review request. Thereby, the quality of the review request to be evaluated can be efficiently and objectively evaluated based on the relationship between each piece of worker information illustrated here and the quality of the review request.

The server 30 may evaluate the review request by applying a trained model that makes inferences about the risk of occurrence of a failure due to the review request to the model input information. This makes it possible to efficiently and objectively evaluate the risk of failure caused by the review request being evaluated.

The server 30 may evaluate the review request by applying a learned model that makes inferences about the lead time regarding the review request to the model input information. This makes it possible to efficiently and objectively evaluate the lead time regarding the review request that is the subject of evaluation.

Lead time is the time required from reviewer assignment to the first review, from the review to the time the file corresponding to the review request is approved for integration into the main project, or from the time the review request transitions to the open state to the time required for the corresponding review request. It may also include at least one of the time required until the integration of the file corresponding to the review request into the project body is approved. Thereby, each lead time illustrated here can be efficiently and objectively evaluated.

The learned model may be constructed for each engineer, team including multiple engineers, organizational unit including multiple teams, or industry unit including multiple organizations. This increases the degree of freedom in constructing trained models.

A trained model may be constructed for each programming language or markup language. As a result, a trained model is constructed using training data obtained regarding a review request regarding a specific programming language or markup language, so that the accuracy of evaluation can be improved.

(6) Other Modifications The storage device 11 may be connected to the client device 10 via the network NW. Display 21 may be integrated with client device 10 . The storage device 31 may be connected to the server 30 via the network NW.

Each step of the above information processing can be executed by either the client device 10 or the server 30. For example, the client device 10 may execute the processing described as being executed by the server 30.

In the above description, an example was shown in which the server 30 outputs the evaluation result (S134) regardless of whether the evaluation result is good or bad. However, the server 30 may, for example, notify the user if a predetermined condition is satisfied regarding the evaluation result. The predetermined condition may be, for example, that the evaluation result is below the lower limit or that the evaluation is above the upper limit. The lower limit or upper limit may be set by the user, or may be set based on the evaluation results of review requests made in the past by the engineer who made the review request to be evaluated or by an engineer in the same team as the engineer. good. Notifications may be made on the app screen or website screen, or via email or other messages (e.g., SNS (Social Networking Service) messages, SMS (Short Message Service) messages, chat tool messages, etc.) It may also be carried out using The notification may include information that indicates that a predetermined condition has been met, or may include information about the evaluation result. Thereby, the user can quickly grasp the evaluation result when the predetermined condition is met without actively checking the evaluation result frequently.

The server 30 may analyze the influence of the elements included in the model input information on the evaluation results of the review request, and output the analysis results (for example, transmit information based on the analysis results to the client device 10). This allows the user (for example, the engineer who requested the review to be evaluated) to recognize the factors that contributed to a positive evaluation or the factors that contributed to a negative evaluation, and improves the reproducibility of high-quality review requests. Or, growth can be promoted by improving poor quality review requests.

The server 30 may evaluate the same review request at multiple times (for example, every time the status changes) and output the evaluation results at each of the multiple times of the review request. As a result, the evaluation results can be updated when the amount of information that can be referred to for generating model input information increases, for example, so that more accurate evaluation results can be provided to the user.

Further, in the above description, an example was shown in which each step is executed in a specific order in information processing, but the execution order of each step is not limited to the example described as long as there is no dependency relationship.

(7) Additional Notes The matters explained in the embodiment and the modified example are added below.

(Additional note 1)
computer (30),
The content of the review request requesting a review of a file that the first engineer has added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review request. means for acquiring additional information related to (S131);
means for evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request (S133);
means for outputting the evaluation results of the review request (S134);
A program that functions as

(Additional note 2)
The model input information includes volume information regarding review requests;
The program described in Appendix 1.

(Additional note 3)
The volume information is related to at least one of the number of modified lines or the number of modified files in the review request.
The program described in Appendix 2.

(Additional note 4)
The model input information includes character information regarding the review request.
The program described in Appendix 1.

(Appendix 5)
The character information is related to at least one of information regarding the extension of the modified file of the review request, information regarding the character string appearing in the modified file name, information regarding the character string appearing in the modified file, or a feature vector extracted from the modified file. ,
The program described in Appendix 4.

(Appendix 6)
The model input information includes time information regarding review requests;
The program described in Appendix 1.

(Appendix 7)
The time information includes the actual or predicted value of the time required from the creation of the review request until the review is completed, the time when the file corresponding to the review request was recorded in the local repository, the time when the review request was created, and whether the review request is in draft status. Regarding at least one of the actual measured value or predicted value of the time required to transition to the open state, or the actual measured value or predicted value of the required time from the time the review request transitions to the open state until the reviewer is assigned,
The program described in Appendix 6.

(Appendix 8)
The model input information includes worker information regarding the review request.
The program described in Appendix 1.

(Appendix 9)
The worker information is related to at least one of the first engineer or the reviewer assigned to the review request.
The program described in Appendix 8.

(Appendix 10)
The means for evaluating a review request is to apply a trained model that makes inferences about the risk of failure due to a review request to the model input information, and to evaluate the review request.
The program described in Appendix 1.

(Appendix 11)
The means for evaluating review requests is to apply a trained model that makes inferences about lead time regarding review requests to model input information, and to evaluate review requests.
The program described in Appendix 1.

(Appendix 12)
Lead time is the time required from reviewer assignment to the first review, from the review to the time the file corresponding to the review request is approved for integration into the main project, or from the time the review request transitions to the open state to the time required for the corresponding review request. including at least one of the time required for a file corresponding to a review request to be approved for integration into the project body;
The program described in Appendix 11.

(Appendix 13)
The trained model is built for each engineer, team containing multiple engineers, organizational unit containing multiple teams, or industry unit containing multiple organizations.
The program described in Appendix 1.

(Appendix 14)
A trained model is built for each programming language or markup language.
The program described in Appendix 1.

(Appendix 15)
computer,
A means for analyzing the influence of elements included in the model input information on the evaluation results of the review request;
means for outputting impact analysis results;
The program described in Appendix 1, which functions as

(Appendix 16)
The means of evaluation is to evaluate the review request at multiple points in time,
The outputting means outputs the evaluation results at each of multiple points in time of the review request.
The program described in Appendix 1.

(Appendix 17)
The computer (30)
The content of the review request requesting a review of a file that the first engineer has added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review request. a step of acquiring additional information related to (S131);
a step of evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request (S133);
A method of performing the steps of outputting an evaluation result of a review request (S134).

(Appendix 18)
The content of the review request requesting a review of a file that the first engineer has added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review request. means (S131) for acquiring additional information related to;
means for evaluating a review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request (S133);
An information processing device (30) comprising means (S134) for outputting an evaluation result of a review request.

(Appendix 19)
A system (1) comprising a first information processing device (30) and a second information processing device (10),
The first information processing device is
The content of the review request requesting a review of a file that the first engineer has added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review request. means (S131) for acquiring additional information related to;
means for evaluating a review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request (S133);
means (S134) for outputting the evaluation result of the review request to the second information processing device;
system.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above embodiments. Moreover, various improvements and changes can be made to the embodiments described above without departing from the spirit of the present invention. Furthermore, the above embodiments and modifications can be combined.

1 : Information processing system 10 : Client device 11 : Storage device 12 : Processor 13 : Input/output interface 14 : Communication interface 21 : Display 30 : Server 31 : Storage device 32 : Processor 33 : Input/output interface 34 : Communication interface 50 : External system

Claims (21)

computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
By applying a trained model that infers the lead time related to the review request to model input information including volume information related to the review request based on the content of the review request and additional information related to the review request, means for evaluating the review request;
means for outputting the evaluation results of the review request;
function as
The lead time is the time required from reviewer assignment to the first review, the time required from review until the file corresponding to the review request is approved for integration into the project body, or the time required for the review request to transition to an open state. including at least one of the time required for the file corresponding to the review request to be approved for integration into the project body;
program. The volume information relates to at least one of the number of changed lines or the number of changed files in the review request.
The program according to claim 1. computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
A trained model that performs inference for evaluation of the review request is based on the content of the review request and additional information related to the review request, and is based on model input information including volume information regarding the number of changed files of the review request. means for evaluating said review request by applying;
means for outputting the evaluation results of the review request;
A program that functions as computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
means for evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
means for outputting the evaluation results of the review request;
function as
The means for evaluating the review request evaluates the review request by applying a trained model that makes inferences about lead time regarding the review request to the model input information including worker information regarding the review request,
The lead time is the time required from reviewer assignment to the first review, the time required from review until the file corresponding to the review request is approved for integration into the project body, or the time required for the review request to transition to an open state. including at least one of the time required for the file corresponding to the review request to be approved for integration into the project body;
program. computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
means for evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
means for outputting the evaluation results of the review request;
function as
The means for evaluating the review request applies a trained model that makes inferences regarding the lead time regarding the review request to the model input information including worker information regarding the reviewer assigned to the review request. evaluate the request,
program. computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
means for evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
means for outputting the evaluation results of the review request;
function as
The means for evaluating the review request evaluates the review request by applying a trained model that makes inferences about the risk of occurrence of a failure due to the review request to the model input information including time information regarding the review request. ,
program. The time information includes the actual measured value or predicted value of the time required from the creation of the review request to the completion of the review, the time when the file corresponding to the review request was recorded in the local repository, the creation time of the review request, and the The actual or predicted value of the time required for a review request to transition from the draft state to the open state, or the actual or predicted value of the time required from the time the review request transitions to the open state until a reviewer is assigned. relating to at least one
The program according to claim 6. computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
means for evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
means for outputting the evaluation results of the review request;
function as
The model input information includes character information regarding the review request.
program. The character information may include information regarding the extension of the modified file of the review request, information regarding the character string appearing in the modified file name, information regarding the character string appearing in the modified file, or information about the feature vector extracted from the modified file. relating to at least one
The program according to claim 8. computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
means for evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
means for outputting the evaluation results of the review request;
function as
The learned model is constructed for each engineer, for each team including multiple engineers, for each organizational unit including multiple teams, or for each industry unit including multiple organizations.
program. computer,
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review means for obtaining additional information related to the request;
means for evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
means for outputting the evaluation results of the review request;
function as
The trained model is constructed for each programming language or markup language,
program. The computer,
means for analyzing the influence of elements included in the model input information on the evaluation results of the review request;
means for outputting the analysis result of said influence;
The program according to any one of claims 1 to 11, which functions as a program. The evaluating means evaluates the review request at a plurality of points in time,
The outputting means outputs evaluation results at each of the plurality of time points of the review request,
The program according to any one of claims 1 to 11. The computer is
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review obtaining additional information related to the request;
By applying a trained model that infers the lead time related to the review request to model input information including volume information related to the review request based on the content of the review request and additional information related to the review request, evaluating the review request;
performing the step of outputting the evaluation result of the review request;
The lead time is the time required from reviewer assignment to the first review, the time required from review until the file corresponding to the review request is approved for integration into the project body, or the time required for the review request to transition to an open state. including at least one of the time required for the file corresponding to the review request to be approved for integration into the project body;
Method. The computer is
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review obtaining additional information related to the request;
Evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
performing the step of outputting the evaluation result of the review request;
In the step of evaluating the review request, evaluating the review request by applying a trained model that makes an inference about the lead time regarding the review request to the model input information including worker information regarding the review request,
The lead time is the time required from reviewer assignment to the first review, the time required from review until the file corresponding to the review request is approved for integration into the project body, or the time required for the review request to transition to an open state. including at least one of the time required for the file corresponding to the review request to be approved for integration into the project body;
Method. The computer is
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review obtaining additional information related to the request;
Evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
performing the step of outputting the evaluation result of the review request;
In the step of evaluating the review request, the review request is evaluated by applying a trained model that makes an inference regarding the risk of occurrence of a failure due to the review request to the model input information including time information regarding the review request. ,
Method. The computer is
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review obtaining additional information related to the request;
Evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
performing the step of outputting the evaluation result of the review request;
The model input information includes character information regarding the review request.
Method. The computer is
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review obtaining additional information related to the request;
Evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
performing the step of outputting the evaluation result of the review request;
The learned model is constructed for each engineer, for each team including multiple engineers, for each organizational unit including multiple teams, or for each industry unit including multiple organizations.
Method. The computer is
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review obtaining additional information related to the request;
Evaluating the review request by applying a trained model that performs inference for evaluating the review request to model input information based on the content of the review request and additional information related to the review request;
performing the step of outputting the evaluation result of the review request;
The trained model is constructed for each programming language or markup language,
Method. The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review a means for obtaining additional information related to the request;
By applying a trained model that infers the lead time related to the review request to model input information including volume information related to the review request based on the content of the review request and additional information related to the review request, means for evaluating the review request;
and means for outputting the evaluation results of the review request ,
The lead time is the time required from reviewer assignment to the first review, the time required from review until the file corresponding to the review request is approved for integration into the project body, or the time required for the review request to transition to an open state. including at least one of the time required for the file corresponding to the review request to be approved for integration into the project body;
Information processing device. A system comprising a first information processing device and a second information processing device,
The first information processing device includes:
The content of the review request requesting a review of a file that the first engineer added or changed to the local repository of the issue management service based on the information stored in the issue management service when the engineer uses the issue management service and the review a means for obtaining additional information related to the request;
By applying a trained model that infers the lead time related to the review request to model input information including volume information related to the review request based on the content of the review request and additional information related to the review request, means for evaluating the review request;
means for outputting an evaluation result of the review request to the second information processing device ,
The lead time is the time required from reviewer assignment to the first review, the time required from review until the file corresponding to the review request is approved for integration into the project body, or the time required for the review request to transition to an open state. including at least one of the time required for the file corresponding to the review request to be approved for integration into the project body;
system.