JP2021089494A - System, method and program - Google Patents

Abstract

To provide a system capable of providing an estimation result of machine learning in near real time.SOLUTION: The system provides an estimation result of a learning model using a plurality of parameters and includes accepting means which accepts input of a plurality of parameters relating to a destination to which an estimation result should be provided, providing means which provides the destination with a first estimation result of the learning model using the plurality of accepted parameters, and monitoring means which executes monitoring of accuracy of the estimation result on the basis of information obtained about the destination after provision of the first estimation result. The accepting means accepts input of a plurality of further parameters relating to the destination after the first estimation result is provided, and the providing means provides the destination with a second estimation result of the learning model using the plurality of further accepted parameters if the accuracy of the first result obtained by the monitoring is lower than a threshold and the second estimation result is different from the first estimation result.SELECTED DRAWING: Figure 1

Description

The present invention relates to systems, methods, and programs that provide estimation results by machine learning.

In recent years, due to the evolution of computer processors and the development of algorithms, machine learning (including deep learning), which is a type of artificial intelligence, has evolved to the point where it can be applied to actual work.
In machine learning, a learning model is created and adjusted using learning data, and the adjusted learning model is used as a learned model. An estimation system that estimates a certain matter using such a trained model is known. Application examples include search engines, medical diagnostics, and financial market estimates.
Patent Document 1 discloses a technique of detecting a prediction error in a trained model, transmitting an actually occurring event as teacher data, and updating parameters by machine learning again. According to Patent Document 1, improvement of the medium- to long-term model is expected.

Japanese Unexamined Patent Publication No. 2017-21588898

Assume a use case that adds an estimation function to a business system using a trained model. For example, a vehicle allocation system in which a large number of drivers and vehicles are registered and dispatched to passengers who wish to board the vehicle can be considered. Normally, the driver patrols any place or stops somewhere and waits for a vehicle dispatch instruction from the vehicle dispatch system. By adding a function to estimate the number of passengers who wish to board and the patrol area as an estimation function to this vehicle allocation system, the occupancy rate can be improved and the sales of the vehicle allocation system operating company and individual drivers can be improved. It can be planned.
The estimation in the vehicle allocation system is a near real-time estimation that is directly linked to the daily sales, and it is desirable to monitor the estimation accuracy and sales at short intervals such as 30 minutes or 15 minutes. According to Patent Document 1, improvement of the model in the medium to long term is expected, but near real-time improvement is not considered.
An object of the present invention is to provide a system capable of providing estimation results of machine learning in near real time.

The system of one embodiment of the present invention is a system that provides an estimation result of a learning model using a plurality of parameters, and is a receiving means that accepts input of a plurality of parameters related to a destination to which the estimation result should be provided, and the above-mentioned system. The estimation result is based on the providing means for providing the first estimation result of the learning model using the plurality of received parameters to the providing destination and the information obtained about the providing destination after providing the first estimation result. The receiving means further accepts the input of a plurality of parameters relating to the providing destination after providing the first estimation result, and the providing means further receives the input of the plurality of parameters. When the accuracy of the first estimation result is lower than the threshold value and the second estimation result of the learning model using the plurality of received parameters is different from the first estimation result, the second estimation result is referred to as the above. It is characterized by providing it to the provider.

According to the present invention, it is possible to provide a system capable of providing the estimation result of machine learning in near real time.

It is a schematic diagram which shows the whole structure of the system which concerns on Example 1 of this invention. It is a block diagram which shows an example of the internal structure of the information processing apparatus in Example 1 of this invention. It is a block diagram which shows an example of the software structure in Example 1 of this invention. It is a figure which shows an example of UI of a business client application. It is a figure which shows the monitoring method of the estimation result and the actual result. It is a figure which shows the background re-estimation method. It is a flowchart of the process executed in the background every minute. It is a flowchart which shows the flow of the whole process of accuracy monitoring. It is a flowchart of the process to execute every 15 minutes in the whole process of accuracy monitoring. It is a flowchart of the process to execute every 30 minutes in the whole process of accuracy monitoring. It is a flowchart of the process which receives the judgment input for the re-estimation result from the user. This is a flowchart of a process that requires the user to input a judgment on the re-estimation result and an UI display example of a business client application.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
Hereinafter, as an embodiment of the present invention, for example, a vehicle allocation system for allocating a vehicle in a taxi company will be described as an example, but the present invention is not limited to the vehicle allocation system and is also applicable to other systems. It is possible.

<System configuration>
FIG. 1 is a schematic view showing an overall configuration of a system according to a first embodiment of the present invention.
This system includes a business server 111, a client terminal 112, an estimation server 113, and a monitoring server 114. The business server 111, the client terminal 112, the estimation server 113, and the monitoring server 114 are each composed of one or more information processing devices. These information processing devices include computers, virtual machines, smartphones, tablets, and the like.

<Internal configuration of information processing device>
FIG. 2 is a block diagram showing an example of the internal configuration of the information processing device according to the first embodiment of the present invention, and shows the module configuration of the information processing function of the information processing device.

The network interface 202 is a network interface that connects to a network 105 such as a LAN and communicates with another computer or network device. LAN is an abbreviation for Local Area Network. The communication method handled by the network interface 202 may be either wired or wireless. ROM 204 is a ROM in which embedded programs and data are recorded. ROM is an abbreviation for Read Only Memory. The RAM 205 is a RAM in a temporary memory area. RAM is an abbreviation for Random Access Memory. The secondary storage device 206 is a secondary storage device typified by an HDD or a flash memory. HDD is an abbreviation for hard disk drive. The CPU 203 is a CPU that executes a program read from the ROM 204, the RAM 205, the secondary storage device 206, and the like. CPU is an abbreviation for Central Processing Unit. The user interface 201 is a user interface for inputting / outputting information and signals using a display, a keyboard, a mouse, buttons, a touch panel, and the like. Each unit is connected via the input / output interface 207. Even a computer that does not have these hardware can be used as the information processing device in the first embodiment by operating from another computer using a remote desktop, a remote shell, or the like.

<Functional configuration of this system>
FIG. 3 is a diagram showing a software configuration according to the first embodiment of the present invention. The software installed on the hardware of each server 111, 113 and 114 and the client terminal 112 is executed by the respective CPU 203 and has a configuration capable of communicating with each other as shown by the arrow of the network connection.

The business application 311 is an application executed on the business server 111. In this embodiment, the case where the business application 311 is the business application of the vehicle allocation system described above will be described as an example. The vehicle dispatch system operating company and the individual driver are examples of the providers to which the estimation results should be provided. The data store 312 is a data store for storing and querying business data used by the business application 311. The business application 311 includes an API and a Web UI. API is an abbreviation for Application Programming Interface. Data used in the business application 311 is input / output by API. Web UI is an abbreviation for Web User Interface. The Web UI is a user interface composed of HTML, Javascript (registered trademark), and the like. HTML is an abbreviation for HyperText Markup Language. The business client application 321 is an application executed by the client terminal 112. The business client application 321 loads the Web UI from the business application 311 and calls the API to acquire and display the necessary data. The business client application 321 may be a Web application format that can be executed by a browser, or may be an application format that is individually installed on the client terminal 112. Here, the case where the business application 311 is a vehicle allocation system will be described. When the business client application 321 is executed, the business client application 321 communicates with the business application 311 to display a vehicle allocation request for a passenger who wishes to board the vehicle, and navigates the driver.

The estimation application 331 is an application executed by the estimation server 113. The trained model 332 is a trained model used for estimation. The trained model 332 is an example of a training model. The trained model 332 is created and arranged in advance by machine learning. Parameter 333 is an input parameter used for estimation. Parameter 333 is an example of a plurality of parameters relating to the destination to which the estimation result should be provided. The estimation application 331 collects data used for estimation from the business application 311 and other external systems in real time, and uses it as a parameter 333. The estimation application 331 is an example of a reception means that accepts input of a plurality of parameters relating to a destination to which the estimation result should be provided. When the business application 311 is a vehicle allocation system, examples of parameter 333 include own vehicle position, other vehicle position, date, day of the week, time zone, past boarding record, current weather, weather forecast, and manual estimation by district. Information on event holdings and railway operation status can be mentioned. Since there are many other examples of parameter 333 as a useful input parameter, not all of them are listed here, but the estimation server 113 creates a trained model 332 by machine learning in advance using parameter 333. Keep it.

The monitoring application 341 is an application executed by the monitoring server 114. The monitoring condition storage unit 342 stores the monitoring condition. The monitoring history storage unit 343 stores the monitoring history. The monitoring application 341 receives a copy of the estimation result from the estimation application 331. In addition, the monitoring application 341 receives a copy of the actual results from the business application 311. Here, the actual result is, for example, the history of the occupancy rate for each vehicle. The occupancy rate is the ratio of the time the passenger is in the vehicle to the unit time. The monitoring application 341 stores a copy of the received estimation result and a copy of the actual result in the monitoring history storage unit 343. The monitoring application 341 monitors changes in the monitoring history and generates an event when the monitoring conditions defined in the monitoring condition storage unit 342 are met. In the case of a vehicle allocation system, the monitoring application 341 sets a target value of the occupancy rate per unit time for each driver in the monitoring conditions. The monitoring application 341 raises an event when the occupancy rate per unit time is expected to be below the target value. Details of monitoring will be described later in a separate figure.

FIG. 4 is an example of a user interface (UI) of the business client application 321. The UI 401 is the UI of the business client application 321. The rectangle 402 is a rectangle that distinguishes areas on the map. The icon 403 is an icon indicating the position of the own vehicle. Message 404 is a message indicating a patrol recommended area in the current time zone estimated by the estimation application 321. The UI 401 of FIG. 4 has an area "1", an area "2", an area "3", an area "4", an area "5", an area "6", an area "7", an area "8", and an area "9". 9 areas are displayed.
The UI401 displays a map and the position of the own vehicle as an application of the vehicle allocation system. In addition, the UI401 navigates the driver to the position of the passenger when the vehicle is dispatched to the passenger who wishes to board the vehicle. When waiting for a vehicle to be dispatched, the driver is navigated to the estimated recommended patrol area.

FIG. 5 is a diagram showing a monitoring method of estimation results and actual results. A method of monitoring the estimation result and the actual result output by the estimation application 321 by the monitoring application 341 will be described with reference to FIG. The monitoring application 341 performs near real-time monitoring in units of 15 minutes and 30 minutes as described below.
Graph 501 is a graph in which the X-axis is the time and the Y-axis is the accuracy. In this embodiment, the occupancy rate per unit time for each vehicle of the vehicle allocation system, that is, for each driver is used as the accuracy. In this embodiment, as indicated by reference numeral 502, 30 minutes is set as a unit time, and the occupancy rate target value is set to 50%. That is, if the total time used by the passengers out of 30 minutes is 15 minutes or more, the target is achieved. Reference numeral 503 indicates an accuracy monitoring interval by the monitoring application 341. In this embodiment, the accuracy (boarding rate) per unit time is monitored every 15 minutes, and an event is generated when it is determined that the target value cannot be exceeded. These unit time, target value, and accuracy monitoring interval setting values are set in the monitoring condition storage unit 342.

It will be explained in more detail in chronological order. In Graph 501, the accuracy monitoring was OK because the occupancy rate was 50% during the time zone from 15:00 to 15:15. Similarly, the accuracy monitoring was OK because the occupancy rate exceeded 50% in the time zone of 15: 15-15: 30. However, during the time zone of 15: 30-15: 45, the occupancy rate is only 10%, and it is judged that it is difficult to achieve the target value as the occupancy rate per unit time at 15:45 at this pace. To. At this time, the monitoring application 341 generates an event. The criterion for judging that it is difficult to achieve the target value is that the target value is not achieved even if the predetermined occupancy rate (for example, a few percent increase in the occupancy rate in the previous 15 minutes) is achieved in the next 15 minutes. Can be.

FIG. 6 is a diagram showing a background re-estimation method. FIG. 6 shows a process in which the estimation application 331 acquires the latest parameter 333 every minute and executes re-estimation in the background, in addition to the near real-time monitoring by the monitoring application 341 described with reference to FIG. It is a figure explaining. FIG. 6 displays the area numbers estimated by the estimation application 331 every minute in chronological order from the left to the right among the nine areas displayed by the UI 401 of FIG.
Reference numeral 601 indicates the first pattern of the occurrence patterns of the re-estimation result. In the first pattern, the latest estimated area is "7" with respect to the first estimated area "6", but from the dispersion status of the estimation results within the time zone of time 15: 30-15: 45. Then, "7" indicates a pattern that is not a peculiar estimation result.
On the other hand, reference numeral 602 indicates a second pattern of occurrence of the re-estimation result. In the second pattern, the latest estimation area is "7" as in the first pattern, but from the dispersion status of the estimation results within the time zone of time 15: 30-15: 45, " 7 ”shows a pattern in which a peculiar estimation result is obtained. The determination of the appearance of these patterns will be described later.

FIG. 7A is a flowchart of a process executed in the background every minute. FIG. 7B is a flowchart showing the overall processing flow of accuracy monitoring. FIG. 7C is a flowchart of a process executed every 15 minutes in the entire process of accuracy monitoring. FIG. 7D is a flowchart of a process executed every 30 minutes in the entire process of accuracy monitoring.

First, as shown in FIG. 7A, the estimation application 331 executes re-estimation every minute as described with reference to FIG. 6, and passes the re-estimation result to the monitoring application 341 (step S711). The monitoring application 341 stores the history of the received re-estimation result in the monitoring history storage unit 343.

Hereinafter, description will be made with reference to FIGS. 7B to 7D.
First, it will be described with reference to FIG. 7B. The driver of the vehicle using the vehicle dispatch system operates the client terminal 112 mounted on the vehicle to activate the business client application 321. The business client application 321 makes an inquiry about vehicle allocation to the business application 311. The business application 311 pushes the vehicle allocation instruction to the business client application 321 when there is an immediate vehicle allocation instruction such as a desire to board the vehicle of the business client application 321 that has been inquired. The business application 311 push-notifies the estimation result by the estimation application 331 to the business client application 321 when the vehicle allocation instruction is not immediately given (step S701). Here, the estimation result that the business application 311 pushes and notifies the business client application 321 is, for example, a patrol recommended area that estimates the area that the vehicle should patrol in order to improve sales. This estimation result is an example of the first estimation result of the learning model. The business application 311 is an example of a providing means for providing the first estimation result to the providing destination.
Upon receiving the push notification, the business client application 321 navigates the driver to the patrol recommended area based on the estimation result. The driver patrols the vehicle within the recommended area according to the navigation of the recommended patrol area estimation displayed on the client terminal 112.

The monitoring application 341 executes a 30-minute process (step S712) executed every 30 minutes and a 15-minute process (step S713) executed every 15 minutes. The processing every 15 minutes will be described later with reference to FIG. 7C. The processing every 30 minutes will be described later with reference to FIG. 7D.

This will be described with reference to FIG. 7C. The monitoring application 341 determines whether the current occupancy rate per unit time is expected to be lower than the target value during accuracy monitoring every 15 minutes (step S702). The driver is patrolling the recommended patrol area pushed and notified in step S701, and the occupancy rate per unit time in step S702 is an example of the accuracy of the first estimation result. The target value in step S702 is an example of a threshold value to be compared with the accuracy of the first estimation result. If the determination in step S702 is No, the monitoring application 341 waits until the next accuracy monitoring every 15 minutes is executed. When the determination in step S702 is Yes, the monitoring application 341 notifies the business application 311 of the decrease in accuracy as an event. Upon receiving the event notifying the decrease in accuracy, the business application 311 requests the estimation application 331 to perform re-estimation. Upon receiving the request for re-estimation execution, the estimation application 331 executes re-estimation (step S703), and determines whether the previous estimation result 15 minutes ago and the current re-estimation result are different (step S704). The result of re-estimation in step S704 is an example of the second estimation result.

If the determination in step S704 is No, the monitoring application 341 waits until the next accuracy monitoring every 15 minutes is executed. When the determination in step S704 is Yes, the monitoring application 341 acquires the history of the re-estimation result for each minute re-estimated in step S711 from the monitoring history storage unit 343 (step S705). The monitoring application 341 determines whether or not the latest re-estimation result is a peculiar estimation from the history of the re-estimation result acquired in step S705 (step S706). As a method for determining whether the result is a peculiar estimation result, the probability, variance, standard deviation, etc. of the number of occurrences may be calculated and determined. If the determination in step S706 is Yes, the monitoring application 341 waits until the next accuracy monitoring every 15 minutes is executed without adopting the re-estimation result. If the determination in step S706 is No, the monitoring application 341 adopts the re-estimation result and notifies the business application 311 of the new estimation result.

The business application 311 push-notifies the business client application 321 of the new estimation result (step S707). Upon receiving the push notification, the business client application 321 navigates the driver to the patrol recommendation area based on the new estimation result.

This will be described with reference to FIG. 7D. The business application 311 push-notifies the business client application 321 of the latest re-estimation result when 30 minutes, which is the actual measurement unit time of the occupancy rate, has elapsed (step S708).

The above is the basics of the system and method that monitors and detects changes in estimation results and actual results in near real time, executes re-estimation, and provides the latest estimation results to users (system users including drivers). It is a configuration.

By the way, with the above-mentioned basic configuration alone, under certain conditions, user intervention may be required rather than relying solely on the estimation of artificial intelligence. In the above-mentioned basic configuration, since the re-estimation every minute and the re-estimation every 15 minutes in the background are performed at relatively short time intervals, the change of the parameter 333 input to the estimation application 331 changes. It is possible that there is little change in the re-estimation result.
However, in this embodiment, the purpose of near real-time accuracy monitoring such as every 15 minutes is that there is a target that must be dealt with immediately in a short time, such as a occupancy rate target (taxi sales target). Since the estimation of artificial intelligence is only an estimation and is not guaranteed, if the occupancy rate and accuracy are low, we would like to request the acquisition of another re-estimation result at the user's own discretion.
Therefore, in this embodiment, the business application 311 also acquires the certainty of the estimation result and the explanation when acquiring the estimation result in the trained model, and uses this to input the judgment for the re-estimation result from the user. Provide a means of acceptance. The certainty of an estimation result is the probability that the estimation result will actually occur. The description of the estimation result is a value indicating which of the input parameters used for the estimation is effective for the estimation result at that time by weighting.

FIG. 8 is a flowchart of a process of receiving a judgment input for the re-estimation result from the user.
The business application 311 acquires the re-estimation result from the estimation application 331 before issuing the push notification in step S707 or S708 (step S801). The business application 311 determines whether the acquired re-estimation result has changed from the previous estimation result (step S802).
When the determination result in step S802 is No, the business application 311 determines whether the certainty of the re-estimation result acquired this time is equal to or greater than the threshold value (step S803). When the determination in step S803 is Yes, the business application 311 keeps the current estimation result (step S804). When the determination in step S803 is No, the business application 311 requests the user to input a determination on the re-estimation result via the business client application 321 (S807).
When the determination result in step S802 is Yes, the business application 311 determines whether the certainty of the re-estimation result acquired this time is equal to or greater than the threshold value (step S805). When the determination in step S805 is Yes, the business application 311 push-notifies the business client application 321 of the re-estimation result (step S806). When the determination in step S805 is No, the business application 311 requests the user to input a determination on the re-estimation result via the business client application 321 (S807).

FIG. 9 is a flowchart of a process for requesting the user to input a judgment for re-estimation, and a UI display example of the business client application 321.
The business client application 321 displays the recommended area of the re-estimation result among the nine areas of FIG. 4 in UI911, and also displays and provides the certainty of the re-estimation result and the estimation explanation (step S901). .. Further, the business client application 321 provides the user with another re-estimation result as an option (step S902).
In the first option provision pattern, UI 912 is provided in which the re-estimation results are listed in order of certainty and the user is made to select.
In the second option providing pattern, the UI 913 that can list the parameter weights at the time of estimation and remove unnecessary parameters is provided.
For example, it is assumed that the estimation result of the corresponding time zone is based on the expectation that many passengers will be generated because the event is held in the recommended patrol area. However, when the event is not actually held in rainy weather, the user can request re-estimation by excluding the parameter of the event information in UI913.

As described above, according to this embodiment, it is possible to monitor and detect changes in estimation results and actual results in near real time, execute re-estimation, and provide the latest estimation results to the user. In addition, according to this embodiment, even in a case where the re-estimation does not change within a short period of time, by requesting the user to input a judgment on the re-estimation, the user himself / herself makes another re-estimation according to the business request. It is possible to request an estimate.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Network 111 Business server 112 Client terminal 113 Estimated server 114 Monitoring server

Claims (7)

A system that provides estimation results of a learning model using multiple parameters.
A reception means that accepts input of multiple parameters related to the destination to which the estimation result should be provided,
A providing means for providing the first estimation result of the learning model using the plurality of received parameters to the providing destination, and
It has a monitoring means for executing monitoring of the accuracy of the estimation result based on the information obtained about the provision destination after providing the first estimation result.
After providing the first estimation result, the receiving means further accepts the input of a plurality of parameters relating to the providing destination, and then receives the input.
When the accuracy of the first estimation result by the monitoring is lower than the threshold value and the second estimation result of the learning model using the plurality of received parameters is different from the first estimation result, the providing means is used. The second estimation result is provided to the provider.
A system characterized by that. When the second estimation result is not a peculiar estimation result, the providing means provides the second estimation result to the providing destination.
The system according to claim 1. When the certainty of the second estimation result is less than the threshold value, the providing means provides the providing destination with the certainty of the second estimation result and the explanation together with the second estimation result.
The system according to claim 1 or 2. The receiving means further accepts the input of a plurality of parameters relating to the providing destination after providing the certainty of the second estimation result and the explanation together with the second estimation result.
The system according to claim 3, wherein the system is characterized by the above. The description is a value representing by weighting which parameter is effective for the estimation result at that time among the plurality of parameters used for estimating the second estimation result.
The system according to claim 3 or 4. A method performed in a system that provides estimation results for a learning model using multiple parameters.
A reception process that accepts input of multiple parameters related to the destination to which the estimation result should be provided,
A providing step of providing the first estimation result of the learning model using the received plurality of parameters to the providing destination, and a providing step.
It has a monitoring step of executing monitoring of the accuracy of the estimation result based on the information obtained about the provision destination after providing the first estimation result.
After providing the first estimation result, the receiving process further accepts input of a plurality of parameters relating to the providing destination.
In the providing step, when the accuracy of the first estimation result by the monitoring is lower than the threshold value and the second estimation result of the learning model using the plurality of received parameters is different from the first estimation result. The second estimation result is provided to the provider.
A method characterized by that. A program for operating a computer as each means of the system according to any one of claims 1 to 5.

Images