JP6934351B2 - AI service usage support system - Google Patents

Description

The present invention relates to an AI service use support system when using an AI service.

In recent years, AI (artificial intelligence) technology has been developed, and companies and the like have begun to use AI. In recent years, various companies have begun to provide computer systems (AI service systems) that realize services equipped with AI (AI services). In a general AI service system, when data such as a question is input to the AI service system, the AI service system analyzes the data and returns the result to the user.

As shown in the following non-patent documents, each company provides various AI service systems.

IBM Japan, Ltd., "IBM-Visual Recognition | IBM Watson Developer Cloud --Japan", [online], Internet <URL: https://www.ibm.com/watson/jp-ja/developercloud/visual-recognition.html ＞ Google Inc., "Vision API-Image Content Analysis | Google Cloud Platform", [online], Internet <URL: https://cloud.google.com/vision/> Microsoft Japan Co., Ltd., "Computer Vision-Image Processing and Image Analysis | Microsoft Azure", [online], Internet <URL: https://azure.microsoft.com/ja-jp/services/cognitive-services/computer-vision /> ＞

The reliability and usage fees of AI service systems provided by each company vary. The reliability of the response data obtained from the AI service system is also not constant. Therefore, it is conceivable to send the same inquiry data (question) to multiple AI service systems and compare the response data from each AI service system to ensure the reliability of the response data. In that case, There is a problem that the cost is high.

Therefore, in view of the above problems, the present inventor has invented an AI service use support system that can ensure the reliability of a certain amount of response data while keeping costs down.

The first invention is an AI service use support system that supports the use of an AI service system, and the AI service use support system sends inquiry data to an AI service system set for normal use. By doing so, the answer data and the certainty are accepted, and if the certainty does not satisfy the condition, the inquiry data is sent to the AI service system other than the AI service system set for normal use. This is an AI service usage support system that has an AI service inquiry processing unit.

AI service systems have strengths and weaknesses, and the reliability of their response data is not always constant. On the other hand, if you always make inquiries to multiple AI service systems, the cost will increase. Therefore, by configuring as in the present invention, the reliability of the response data can be ensured while suppressing the cost.

In the above-described invention, the AI service inquiry processing unit receives the response data to the inquiry data sent to the AI service system other than the AI service system set for the normal use, and is used for the normal use. Based on the response data received from the set AI service system and the response data received from the AI service system other than the normal time, the number of matching response data is used to determine the response data to be output as a result. , It can be configured like an AI service usage support system.

In the above-described invention, the AI service inquiry processing unit compares the response data received from the AI service system set for use during the normal time with the response data received from the AI service system other than the normal time. It can be configured like an AI service usage support system that determines the largest number of answer data as the answer data to be output as a result.

When answer data is received from a plurality of AI service systems, for example, if the number of answer data is large, the reliability of the answer data is enhanced. Therefore, as in these inventions, it is preferable to determine the answer data to be output as a result based on the number of answer data.

In the above-described invention, the AI service inquiry processing unit receives the response data and the certainty of the inquiry data sent to the AI service system other than the AI service system set for normal use, and receives the usual The answer data to be output as a result is determined by using the certainty of the answer data received from the AI service system set for use at the time and the certainty of the answer data received from the AI service system other than the normal time. , It can be configured like an AI service usage support system.

In the above-described invention, the AI service inquiry processing unit compares the response data received from the AI service system set for use during the normal time with the response data received from the AI service system other than the normal time. By calculating the number of matching answer data and the certainty, it can be configured like an AI service use support system that determines the answer data to be output as a result.

If certainty is accepted in addition to the answer data, the reliability of the output answer data can be improved by reflecting the certainty and determining the answer data to be output as a result.

In the above-described invention, the AI service inquiry processing unit receives response data for the inquiry data sent to an AI service system other than the AI service system set for use during the normal time, and receives the response data for the inquiry data other than the AI service system set for normal use. It can be configured like an AI service usage support system that determines the response data to be output as a result by using the number of matching response data among the response data received from the service system.

In the above-described invention, the AI service inquiry processing unit receives the response data and the certainty of the inquiry data sent to the AI service system other than the AI service system set for normal use, and receives the usual Of the answer data received from the AI service system other than the time and the certainty, the answer data of the AI service system with the highest certainty is determined as the answer data to be output as a result, and it is configured like an AI service usage support system. be able to.

If the response data received from the AI service system normally used does not satisfy the conditions, the reliability of the response data is not high. Therefore, the reliability of the response data can be improved by determining the response data to be output as a result based on the response data and the certainty of the AI service system other than the AI service system.

In the above-described invention, the AI service use support system further receives answer data by sending inquiry data to the AI service system to be evaluated, and uses the correct answer data corresponding to the inquiry data and the answer data. Therefore, it can be configured like an AI service use support system having an accuracy evaluation processing unit for calculating accuracy information of the AI service system.

The reliability of AI service systems provided by each company varies. Therefore, the company must make its own judgment on what kind of AI service system to use before considering it. However, it is not easy to determine which AI service system should be used. Therefore, by using the present invention, a company or the like using the AI service system can grasp the reliability of the answer of the AI service system. Therefore, it can be used as an aid in making a decision when using the AI service system.

In the above-described invention, the AI service use support system further includes an AI service information storage unit that stores accuracy information and charge information for each AI service system, and accuracy information and charge information that are stored in the AI service information storage unit. Based on the above, it can be configured like an AI service use support system having a use AI determination processing unit for determining an AI service system to be used at normal times.

Companies, etc., who are users, want to use an AI service system that has a good balance between reliability and price. Therefore, by using the present invention, it is possible to determine the AI service system in consideration of them.

In the above-described invention, the AI service use support system further includes an AI switching processing unit having a driver for each AI service system and a driver switching processing unit for switching the driver, and in a normal state, the AI service usage support system has an AI switching processing unit. , The confidence level received from the AI service system used during normal times is determined by executing input / output processing with the AI service system used during normal times using the driver corresponding to the AI service system used during normal times. If the conditions are not satisfied, it can be configured as an AI service use support system in which the driver switching processing unit switches the driver and executes input / output processing for each AI service system that sends the inquiry data. ..

Since the AI service system is provided by each company, the format may differ. Therefore, as in the present invention, by switching between the driver for each AI service system and the driver, it is possible to use a plurality of AI services with one type of inquiry data to be input and switch the use as appropriate.

The AI service utilization support system of the first invention can be realized by reading the program of the present invention into a computer and executing it. That is, the computer accepts the answer data and the certainty by sending the inquiry data to the AI service system set for normal use, and if the certainty does not satisfy the condition, the normal This is an AI service usage support program that functions as an AI service inquiry processing unit that sends the inquiry data to an AI service system other than the AI service system set for use at the time.

By using the AI service use support system of the present invention, it is possible to secure a certain accuracy of the response data while suppressing the cost.

It is a block diagram which shows typically an example of the whole structure of the AI service use support system of this invention. It is a block diagram which shows typically an example of the hardware composition of the computer used in the AI service use support system of this invention. It is a flowchart which shows an example of the processing process of the accuracy evaluation processing in the AI service use support system of this invention. It is a flowchart which shows an example of the processing process of the AI service switching process in the AI service use support system of this invention. It is a flowchart which shows an example of the processing process of AI inquiry processing in the AI service use support system of this invention. It is a figure which shows an example of the test data. It is a figure which shows another example of the test data. It is a figure which shows an example of the AI service information storage part typically. It is a block diagram which shows typically the relationship between the AI switching processing part and the AI service system. It is a figure which shows an example of the inquiry data which is input to the AI service system which is usually used.

FIG. 1 shows an example of the overall system configuration of the AI service use support system 1 of the present invention, and FIG. 2 shows an example of the hardware configuration of the computer used in the AI service use support system 1.

The AI service use support system 1 uses the management terminal 2, the AI service system 3, and the person in charge terminal 4. Each of these terminals and systems is realized by a computer. The computer includes an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display, and an input device 73 that inputs information. It has a communication device 74 for communicating the processing result of the arithmetic unit 70 and the information stored in the storage device 71. When the computer is provided with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used, for example, in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The management terminal 2 of the AI service use support system 1 may be realized by one computer, but its function may be realized by a plurality of computers. The computer in this case may be, for example, a cloud server.

Each means in the AI service use support system 1 of the present invention is only logically distinguished in its function, and may form the same area physically or substantially.

The AI service system 3 is a computer system of AI services provided by each company. Examples of the AI service system 3 include "Visual Recognition" provided by IBM, "Cloud Vision API" provided by Google, and "Computer Vision API" provided by Microsoft. The above is an example, and any AI service system 3 can be used. FIG. 1 illustrates a case where there are three types of AI service systems 3 as AI service system 3A to AI service system 3C, but a plurality of AI service systems 3 is sufficient, and the number is not limited.

The person-in-charge terminal 4 is a computer used by a person in charge in a company that uses the AI service system 3, and is an AI service system by inputting test data to the management terminal 2 described later and performing a predetermined operation. The switching process of 3 is executed.

The management terminal 2 includes a test data input reception processing unit 21, a test data storage unit 22, an accuracy evaluation processing unit 23, a charge information input reception processing unit 24, an AI service information storage unit 25, a usage AI judgment processing unit 26, and an AI switching process. It has a unit 27 and an AI service inquiry processing unit 28.

The test data input reception processing unit 21 receives the test data to be input to the AI service system 3 from the person in charge terminal 4, and stores it in the test data storage unit 22 described later. The test data includes inquiry data to be input to the AI service system 3 and correct answer data for the inquiry. An example of test data is shown in FIG. FIG. 6 (a) is the image data of the inquiry data “bear stuffed animal”, and FIG. 6 (b) is the text data of the correct answer data “fabric toy”. The AI service system 3 has strengths and weaknesses to be analyzed. Therefore, it is advisable to accept input of various types of test data (inquiry data) to be analyzed. Types of analysis targets include, for example, image analysis, translation, voice text conversion, and face recognition. As a combination of inquiry data and correct answer data in the test data, the inquiry data is image data, the correct answer data is text data that the image data means, the inquiry data is text data of any language, and the correct answer data is a language different from the inquiry data. Text data translated into, inquiry data is voice data, correct answer data is voice text data, inquiry data is face image data, correct answer data is the gender of the person in the face image data, text data of age, etc. be. The combination of inquiry data and correct answer data is not limited to these, and can be set arbitrarily.

The test data input reception processing unit 21 may receive, for example, about 100 to 1000 test data inputs, but the number is not limited.

The test data storage unit 22 stores the test data received by the test data input reception processing unit 21. The test data storage unit 22 supports test data identification information, test data (inquiry data and corresponding correct answer data), and types of analysis targets (image analysis, translation, voice text conversion, face recognition, etc.) for each test data. Attach and memorize.

The accuracy evaluation processing unit 23 executes the accuracy evaluation processing of each AI service system 3. That is, the inquiry data in the test data stored in the test data storage unit 22 is input to each AI service system 3, and the response data for the inquiry data is received from each AI service system 3. Then, the correct answer data corresponding to the inquiry data stored in the test data storage unit 22 is compared with the answer data received from the AI service system 3, and if they match or are included, it is determined to be the correct answer. If they do not match or are not included, the answer is judged to be incorrect. Then, based on the number of inquiry data input to the AI service system 3 and the number of correct answers,
Correct answer rate = number of correct answers ÷ number of inquiry data entered in the AI service system x 100
The correct answer rate is calculated by calculating, and it is used as accuracy information.

The method of calculating the correct answer rate used for accuracy information is not limited to the above. For example, the correct answer rate may be calculated by taking into account the accuracy of the correct answer (correct answer value) of the answer data for each inquiry data. The accuracy of the correct answer can be calculated using, for example, the number of answer data that matches the correct answer data among the answer data for the inquiry data. That is, for the inquiry data, the correct answer value is calculated based on the number of correct answer data for the inquiry data and the number of correct answer data included in the answer data.

For example, when the number of correct answer data for the inquiry data is two and the number of answer data output by the AI service system 3 is two, if each of the answer data matches the correct answer data, the correct answer value becomes "1". .. In addition, the number of correct answer data for the inquiry data is 5, and the correct answer value when the AI service system 3 outputs 8 answer data including 2 correct answer data is "0.4" (= 2/5). Become.

Then, the correct answer rate of the AI service system 3 is calculated by calculating using the correct answer value for each inquiry data.

By adopting such a method, it is possible to realize an accuracy evaluation that emphasizes the inclusion of correct answer data even if the number of answer data is large. The correct answer rate in this case can be calculated by the equation 1. Note that N in Equation 1 is the number of inquiry data to be input to the AI service system 3 as test data.

In addition, as the correct answer value, which is the accuracy of the correct answer, the number and ratio of the answer data (noise) that do not match the correct answer data among the answer data for the inquiry data can be considered. For example, the correct answer value can be calculated based on the number of answer data for the inquiry data and the number of correct answer data included in the answer data. In this case, the correct answer value when the answer data and the correct answer data completely match is set to "1". In addition, when there are two correct answer data for the inquiry data and the AI service system 3 outputs four answer data including two correct answer data, although all the correct answer data are included, two answers are made as noise. Since it contains data, the correct answer value is set to "0.5". Further, when there are 5 correct answer data for the inquiry data, it is assumed that 8 answer data including 2 correct answer data are output. The correct answer value in this case is "0.25" (= 2/8).

The inquiry data is the image data of the "husky dog" as shown in FIG. 7 (a), and the correct answer data is the text data of "animal", "dog", and "husky" as shown in FIG. 7 (b). Suppose there is. It is assumed that the image data shown in FIG. 7A is input to each of the AI service system 3A, the AI service system 3B, and the AI service system 3C as inquiry data. At that time, the answer data of "animal", "dog", "husky", "wolf", "sled dog" from AI service system 3A, and "animal", "dog", "husky" from AI service system 3B. , "Pet", and "animal", "dog", and "husky" are output from the AI service system 3C, respectively. In this case, since the answer data from the AI service systems 3A to 3C includes "animal", "dog", and "husky", it can be determined that the answer data for the inquiry data is correct. However, while the AI service system 3A outputs five response data, the AI service system 3C outputs only three response data. In this case, the accuracy of the correct answer may be the same, but the ease of use for the user is not necessarily the same. This is because the greater the number of answer data output by the AI service system 3, the higher the possibility that correct answer data will be included, but the fact that more answer data is output means that the AI service system 3 is actually used. This is because it means that the answer data that the user must consider when doing so increases. Therefore, just because the correct answer data is included does not mean that the number of answer data is large. Rather, it is preferable that the answer data completely matches the correct answer data. Therefore, the accuracy rate of the correct answer for the AI service system 3 may be calculated in consideration of the accuracy of the correct answer in consideration of the noise as described above.

By adopting such a method, it is possible to realize an accuracy evaluation that emphasizes noise removal. The accuracy rate in this case can be calculated by Equation 2. Note that N in Equation 2 is the number of inquiry data to be input to the AI service system 3 as test data.

The method of calculating the accuracy of the correct answer (correct answer value) is not limited to the above.

The accuracy evaluation processing unit 23 may calculate the correct answer rate for each type of analysis target, or may calculate the correct answer rate regardless of the type of analysis target. The AI service system 3 has strengths and weaknesses for each analysis target. Therefore, more accurate accuracy evaluation can be performed by calculating the accuracy information for each type of analysis target of the AI service system 3.

The accuracy evaluation processing unit 23 executes the accuracy evaluation process by sending the same test data (inquiry data) to each AI service system 3, and uses the correct answer rate as accuracy information in the AI service information storage unit 25 described later. Remember. The accuracy information may be stored in association with each AI service system 3. Further, the accuracy information may be an index for evaluating the accuracy of the analysis of the AI service system 3, and an index calculated by another evaluation method such as an incorrect answer rate other than the correct answer rate can be used.

The charge information input reception processing unit 24 receives input of charge information when using the AI service system 3 from the person in charge terminal 4 or the AI service system 3, and stores it in the AI service information storage unit 25 described later.

The AI service information storage unit 25 stores accuracy information, charges, and usage frequencies for each AI service system 3 in association with each other. FIG. 8 shows an example of the AI service information storage unit 25. The accuracy information is the accuracy information executed by the accuracy evaluation processing unit 23. At this time, the accuracy information of the entire AI service system 3 may be stored, or the accuracy information of each analysis target in the AI service system 3 may be stored. For example, in AI service system 3A, the analysis target is 98% accuracy information for image analysis, 80% accuracy information for translation, 30% accuracy information for voice text conversion, and 94% accuracy information for face recognition. It may be memorized as. The charge is a unit price table for inquiries about the AI service system 3. It should be noted that the information is not limited to the unit price, but may be information on the charge. The frequency of use is the number of inquiries made by the user to the AI service system 3 for a predetermined period, for example, per month. The frequency of use may be any of the number of inquiries actually made to the AI service system 3 and the assumed number of inquiries to the AI service system 3.

The usage AI determination processing unit 26 determines the AI service system 3 to be used as a result of the accuracy evaluation processing of each AI service system 3 in the accuracy evaluation processing unit 23. The usage AI determination processing unit 26 determines the AI service system 3 whose accuracy information is equal to or higher than a predetermined reference value and whose charge is the cheapest based on the frequency of use. If all the AI service systems 3 are below a predetermined reference value, the AI service system 3 having the highest accuracy information is determined. The predetermined reference value can be, for example, 95%, but can be appropriately set depending on the purpose of using the AI service system 3, the required criteria, and the like. The usage AI determination processing unit 26 may specify the charge by accepting the input of the information of the number of inquiries expected in the AI service system 3.

The AI switching processing unit 27 includes each AI service system driver 27a that arranges the format for each AI service system 3 and performs input / output, and a driver switching processing unit 27b that switches the AI service system driver 27a to be used. That is, the AI switching processing unit 27 includes a driver 27a that arranges input / output data in a format corresponding to each AI service system 3, and the driver switching processing unit 27b switches the input / output processing via the driver 27a. .. The relationship between the AI switching processing unit 27 and the AI service system 3 is schematically shown in a block diagram in FIG.

The AI service inquiry processing unit 28 inputs inquiry data to the AI service system 3 that is normally used, which is determined by the usage AI determination processing unit 26, and receives the answer data. This response data is associated with information on the degree of certainty indicating the reliability of the response data by the AI service system 3. The certainty is preferably numerical data, but is not limited to it.

Then, the AI service inquiry processing unit 28, when the certainty degree satisfies the predetermined condition based on the certainty degree information of the answer data received from the AI service system 3 normally used, the AI service inquiry processing unit 28. The answer data is displayed as it is as an output result. On the other hand, when the certainty does not satisfy the predetermined condition, the AI service inquiry processing unit 28 sends the same inquiry data to the AI service system 3 other than the AI service system 3 which is normally used. Input and accept the response data from each AI service system 3. Then, the answer data to be output is determined and displayed based on the answer data received from each AI service system 3 and / or the certainty. The condition for the certainty is whether or not the certainty is above a certain threshold. That is, if the certainty of the answer data received from the AI service system 3 normally used is equal to or higher than the threshold value, it is displayed as it is, and if it is less than a certain threshold value, the same inquiry data is sent to another AI service system 3. Is input, and the response data from each AI service system 3 is accepted. If the certainty of the answer data received from the AI service system 3 normally used is greater than the threshold value, it is displayed as it is, and if it is below a certain threshold value, the same inquiry data is sent to another AI service system 3. May be entered to accept response data from each AI service system 3.

As a method of determining the answer data to be output based on the answer data received from each AI service system 3 and / or the certainty, for example, there is a method of majority voting. The AI service inquiry processing unit 28 determines the AI service system 3C as the AI service system 3 that is normally used, and the AI service inquiry processing unit 28 uses the image data shown in FIG. 10 as inquiry data to determine the AI service system 3C. Suppose you typed in. Further, it is assumed that the certainty of the threshold value is 0.95. The certainty that serves as the threshold value may be, for example, the same as the reference value to be compared with the accuracy information by the utilization AI determination processing unit 26, or any numerical value higher or lower than that may be used.

It is assumed that the AI service inquiry processing unit 28 inputs the image data of FIG. 10 as inquiry data into the AI service system 3C, the answer data is "killer wheel", and the certainty is 0.71. Then, since the certainty is less than 0.95, the AI service inquiry processing unit 28 inputs the image data shown in FIG. 10 as inquiry data to the AI service systems 3A and 3B other than the AI service system 3C. At the time of this input, the driver switching processing unit 27b of the AI switching processing unit 27 performs the AI service system A driver and the AI service in the AI service system driver 27a at the time of inputting to the AI service systems 3A and 3B. After switching each driver to the system B driver, input the inquiry data.

Then, the AI service system 3A accepts "whale" as the answer data with the certainty, and the AI service system 3B accepts the "killer whale" as the answer data with the certainty. Then, the AI service inquiry processing unit 28 has 2 votes (2) for "killer wheel" and 1 vote (1) for "whale" among the answers received from the AI service systems 3A to 3C, so that they are the same. "Killer wall", which is the most abundant answer data among the answer data of, is determined as the final answer data.

As described above, the AI service inquiry processing unit 28 may determine the answer data having a large number of matching answer data among the answer data received from each AI service system 3A to 3C as the final answer data. Instead, the certainty of the response data received from each AI service system 3 may be considered. For example, it is assumed that the AI service system 3A accepts 0.4 as a certainty level in addition to the answer data "whale", and the AI service system 3B accepts 0.82 as a certainty level in addition to the answer data "killer whale". .. In this case, the number of votes (= 1 vote x 0.71 + 1 vote x 0.82) for "killer why" and 0.4 (= 1 vote x 0.4) for "whale" (= 1 vote x 0.4). Confidence may be added to the number of response data). Furthermore, the average value of the degree of certainty may be taken. For example, 0.765 (= (1 vote x 0.71 + 1 vote x 0.82) / 2 votes) for "killer wheel" and 0.4 (for "whale"). = (1 vote x 0.4) ÷ 1 vote) may be calculated and determined.

Furthermore, from the AI service system 3A, in addition to the answer data "whale", the certainty level is 0.8, and from the AI service system 3B, the answer data "killer whale" is the certainty level 0.4, and the "whale" is the certainty level 0. It is assumed that 0.4 has been accepted. In this case, for example, since each vote is 2 votes each, the answer data including the AI service system 3C used at normal times can be used as the final result. In addition, "whale" is given as 1.11 votes (= 1 x 0.71 + 1 x 0.4) for "killer wheel" and 1.2 votes (= 1 x 0.8 + 1 x 0.4) for "whale". It may be used as answer data.

Furthermore, when a plurality of response data are included, the value of the number of votes may be divided by the number of response data. In the above case, for example, 1.5 votes (= 1 vote + 0.5 votes) for "killer why" and 1.5 votes (= 1 vote + 0.5 votes) for "whale" may be used as the same number. The processing, that is, the answer data including the answer data of the AI service system 3 at the time of normal use can be determined as the final result.

In addition, by adding certainty to the number of votes divided by the number of response data, 0.91 votes (= 1 vote x 0.71 + 0.5 votes x 0.4) for "killer whale" and 1 for "whale". You may decide "where" as the final result with 0.0 votes (= 1 vote x 0.8 + 0.5 votes x 0.4).

Further, among the response data received from the AI service systems 3A to 3C, the response data from the AI service system 3 with the highest certainty may be determined as the final result.

When the AI service inquiry processing unit 28 makes an inquiry to the AI service system 3 other than the AI service system 3 normally used, the AI having accuracy information equal to or higher than the predetermined reference value set by the usage AI determination processing unit 26. An inquiry may be made only to the service system 3, an inquiry may be made to the AI service system 3 which is lower or higher accuracy information than the above reference value, and the AI switching processing unit 27 may make an inquiry. You may make inquiries to all of the switchable AI service systems 3. Further, an inquiry destination is set in advance, and the AI service system 3 of the set inquiry destination may be inquired.

The above determination method is merely an example, and any method can be used.

Next, an example of the processing process of the AI service utilization support system 1 of the present invention will be described with reference to the flowcharts of FIGS. 3 and 4.

First, a case where accuracy evaluation processing is performed for each AI service system 3 will be described. The accuracy evaluation process is preferably performed on a regular basis, for example, every three months. The test data used in the accuracy evaluation process may be the same each time, or may be part or all of the test data changed.

The person in charge of the AI service use support system 1 creates a set of 1000 inquiry data and answer data as test data, and sends the set from the person in charge terminal 4 to the management terminal 2. The test data input reception processing unit 21 of the management terminal 2 receives test data input from the person in charge terminal 4, attaches test data identification information to each test data, and stores the test data in the test data storage unit 22 (S100). ..

Then, the person in charge performs an operation of executing the accuracy evaluation process by performing a predetermined operation from the person in charge terminal 4. The accuracy evaluation processing unit 23 of the management terminal 2 that has received such an execution operation from the person in charge terminal 4 executes the accuracy evaluation processing for each AI service system 3 based on the test data stored in the test data storage unit 22 (S110). ).

First, the accuracy evaluation processing unit 23 extracts 1000 test data to be stored in the test data storage unit 22. Then, when the accuracy evaluation processing unit 23 passes a control instruction for executing the accuracy evaluation processing of the AI service system 3A to the AI switching processing unit 27, the driver switching processing unit 27b in the AI switching processing unit 27 receives the AI service system. Switch to the AI service system 3A driver in the driver 27a. Then, the accuracy evaluation processing unit 23 sends the inquiry data of the test data to the AI service system 3A via the AI service system 3A driver of the AI service system driver 27a in the AI switching processing unit 27.

The AI service system 3A executes analysis processing for the received inquiry data, returns the response data to the management terminal 2, and receives the response data from the accuracy evaluation processing unit 23 via the AI service system 3A driver of the management terminal 2. Then, the accuracy evaluation processing unit 23 refers to the correct answer data corresponding to the inquiry data stored in the test data storage unit 22, and calculates whether or not the answer data is the correct answer or the correct answer value.

When the process for one inquiry data is executed, the next test data is sent to the AI service system 3A via the driver for the AI service system 3A.

In this way, the accuracy evaluation processing unit 23 sends inquiry data of 1000 test data to the AI service system 3A, and calculates whether or not the answer is correct or the correct answer value. Then, the correct answer rate for each type of analysis target in the AI service system 3A or the AI service system 3A is calculated.

The accuracy rate calculated by the accuracy evaluation processing unit 23 is stored in the AI service information storage unit 25 as accuracy information (S120).

When the accuracy evaluation processing for the AI service system 3A is completed, the accuracy evaluation processing unit 23 passes a control instruction for executing the accuracy evaluation processing for the AI service system 3B to the AI switching processing unit 27. The driver switching processing unit 27b in the AI switching processing unit 27 switches to the driver for the AI service system 3B in the driver 27a for the AI service system. Then, the accuracy evaluation processing unit 23 sends the inquiry data of the test data to the AI service system 3B via the AI service system 3B driver of the AI service system driver 27a in the AI switching processing unit 27.

Similar to the case of the AI service system 3A, the AI service system 3B executes an analysis process for the received inquiry data, returns the answer data to the management terminal 2, and then returns the response data to the management terminal 2 via the driver for the AI service system 3B of the management terminal 2. The accuracy evaluation processing unit 23 receives the data. Then, the accuracy evaluation processing unit 23 refers to the correct answer data corresponding to the inquiry data stored in the test data storage unit 22, and calculates whether or not the answer data is the correct answer or the correct answer value.

When the process for one inquiry data is executed, the next test data is sent to the AI service system 3B via the AI service system 3B driver.

In this way, the accuracy evaluation processing unit 23 sends inquiry data of 1000 test data to the AI service system 3B, and calculates whether or not the answer is correct or the correct answer value. Then, the correct answer rate for each type of analysis target in the AI service system 3B or the AI service system 3B is calculated.

The accuracy rate calculated by the accuracy evaluation processing unit 23 is stored in the AI service information storage unit 25 as accuracy information.

By executing the above processing, the accuracy information for each AI service system 3 can be stored in the AI service information storage unit 25.

Then, the charge information input reception processing unit 24 receives input of charge information when using the AI service system 3 from the person in charge terminal 4 or the AI service system 3 (S130), and stores it in the AI service information storage unit 25. (S140).

By executing the above processing, the accuracy evaluation processing of each AI service system 3 can be executed.

Next, the process of determining the AI service system 3 to be used will be described. The determination process of the AI service system 3 is preferably performed periodically, for example, every three months.

The person in charge of the AI service use support system 1 performs an operation of executing the determination process of the AI service system 3 by performing a predetermined operation from the person in charge terminal 4. The use AI determination processing unit 26 of the management terminal 2 that has received such an execution operation from the person in charge terminal 4 is based on the accuracy information, unit price, and usage frequency information of each AI service system 3 stored in the AI service information storage unit 25. Then, the AI service system 3 to be used is determined (S200).

That is, the usage AI determination processing unit 26 refers to the accuracy information of each AI service system 3 stored in the AI service information storage unit 25, and identifies the AI service system 3 which is equal to or higher than a predetermined reference value. For example, when the AI service information storage unit 25 is shown in FIG. 8 and the reference value is 95%, the AI service system 3A (98%) and the AI service system 3C (95%) are specified.

Then, the usage AI determination processing unit 26 identifies the AI service system 3 having the lowest charge based on the charge information based on the usage frequency information. The frequency of use is "1,000,000", and the charge in that case is "2,220,000 yen" for the AI service system 3A and "1,020,000 yen" for the AI service system 3C. It is determined as the AI service system 3 that uses the service system 3C.

If the AI service system 3 to be used is specified based on the accuracy information and the charge information, the usage AI determination processing unit 26 can use a determination method other than the above.

Then, since the use AI determination processing unit 26 determines the AI service system 3C as the AI service system 3 to be used, it passes a control instruction for controlling the AI service system 3C to be used to the AI switching processing unit 27. The driver switching processing unit 27b in the AI switching processing unit 27 switches to the driver for the AI service system 3C in the driver 27a for the AI service system.

By executing the above processing, it is possible to set the AI service system 3C to be used as the AI service system 3 that is normally used.

Next, when the AI service system 3C is set as the AI service system 3 to be used, the case where the AI service system 3C is used in the normal state will be described.

The AI service inquiry processing unit 28 inputs the image data of FIG. 10 as inquiry data to the AI service system 3C which is normally used (S300). In response to this input, the AI service system 3C accepts "killer wheel" as the response data and 0.71 as the certainty (S310). Since the received certainty is 0.71, the predetermined condition is not satisfied (S320), so that the AI service inquiry processing unit 28 inquires of the AI service inquiry processing unit 27 to the AI service system 3A. When a control instruction for processing is passed, the driver switching processing unit 27b in the AI switching processing unit 27 switches to the AI service system 3A driver in the AI service system driver 27a. Then, the AI service inquiry processing unit 28 sends the image data of FIG. 10 as inquiry data to the AI service system 3A via the AI service system 3A driver of the AI service system driver 27a in the AI switching processing unit 27 (). S330).

The AI service system 3A executes an analysis process for the inquiry data, returns the answer data and the certainty to the management terminal 2, and receives the response data and the certainty level from the AI service inquiry processing unit 28 via the driver for the AI service system 3A of the management terminal 2. ..

Further, when the AI service inquiry processing unit 28 passes a control instruction for performing inquiry processing to the AI service system 3B to the AI switching processing unit 27, the driver switching processing unit 27b in the AI switching processing unit 27 is for the AI service system. Switch to the AI service system 3B driver in the driver 27a. Then, the AI service inquiry processing unit 28 sends the image data of FIG. 10 as inquiry data to the AI service system 3B via the AI service system 3B driver of the AI service system driver 27a in the AI switching processing unit 27 ( S330).

The AI service system 3B executes an analysis process for the inquiry data, and when the answer data and the certainty are returned to the management terminal 2, the AI service inquiry processing unit 28 receives the response data via the driver for the AI service system 3B of the management terminal 2. ..

As described above, when the response data and the certainty are received from the AI service system other than the AI service system 3C set as the AI service system 3 to be used in the normal time, the AI service inquiry processing unit 28 receives each AI service. Based on the response data and certainty received from the systems 3A to 3C, for example, the most frequent answer data is determined as the final result (S340), and the answer is output (S350).

After the output of the final answer is completed, the AI service inquiry processing unit 28 passes a control instruction to the AI switching processing unit 27 to return to the AI service system 3C used at normal times, and the AI switching processing unit 27 switches the driver. The processing unit 27b switches to the AI service system 3C driver in the AI service system driver 27a.

By performing the above processing, it is possible to reduce the cost while maintaining the reliability of the response data obtained when using the AI service system.

By using the AI service use support system 1 of the present invention, it is possible to secure the reliability of a certain amount of response data while suppressing the cost.

1: AI service usage support system 2: Management terminal 3: AI service system 4: Person in charge terminal 21: Test data input reception processing unit 22: Test data storage unit 23: Accuracy evaluation processing unit 24: Charge information input reception processing unit 25 : AI service information storage unit 26: Usage AI determination processing unit 27: AI switching processing unit 27a: AI service system driver 27b: Driver switching processing unit 28: AI service inquiry processing unit 70: Arithmetic device 71: Storage device 72: Display Device 73: Input device 74: Communication device

Claims (6)

It is an AI service usage support system that supports the use of an AI service system that returns answer data and certainty for one inquiry data.
The AI service usage support system has an AI service inquiry processing unit and has an AI service inquiry processing unit.
The AI service inquiry processing unit
By sending inquiry data to the AI service system set for normal use, answer data (hereinafter, "normal time answer data") and certainty (hereinafter, "normal time certainty") are accepted. ，
If the received normal time certainty does not satisfy the condition, the same inquiry data as the inquiry data is sent to each of the plurality of AI service systems other than the AI service system set for normal use. feed Ri, answer data (hereinafter, "auxiliary answer data") with respect to the inquiry data accept,
AI service usage support characterized in that the largest number of answer data among the same answer data among the normal answer data and the auxiliary answer data is determined as the answer data to be output as a result. system.
It is an AI service usage support system that supports the use of an AI service system that returns answer data and certainty for one inquiry data.
The AI service usage support system has an AI service inquiry processing unit and has an AI service inquiry processing unit.
The AI service inquiry processing unit
By sending inquiry data to the AI service system set for normal use, answer data (hereinafter, "normal time answer data") and certainty (hereinafter, "normal time certainty") are accepted. ，
If the received normal time certainty does not satisfy the condition, the same inquiry data as the inquiry data is sent to each of the plurality of AI service systems other than the AI service system set for normal use. feed Ri, reply data (hereinafter, "auxiliary answer data") with respect to said inquiry data and confidence (hereinafter, "auxiliary confidence") accept from each AI service system,
Each of the supplementary response data is regarded as one vote, and each vote is obtained by adding the supplementary certainty corresponding to each of the supplementary response data (hereinafter, "weighted supplementary vote"), and the above. One vote is the normal answer data, and the same answer data is obtained from the votes obtained by adding the normal time certainty corresponding to the normal time answer data (hereinafter, "weighted normal time vote"). An AI service usage support system characterized in that the answer data with the largest number of votes is determined as the answer data to be output as a result.
The AI service usage support system further
The accuracy evaluation processing unit that accepts answer data by sending inquiry data to the AI service system to be evaluated and calculates the accuracy information of the AI service system using the correct answer data corresponding to the inquiry data and the answer data. ，
AI service support system according to any one of claims 2 to claim 1, characterized in that it comprises a.
The AI service usage support system further
An AI service information storage unit that stores accuracy information and charge information for each AI service system,
Based on the accuracy information and charge information stored in the AI service information storage unit, the usage AI determination processing unit that determines the AI service system to be used at normal times, and the usage AI determination processing unit.
The AI service use support system according to any one of claims 1 to 3 , wherein the AI service is used.
The AI service usage support system further
It has an AI switching processing unit that has a driver for each AI service system and a driver switching processing unit that switches the driver.
In the normal time, the driver corresponding to the AI service system used in the normal time is used to execute the input / output process with the AI service system used in the normal time.
If the certainty received from the AI service system used at normal times does not satisfy the conditions, the driver switching processing unit switches the driver and executes input / output processing for each AI service system that sends the inquiry data. ，
AI service support system according to any one of claims 1 to 4, characterized in that.
Computer,
By sending inquiry data to the AI service system set for normal use, answer data and certainty are accepted, and if the certainty does not satisfy the conditions, set for normal use. It is characterized in that it functions as an AI service inquiry processing unit that sends the inquiry data to an AI service system other than the AI service system that has been created .
The AI service inquiry processing unit
By sending inquiry data to the AI service system set for normal use, answer data (hereinafter, "normal time answer data") and certainty (hereinafter, "normal time certainty") are accepted. ，
If the received normal time certainty does not satisfy the condition, the same inquiry data as the inquiry data is sent to each of the plurality of AI service systems other than the AI service system set for normal use. Send, accept the answer data for the inquiry data (hereinafter, "auxiliary answer data"),
Of the normal response data and the auxiliary response data, the largest number of response data among the same response data is determined as the response data to be output as a result.
An AI service usage support program characterized by this.

Images