JP7448881B2 - Data linkage system and API platform - Google Patents

Description

The present invention relates to a data collaboration system and an API platform that collect and accumulate data held by multiple information systems.

BACKGROUND ART Conventionally, a system is known that provides an API (Application Program Interface) that obtains the execution result of a predetermined algorithm corresponding to data linkage processing between SaaS (Software as a Service) (for example, see Patent Document 1). .

Japanese Patent Application Publication No. 2016-224578

However, Patent Document 1 does not describe behavior depending on the load of processing executed in response to a request to use an API.

Therefore, an object of the present invention is to provide a data linkage system and an API platform that can behave according to the load of processing executed in response to a request for use of an API.

The data linkage system of the present invention includes a data collection system that collects data held by an information system, a data storage system that stores data collected by the data collection system, and data stored by the data storage system. an API platform that provides an API (Application Program Interface) for acquiring data based on the data from the data accumulation system, and the API platform acquires data from the data accumulation system in response to a request for use of the API. The capacity is increased when the number of unexecuted processes for acquiring the process satisfies a specific condition for increasing the capacity of the process.

With this configuration, the data linkage system of the present invention can be used when the number of unexecuted processes for acquiring data from a data storage system in response to a request for using an API satisfies a specific condition for increasing the capacity of this process. In addition, since the processing capacity is increased, it is possible to behave according to the load of the processing executed in response to the request for using the API.

In the data linkage system of the present invention, the API platform may reduce the capacity when the number satisfies a specific condition for reducing the capacity.

With this configuration, the data linkage system of the present invention can be used when the number of unexecuted processes for acquiring data from a data storage system in response to a request for using an API satisfies a specific condition that reduces the capacity of this process. In addition, since the capacity of this processing is reduced, it is possible to behave according to the load of processing executed in response to a request for use of the API.

In the data linkage system of the present invention, the API platform includes a predicted processing time level indicating a degree of predicted processing time, which is the time predicted to take the process, and data predicted to respond to the request. The ability may be changed for each classification of the processing based on at least one of a predicted response data amount level indicating the extent of the predicted response data amount.

With this configuration, the data linkage system of the present invention can classify data based on at least one of the predicted processing time level and the predicted response data amount level of the process of acquiring data from the data storage system in response to a request for the use of an API. Since the capability of this process is changed, this process can be executed by an appropriate server according to at least one of a predicted processing time level and a predicted response data amount level of this process. Therefore, the data linkage system of the present invention can behave according to the load of processing executed in response to a request for use of the API.

The API platform of the present invention acquires data based on data accumulated by a data accumulation system that accumulates data collected by a data collection system that collects data held by an information system. An API platform that provides an API (Application Program Interface) for use of the API, wherein the number of unexecuted processes for acquiring data from the data storage system in response to a request for use of the API increases the capacity of the process. The feature is that the ability is increased when specific conditions for increasing the ability are met.

With this configuration, the API platform of the present invention can be used when the number of unexecuted processes for acquiring data from a data storage system in response to a request for using the API satisfies a specific condition for increasing the capacity of this process. , increases the capacity of this processing, so that it can behave according to the load of the processing executed in response to the request for the use of the API.

The data linkage system and API platform of the present invention can behave according to the processing load to be executed in response to a request for using the API.

FIG. 1 is a block diagram of a system according to an embodiment of the present invention. 2 is a block diagram of the API platform shown in FIG. 1. FIG. 3 is a diagram showing an example of a pattern of combinations of a predicted processing time level and a predicted response data amount level acquired by the API controller shown in FIG. 2. FIG. 3 is a diagram illustrating an example of the correspondence between a queue ID, which is the identification information of the queue illustrated in FIG. 2, and a pattern of combinations of a predicted processing time level and a predicted response data amount level. FIG. FIG. 2 is a sequence diagram schematically illustrating the operation of the system shown in FIG. 1 when responding to a request to use an API from an external system. FIG. 2 is a sequence diagram of the operation of the system shown in FIG. 1 when identifying a queue that stores a processing request message. FIG. 2 is a sequence diagram of the operation of the system shown in FIG. 1 when performing machine learning on the correlation between the contents of a request for API usage from an external system to an API platform, processing time, and response data amount. 3 is a flowchart of the operation of the API controller shown in FIG. 2 when increasing the number of back ends. 3 is a flowchart of the operation of the API controller shown in FIG. 2 when the number of backends is reduced.

Embodiments of the present invention will be described below with reference to the drawings.

First, the configuration of a system according to an embodiment of the present invention will be described.

FIG. 1 is a block diagram of a system 10 according to this embodiment.

As shown in FIG. 1, the system 10 includes a data source unit 20 that generates data, and a data linkage system 30 that links the data generated by the data source unit 20.

The data source section 20 includes an information system 21 that generates data. The information system 21 includes a configuration management server 21a that stores the configuration and settings of the information system 21. The data source unit 20 may include at least one information system other than the information system 21. Examples of information systems include IoT (Internet of Things) systems such as remote management systems that remotely manage image forming devices such as MFPs (Multifunction Peripherals) and dedicated printers, ERP (Enterprise Resource Planning), and production management systems. There are internal systems such as Each of the information systems may be configured by one computer or multiple computers. Each of the information systems may be built on a public cloud. The information system may maintain files of structured data. An information system may maintain files of unstructured data. The information system may maintain a database of structured data.

The data source unit 20 serves as a data collection system that acquires structured data or unstructured data files held by the information system and sends the acquired files to a pipeline described below of the data linkage system 30. It is equipped with a POST connector 22. The data source unit 20 may include at least one POST connector having the same configuration as the POST connector 22 in addition to the POST connector 22. The POST connector may be configured by a computer that constitutes an information system from which the POST connector itself retrieves files. Note that the POST connector is also a component of the data linkage system 30.

The data source unit 20 serves as a data collection system that acquires structured data from a database of structured data held by the information system and sends the acquired structured data to a pipeline described below of the data linkage system 30. It is equipped with a POST agent 23. The data source unit 20 may include at least one POST agent having the same configuration as the POST agent 23 in addition to the POST agent 23. The POST agent may be configured by a computer that configures an information system from which the POST agent itself acquires structured data. Note that the POST agent is also a component of the data coordination system 30.

The data source unit 20 includes a GET agent 24 as a data collection system that generates structured data for collaboration based on data held by the information system. The data source unit 20 may include at least one GET agent having the same configuration as the GET agent 24 in addition to the GET agent 24 . The GET agent may be configured by a computer that configures an information system that holds data that is the source of generation of structured data for collaboration. Note that the GET agent is also a component of the data linkage system 30.

The data cooperation system 30 includes a data storage system 40 that stores data generated by the data source section 20, an application section 50 that uses data stored in the data storage system 40, and a data storage system 40 and an application section 50 that use the data stored in the data storage system 40. and a control service unit 60 that executes various types of control.

The data storage system 40 includes a pipeline 41 that stores data generated by the data source unit 20. The data storage system 40 may include at least one pipeline other than the pipeline 41. Since the configuration of data in an information system may differ from one information system to another, the data storage system 40 basically includes a pipeline for each information system. Each of the pipelines may be configured by one computer or multiple computers.

The data storage system 40 includes a GET connector 42 as a data collection system that acquires structured data or unstructured data files held by the information system and links the acquired files to the pipeline. . The data storage system 40 may include at least one GET connector having the same configuration as the GET connector 42 in addition to the GET connector 42. The GET connector may be configured by a computer that configures a pipeline in which the GET connector itself links files.

Note that the system 10 includes a POST connector in the data source unit 20 for information systems that do not support acquiring structured data or unstructured data files from the data storage system 40 side. On the other hand, the system 10 includes a GET connector in the data storage system 40 for information systems that support the acquisition of structured data or unstructured data files from the data storage system 40 side.

The data storage system 40 includes a GET agent 43 as a data collection system that acquires structured data generated by the GET agent and links the acquired structured data to a pipeline. The data storage system 40 may include at least one GET agent having the same configuration as the GET agent 43 in addition to the GET agent 43. The GET agent may be configured by a computer that configures a pipeline in which the GET agent itself links structured data.

Note that the system 10 includes a POST agent in the data source unit 20 for information systems that do not support structured data being acquired from the data storage system 40 side. On the other hand, for an information system that supports structured data being acquired from the data storage system 40 side, the system 10 includes a GET agent in the data source unit 20 and a GET agent in the data storage system 40. Equipped with an agent.

The data accumulation system 40 performs a final conversion process as a data conversion process to convert data accumulated by multiple pipelines into a form that can be searched and aggregated using a query language such as a database language such as SQL. It is equipped with a big data analysis section 44 as a system. The big data analysis unit 44 can also perform a search or aggregation on the data that has undergone the final conversion process in response to a search request or aggregation request from the application unit 50 side. The big data analysis unit 44 may be configured by one computer or multiple computers.

The final conversion process may include a data integration process that integrates data from a plurality of information systems as a data conversion process. A remote management system is located in Asia to remotely manage a large number of image forming devices located in Asia, and a remote management system is located in Europe to remotely manage a large number of image forming devices located in Europe. If the system 10 includes, as an information system, a remote management system located in the United States and a remote management system located in the United States for remotely managing a large number of image forming apparatuses located in the United States, these three Each of the two remote management systems includes a device management table for managing the image forming apparatuses that the remote management system itself manages. The device management table is information indicating various types of information about the image forming apparatus in association with an ID assigned to each image forming apparatus. Here, since each of the three remote management systems has its own device management table, there is a possibility that the same ID is assigned to different image forming apparatuses in the device management tables of the three remote management systems. There is. Therefore, when the big data analysis unit 44 integrates the device management tables of three remote management systems to generate one device management table, the big data analysis unit 44 reassigns the IDs of the image forming apparatuses so that no duplication occurs.

The application unit 50 includes an application service 51 that uses the data managed by the big data analysis unit 44 to perform specific operations instructed by the user, such as data display and data analysis. There is. The application unit 50 may include at least one application service other than the application service 51. Each of the application services may be configured by one computer or multiple computers. Application services include, for example, BI (Business Intelligence) tools, SaaS (Software as a Service) servers, and the like.

The application unit 50 includes an API platform 52 that provides an API (Application Program Interface) that executes specific operations using data managed by the big data analysis unit 44. The API platform 52 may be configured by one computer or multiple computers. The API provided by the API platform 52 may be called from a system external to the system 10, such as a BI tool or a SaaS server, or may be called from an application service of the application unit 50. The API provided by the API platform 52 is an API for acquiring data from the data storage system 40 based on data stored by the data storage system 40. For example, the API provided by the API platform 52 is an API provided to a consumables ordering system external to the system 10 that orders consumables when the remaining amount of consumables such as toner in an image forming apparatus is below a certain amount. , an API that sends data on the remaining amount of consumables collected from the image forming apparatus by the remote management system, and a failure prediction system external to the system 10 that predicts failures of the image forming apparatus. An API for transmitting various data collected from the apparatus, an API for transmitting counter information indicating the number of printed sheets collected from the image forming apparatus by a remote management system to a system external to the system 10, and use of the system 10. There is an API that sends data indicating user usage status to a system outside the system 10, and an API that accepts a search query to obtain arbitrary data based on data managed by the system 10. .

Hereinafter, a system external to the API platform 52 will be simply referred to as an external system. The external system includes a system external to the system 10 and an application service of the application unit 50.

FIG. 2 is a block diagram of API platform 52.

As shown in FIG. 2, the API platform 52 includes a front end unit 70 that receives requests for API usage from external systems, and a data creation unit 80 that creates response data in response to requests received by the front end unit 70. It is equipped with

The front end unit 70 includes an LB (Load Balancer) 71 that is an end point exposed to an external system, and a front end Proxy 72 that establishes a connection with the external system. In addition to the front end Proxy 72, the front end unit 70 may include at least one front end Proxy having the same configuration as the front end Proxy 72.

The LB 71 notifies one of the front end proxies of a request to use the API from an external system using a round robin method.

The front end Proxy 72 notifies one of the API servers (described later) of the request to use the API notified from the LB 71 using a round robin method.

The data creation unit 80 includes a data creation unit instance 81. In addition to the data creation unit instance 81, the data creation unit 80 may include at least one data creation unit instance having the same configuration as the data creation unit instance 81.

The data creation unit instance 81 includes an API server 82 that provides an API for executing specific operations using data managed by the big data analysis unit 44 (see FIG. 1). In addition to the API server 82, the data creation unit instance 81 may include at least one API server having the same configuration as the API server 82.

The data creation unit instance 81 has a predicted processing time that indicates the amount of time (hereinafter referred to as "predicted processing time") that is predicted to take to process data acquisition in response to a request to use an API from an external system. control service unit, and a predicted response data volume level indicating the amount of data expected to respond to a request for API usage from an external system (hereinafter referred to as "predicted response data volume"). 60 (see FIG. 1).

FIG. 3 is a diagram showing an example of a pattern of combinations of the predicted processing time level and the predicted response data amount level acquired by the API controller 83.

In the example shown in FIG. 3, the predicted processing time levels are "normal" indicating that the predicted processing time is within a specific range, and "long" indicating that the predicted processing time is longer than the upper limit of the specific range. It includes "short" indicating that the predicted processing time is shorter than the lower limit of the specific range. The predicted response data amount level is "normal" indicating that the amount of predicted response data is within a specific range, "high" indicating that the amount of predicted response data is greater than the upper limit of the specific range, and the amount of predicted response data. contains the word "less" to indicate that it is less than the lower limit of a specific range. In the example shown in FIG. 3, there are nine combination patterns of the predicted processing time level and the predicted response data amount level, pattern A to pattern I.

As shown in FIG. 2, the data creation unit instance 81 includes a queue 84 that stores a processing request message indicating a request for processing according to an API requested to be used by an external system. In addition to the queue 84, the data creation unit instance 81 may include at least one queue having a similar configuration to the queue 84. A queue is provided for each classification based on a pattern of combinations of predicted processing time level and predicted response data amount level.

FIG. 4 is a diagram showing an example of the correspondence between a queue ID, which is queue identification information, and a combination pattern of a predicted processing time level and a predicted response data amount level.

As shown in FIG. 4, the classification based on the pattern of the combination of the predicted processing time level and the predicted response data amount level may be such that, for example, pattern A and pattern D are classified as the same, or pattern H and pattern I are classified as the same. It can also be classified. Note that FIG. 4 is drawn with information about some queues omitted.

As shown in FIG. 2, the data creation unit instance 81 is a server instance that sends a processing request indicated by a processing request message stored in the queue 84 to the big data analysis unit 44 (see FIG. 1). It is equipped with an end 85. In addition to the back end 85, the data creation unit instance 81 may include at least one back end having the same configuration as the back end 85. A backend is provided for each queue. A plurality of back ends may be provided for one queue. The processing request messages stored in the queue are taken out from this queue in order by back ends that are capable of processing processing request messages among the back ends associated with this queue.

As shown in FIG. 1, the control service unit 60 includes a pipeline orchestrator 61 as a process monitoring system that monitors each stage of data processing in the data source unit 20, data storage system 40, and application unit 50. There is. The pipeline orchestrator 61 may be configured by one computer or multiple computers.

For each API usage request from an external system to the API platform 52, the pipeline orchestrator 61 calculates the time required to obtain data in response to this request (hereinafter referred to as "processing time"). , and the amount of data in response to this request (hereinafter referred to as "response data amount") are stored as a history. Then, the pipeline orchestrator 61 performs machine learning on the correlation between the content of the request to use the API from the external system to the API platform 52, the processing time, and the amount of response data. It is possible to determine the predicted processing time level and predicted response data amount level with respect to the content of the request for API usage.

The control service unit 60 includes a configuration management server 62 that stores the configuration and settings of the data storage system 40 and automatically executes deployment as needed. The configuration management server 62 may be configured by one computer, or may be configured by multiple computers. The configuration management server 62 constitutes a configuration change system that changes the configuration of the data linkage system 30.

The control service unit 60 is configured to connect to the configuration management server of the information system and collect information for detecting configuration changes related to databases and unstructured data in the information system, that is, changes in the data configuration in the information system. A management gateway 63 is provided. The configuration management gateway 63 may be configured by one computer, or may be configured by multiple computers.

The control service unit 60 includes a key management service 64 that encrypts and stores security information such as key information and connection strings necessary for linking systems such as information systems. The Key management service 64 may be configured by one computer or multiple computers.

The control service unit 60 includes a management API 65 that accepts requests from the data storage system 40 and the application unit 50. The management API 65 may be configured by one computer or multiple computers.

The control service unit 60 includes an authentication and authorization service 66 that performs authentication and authorization of application services of the application unit 50. The authentication and authorization service 66 may be configured by one computer or multiple computers. The authentication and authorization service 66 can, for example, check whether the application service is permitted to request an update of the information system data stored in the data storage system 40.

Next, the operation of the system 10 when responding to a request for API usage from an external system will be described.

FIG. 5 is a sequence diagram schematically illustrating the operation of the system 10 when responding to a request for API usage from the external system 90.

As shown in FIG. 5, when executing a request to use a specific API, the external system 90 requests the LB 71 of the front end unit 70 to establish a TCP (Transmission Control Protocol)/IP (Internet Protocol) connection ( S101). Here, the request to use a specific API is executed using, for example, HTTPS (Hypertext Transfer Protocol Secure). When the LB 71 receives the request in S101, it identifies the front end Proxy to which the request in S101 is to be notified using a round robin method, and notifies the identified front end Proxy of the request in S101. In the following, in order to simplify the explanation, it is assumed that the front end Proxy to which the request in S101 is notified from the LB 71 is the front end Proxy 72.

When the front end Proxy 72 is notified of the request in S101 from the LB 71, it establishes a TCP/IP connection with the external system 90 (S102).

When the TCP/IP connection with the system 10 is established, the external system 90 notifies the front end Proxy 72 of a request to use a specific API via the established TCP/IP connection (S103).

The front end Proxy 72 notifies one of the API servers of the request to use the API, which was notified from the external system 90 via the LB 71 in S103, using a round robin method (S104). In the following, in order to simplify the explanation, it is assumed that the API server to which the request is notified from the front end Proxy 72 in S104 is the API server 82.

Upon receiving the notification in S104, the API server 82 interprets the request notified in S104 (S105).

FIG. 6 is a sequence diagram of the operation of the system 10 when identifying a queue that stores a processing request message.

After executing the interpretation in S105, the API server 82 inquires of the API controller 83 about the predicted processing time level and predicted response data amount level of the request interpreted in S105, as shown in FIG. 6 (S121).

Upon receiving the inquiry in S121, the API controller 83 notifies the management API 65 of the control service unit 60 of the inquiry in S121 (S122).

Upon receiving the notification in S122, the management API 65 notifies the pipeline orchestrator 61 of the inquiry notified in S122 (S123).

When the pipeline orchestrator 61 receives the notification in S123, the pipeline orchestrator 61 determines the predicted processing time level and prediction for the request subject to the inquiry notified in S123, that is, the content of the request interpreted by the API server 82 in S105. The response data amount level is determined (S124).

Next, the pipeline orchestrator 61 notifies the management API 65 of the results of the determination in S124, that is, the predicted processing time level and predicted response data amount level determined in S124 (S125).

Next, the management API 65 notifies the API controller 83 of the determination result notified from the pipeline orchestrator 61 in S125 (S126).

Next, the API controller 83 replies to the API server 82 with the determination results notified from the management API 65 in S126, that is, the predicted processing time level and the predicted response data amount level (S127).

Upon receiving the response in S127, the API server 82 identifies the queue associated with the combination pattern of the predicted processing time level and predicted response data amount level that was responded in S127 (S128).

Hereinafter, in order to simplify the explanation, it is assumed that the queue identified in S128 is the queue 84.

As shown in FIG. 5, after performing the interpretation in S105, the API server 82 stores a processing request message indicating the request interpreted in S105 in the queue 84 specified by the operation shown in FIG. 6 (S106). Here, the API server 82 , as the API server that stored the processing request message to be stored in the queue 84 , includes identification information of itself in the processing request message stored in the queue 84 .

The processing request message stored in the queue 84 in S106 is taken out from the queue 84 by one of the back ends associated with the queue 84 (S107). Hereinafter, in order to simplify the explanation, it is assumed that the back end that took out the processing request message from the queue 84 in S107 is the back end 85.

When the back end 85 retrieves the processing request message from the queue 84 in S107, it transmits the processing request indicated by this processing request message to the big data analysis unit 44 (S108).

When the processing request is sent in S108, the big data analysis unit 44 executes the processing corresponding to the request sent in S108, thereby acquiring data corresponding to this processing (S109).

Next, the big data analysis unit 44 notifies the back end 85 of the data acquired in S109 (S110).

When the backend 85 is notified of the data in S110, it notifies the API server 82, which stored the processing request message retrieved from the queue 84 in S107 in the queue 84, of the data notified in S110 (S111). Note that the processing request message taken out from the queue 84 in S107 includes identification information of the API server 82 that stored this processing request message in the queue 84.

When the API server 82 is notified of the data in S111, it notifies the front end Proxy 72 of this data (S112).

Therefore, the front end Proxy 72 responds to the external system 90 via the LB 71 with the data notified in S112 (S113). Note that the front end Proxy 72 terminates the TCP/IP connection established in S102 after completing the response in S113.

Next, the operation of the system 10 when performing machine learning on the correlation between the contents of a request for API usage from an external system to the API platform 52, processing time, and response data amount will be described.

FIG. 7 is a sequence diagram of the operation of the system 10 when performing machine learning on the correlation between the contents of a request for API usage from an external system to the API platform 52, processing time, and response data amount.

When the back end 85 starts sending a processing request to the big data analysis unit 44 in S108, as shown in FIG. The API controller 83 is notified of the start (S141).

Next, when the notification of data from the big data analysis unit 44 is finished in S110, the back end 85 finishes the data acquisition process in response to the request for API usage from the external system, and in S110 The amount of data notified from the analysis unit 44 is notified to the API controller 83 (S142).

Upon receiving the notification in S142, the API controller 83 notifies the pipeline orchestrator 61 of the content of the request from the external system to the API platform 52 to use the API, as well as the processing time and response data amount for this request ( S143). Here, the API controller 83 calculates the time from the time when the start of the process is notified in S141 to the time when the end of the process is notified in S142 as the processing time. Further, the API controller 83 sets the amount of data notified in S142 as the response data amount.

When the pipeline orchestrator 61 receives the notification in S143, the pipeline orchestrator 61 checks the content of the request to use the API from the external system to the API platform 52, as well as the processing time and response data amount, which have been notified so far from the API controller 83. Machine learning is performed on the correlation (S144). Therefore, the pipeline orchestrator 61 can determine the predicted processing time level and predicted response data amount level for the content of the request for API usage from the external system to the API platform 52.

Next, the operation of the API controller 83 when increasing the number of backends will be described.

FIG. 8 is a flowchart of the operation of the API controller 83 when increasing the number of backends.

The API controller 83 executes the operations shown in FIG. 8 for each queue belonging to the data creation unit instance 81. The queue 84 will be described below as a representative example.

As shown in FIG. 8, the API controller 83 is configured to increase the number of processing request messages stored in the queue 84 in a process that increases the number of backends provided for the queue 84. It is determined whether the number of processing request messages stored in the queue 84 exceeds the increased processing number threshold (S161). Here, the increase processing number threshold is, for example, 300.

When the API controller 83 determines in S161 that the number of processing request messages stored in the queue 84 exceeds the increase processing number threshold, the API controller 83 performs processing to increase the number of backends provided for the queue 84. The increased processing time indicating the measured time is set to 0 (S162), and measurement of the increased processing time is started (S163).

After the processing in S163, the API controller 83 determines that the increased processing time is equal to or greater than the increased processing time threshold, which is the increased processing time threshold in the process of increasing the number of backends provided for the queue 84. It is determined whether or not (S164). Here, the increase processing time threshold is, for example, 5 minutes.

If the API controller 83 determines in S164 that the increased processing time is not equal to or greater than the increased processing time threshold, the API controller 83 determines whether the number of processing request messages stored in the queue 84 exceeds the increased processing number threshold. (S165).

When the API controller 83 determines in S165 that the number of processing request messages stored in the queue 84 exceeds the increased processing number threshold, it executes the process in S164.

If the API controller 83 determines in S165 that the number of processing request messages stored in the queue 84 does not exceed the increased processing number threshold, it executes the process in S161.

If the API controller 83 determines in S164 that the increased processing time is equal to or greater than the increased processing time threshold, the API controller 83 increases the number of backends provided for the queue 84 by a specific number (S166), and performs the processing in S161. Execute.

By the operation shown in FIG. 8, the API controller 83 causes the state in which the number of processing request messages stored in the queue 84 exceeds the increased processing number threshold (YES in S161 and YES in S165) to exceed the increased processing time threshold. If the time continues (YES in S164), the number of backends provided for the queue 84 is increased by a specific number (S166). For example, if the number of processing request messages stored in the queue 84 continues for more than 5 minutes, the API controller 83 specifies the number of backends provided for the queue 84. Increase by the number of .

Next, the operation of the API controller 83 when reducing the number of backends will be described.

FIG. 9 is a flowchart of the operation of the API controller 83 when reducing the number of backends.

The API controller 83 executes the operations shown in FIG. 9 for each queue belonging to the data creation unit instance 81. The queue 84 will be described below as a representative example.

As shown in FIG. 9, the API controller 83 controls the number of processing request messages stored in the queue 84 to reduce the number of processing request messages in a process that reduces the number of backends provided for the queue 84. It is determined whether the number of processing request messages stored in the queue 84 is less than or equal to the reduced processing number threshold (S181). Here, the number threshold for reduction processing is, for example, 200 pieces.

If the API controller 83 determines in S181 that the number of processing request messages stored in the queue 84 is less than or equal to the reduction processing number threshold, the API controller 83 performs processing to reduce the number of backends provided for the queue 84. The time for reduction processing indicating the measured time is set to 0 (S182), and measurement of the time for reduction processing is started (S183).

After the processing in S183, the API controller 83 determines that the reduction processing time is equal to or greater than a reduction processing time threshold, which is a threshold for reduction processing time in processing for reducing the number of backends provided for the queue 84. It is determined whether or not (S184). Here, the time threshold for reduction processing is, for example, 30 minutes.

If the API controller 83 determines in S184 that the reduction processing time is not equal to or greater than the reduction processing time threshold, the API controller 83 determines whether the number of processing request messages stored in the queue 84 is less than or equal to the reduction processing number threshold. (S185).

If the API controller 83 determines in S185 that the number of processing request messages stored in the queue 84 is less than or equal to the reduced processing number threshold, it executes the process in S184.

If the API controller 83 determines in S185 that the number of processing request messages stored in the queue 84 is not less than the reduced processing number threshold, it executes the process in S181.

If the API controller 83 determines in S184 that the reduction processing time is equal to or greater than the reduction processing time threshold, the API controller 83 reduces the number of backends provided for the queue 84 by a specific number (S186), and performs the processing in S181. Execute.

By the operation shown in FIG. 9, the API controller 83 determines whether the number of processing request messages stored in the queue 84 is equal to or less than the reduction processing number threshold (YES in S181 and YES in S185) is equal to or greater than the reduction processing time threshold. If the time continues (YES in S184), the number of backends provided for the queue 84 is reduced by a specific number (S186). For example, if the number of processing request messages stored in the queue 84 continues to be 200 or less for 30 minutes or more, the API controller 83 specifies the number of backends provided for the queue 84. decrease by the number of .

As explained above, in the data linkage system 30, the number of unexecuted processes (S108 and S110) for acquiring data from the data storage system 40 in response to a request for the use of an API increases the capacity of this process. If specific conditions are met (YES in S161, YES in S164, and YES in S165), the processing capacity is increased (S166), so the processing capacity is increased according to the processing load to be executed in response to the request for using the API. can behave.

If the number of unexecuted processes for acquiring data from the data storage system 40 in response to a request for the use of an API satisfies a specific condition that reduces the capacity of this process (YES in S181), the data linkage system 30 , YES in S184 and YES in S185), the capacity of this processing is reduced (S186), so that it can behave according to the load of processing executed in response to a request for using the API.

The data linkage system 30 classifies the processing of acquiring data from the data storage system 40 in response to a request for the use of an API based on the combination pattern of the predicted processing time level and the predicted response data amount level. Since the capacity is changed, this processing can be executed by an appropriate back end according to the combination pattern of the predicted processing time level and predicted response data amount level of this process. Therefore, the data linkage system 30 can behave according to the load of processing executed in response to a request for use of the API.

A specific processing time backend as a backend for classification based on a specific predicted processing time level and a processing time length as a backend for classification based on a predicted processing time level that has a predicted processing time longer than a specific predicted processing time level. When the API platform 52 includes a back end, the processing time length back end may have a memory capacity and/or network bandwidth greater than the specific processing time back end. With this configuration, the data linkage system 30 performs a process of acquiring data from a data storage system using memory capacity and network capacity when the predicted processing time of this process is long, compared to when the predicted processing time of this process is short. Since at least one of the backends can be executed by many backends, it can behave according to the burden of the processing to be performed in response to the request for the use of the API.

A specific response data volume backend as a backend for classification based on a specific predicted response data volume level, and a backend for classification based on a predicted response data volume level that is greater than a specific predicted response data volume level. When the API platform 52 includes a large response data volume back end as a specific response data volume back end, the large response data volume back end may have at least one of memory capacity and network bandwidth greater than the specific response data volume back end. . With this configuration, the data linkage system 30 allows the process of acquiring data from the data storage system to be performed with less memory capacity when the amount of predicted response data for this process is large than when the amount of predicted response data for this process is small. , and at least one of the network bandwidth can be executed by a large number of backends, so that it can behave according to the processing load to be executed in response to the request for the use of the API.

The data linkage system 30 includes a queue and a back end for each classification based on a pattern of combinations of predicted processing time level and predicted response data amount level. However, the data cooperation system 30 may include a queue and a back end for each classification based on either the predicted processing time level or the predicted response data amount level.

21 Information System 22 POST Connector (Data Collection System)
23 POST agent (data collection system)
24 GET agent (data collection system)
30 Data linkage system 40 Data storage system 42 GET connector (data collection system)
43 GET agent (data collection system)
52 API platform

Claims (3)

a data collection system that collects data held by the information system;
a data accumulation system that accumulates data collected by the data collection system;
an API platform that provides an API (Application Program Interface) for acquiring data based on data accumulated by the data accumulation system from the data accumulation system;
The API platform increases the capacity when the number of unexecuted processes for acquiring data from the data storage system in response to a request for use of the API satisfies a specific condition for increasing the capacity of the process. increase ,
The API platform includes a predicted processing time level indicating a predicted processing time, which is the amount of time that the process is predicted to take, and a predicted response data amount, which is the amount of data expected to respond to the request. A data linkage system characterized in that the ability is changed for each classification of the processing based on at least one of a predicted response data amount level indicating the degree of the processing . The data linkage system according to claim 1, wherein the API platform reduces the capacity when the number satisfies a specific condition for reducing the capacity. API (Application Program Interface) for acquiring data based on data accumulated by a data accumulation system that accumulates data collected by a data acquisition system that collects data held by an information system. ) is an API platform that provides
increasing the capacity when the number of unexecuted processes for acquiring data from the data storage system in response to a request for use of the API satisfies a specific condition for increasing the capacity of the process;
A predicted processing time level that indicates the predicted processing time, which is the time that the processing is predicted to take, and a prediction that indicates the predicted response data amount, which is the amount of data that is predicted to respond to the request. The API platform is characterized in that the capability is changed for each classification of the processing based on at least one of a response data amount level.

Images