JP2020154381A - Information processing system, information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing system, an information processing device, an information processing method, and a program which can suitably collect logs from a plurality of service servers.SOLUTION: An information processing system comprises: a log collection unit which acquires non-structured log information regarding a use situation of a plurality of service servers, and groups the non-structured log information, based on at least service identification information and a time stamp to store the information in a storage unit; a dispersion-treatment unit which dispersion-treats the non-structured log information stored in the storage unit; an access authority management unit which manages an access right of a user to the grouped non-structured log information stored in the storage unit; and an aggregation and analysis unit which receives designation of a reference condition when the user refers to the grouped and dispersion-treated non-structured log information.SELECTED DRAWING: Figure 2

Description

The present invention relates to information processing systems, information processing devices, information processing methods and programs.

Conventionally, technical information regarding an information processing apparatus that groups log data sequentially received with the passage of time for each set of operations set based on log fields has been disclosed (see Patent Document 1).

Japanese Patent No. 6396615

However, in the above-mentioned conventional technique, there is a possibility that the collection of logs from a plurality of service servers whose log output standards are not unified is not sufficiently considered.

The present invention has been made in consideration of such circumstances, and provides an information processing system, an information processing device, an information processing method and a program capable of suitably collecting logs from a plurality of service servers. It is one of the purposes.

One aspect of the present invention is to acquire unstructured log information regarding the usage status of a plurality of service servers, and group the unstructured log information based on at least service identification information and a time stamp in a storage unit. A user for a log collecting unit to be stored, a distributed processing unit for distributed processing of the unstructured log information stored in the storage unit, and a grouped unstructured log information stored in the storage unit. Information processing that includes an access authority management unit that manages the access rights of the information processor and an aggregation / analysis unit that accepts the specification of reference conditions when the user refers to the unstructured log information that has been grouped and distributed. It is a system.

According to one aspect of the present invention, logs can be suitably collected from a plurality of service servers.

The figure for demonstrating the usage environment of the information processing system 1. The schematic diagram of the information processing system 1. The figure which shows an example of the unstructured log information L. The figure which shows typically the grouping process by the grouping unit 130. The figure for demonstrating the grouping process by the grouping unit 130. The figure for demonstrating the usage scene of the information processing system 1 by a service provider. A timing chart showing an example of a processing flow until the unstructured log information L related to a service is collected by the information processing system 1. A timing chart showing an example of a processing flow until the unstructured log information L collected by the information processing system 1 is referred to. The flowchart which shows an example of the flow of the log conversion process of the information processing system 1.

Hereinafter, embodiments of the information processing system, information processing apparatus, information processing method, and program of the present invention will be described with reference to the drawings.

〔Overview〕
The information processing system is a system that collects usage histories (logs) of a plurality of service servers and supports the development of the collected usage histories across services.

Each of the plurality of service servers, for example, communicates with a web server that provides a web page corresponding to a request from a terminal device operated by a user, and a terminal device on which an application is started to exchange various information. An application server that provides content information. The service server provides, for example, a news providing service, a shopping service, an auction service, a matching service, a financial payment service, a navigation service, a webmail service, and the like via a network.

Since the services provided by each service server have different service characteristics and the OS (Operation System) and middleware used to provide the service are different, the output standard of the usage history of all service servers is unified. Things can be difficult.

Therefore, the information processing device outputs the usage history in a format that can be output without significantly changing the application configuration of each service server, and collects the usage history of the end user of the web page provided by a plurality of service servers. Then group by service. As a result, the usage history of the service provided by the service server can be referred to by other services in the same way as the user usage history of the own service, or it can be diverted to another purpose such as statistical data collection. enable.

The information processing system includes, for example, an information processing device realized by one or more processors. The information processing device is a device that manages log data referred to by a plurality of reference subjects (users who provide services using a service server). The log data is a collection of records of events generated by the execution of a service application on an end user's terminal, which are recorded over time.

The information processing device includes, for example, an acquisition unit that acquires unstructured log information that is usage history related to service provision of a plurality of service servers, particularly usage status of end users, and information included in the unstructured log information. It is provided with a grouping unit that groups at least based on the service identification information and the time stamp and stores it in the data lake.

In addition to the above-mentioned information processing apparatus, the information processing system is provided by, for example, a plurality of service servers, terminals used by users who manage services provided by the service servers (hereinafter, service providers), and service servers. It is connected to the user terminal of the user who uses the service (hereinafter referred to as the end user) via the network.

Unlike storage devices such as data warehouses (DWHs), data lakes are unstructured because there is no need to know the data structure of the data to be stored in advance and design in advance for data storage. Data can be stored. A data lake may be interpreted as a repository that can store unstructured data. The data stored in the data lake is interpreted at the time of reading using a dedicated distributed processing device, and is used for subsequent processing such as analysis.

[Overall configuration 1]
FIG. 1 is a diagram showing a usage environment of the information processing system 1. The information processing system 1 includes, for example, service servers SS-1 to SS-N (N is a natural number), end user terminals T-1 to TM (M is a natural number), a service provider terminal D, and a network NW. Communicate via. The network NW includes, for example, a part or all of WAN (Wide Area Network), LAN (Local Area Network), the Internet, a provider device, a wireless base station, a dedicated line, and the like. In the following description, when the individual service servers SS-1 to SS-N are not distinguished, they are simply referred to as the service server SS. Further, in the following description, when the individual end user terminals T-1 to TM are not distinguished, they are simply referred to as end user terminals T.

The end user terminal T and the service provider terminal D are, for example, a mobile phone such as a smartphone, a tablet terminal, a personal computer, and the like. The service provider terminal D transmits a log information reference request (command) to the information processing system 1 in response to the operation, and displays the processing result of the returned request on the display unit of the service provider terminal D.

FIG. 2 is a schematic diagram of the information processing system 1. The information processing system 1 includes, for example, an information processing device 100, a distributed processing device 200, an aggregation / analysis device 300, and an access authority management device 400.

The information processing device 100 includes, for example, an acquisition unit 110, a data lake 120, and a grouping unit 130. These components except the data lake 120 of the information processing device 100 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). It may be realized by the part (including circuitry), or it may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device including a non-transient storage medium) such as an HDD or a flash memory of the information processing device 100, or a removable storage such as a DVD or a CD-ROM. It is stored in a medium (non-transient storage medium), and may be installed in the HDD or flash memory of the information processing device 100 by mounting the storage medium in the drive device. The information processing device 100 is an example of a “log collecting unit”.

The acquisition unit 110 acquires the unstructured log information L from the service application SA or the service server SS and stores it in the data lake 120. The acquisition unit 110 may change the unstructured log information L to a format suitable for storing the unstructured log information L in the data lake 120. For example, the unstructured log information L may be serialized (arranged in order). It is binarized and then stored in the data lake 120. The unstructured log information L will be described later.

For example, when the service application SA transmits a request to the service server SS, the acquisition unit 110 causes the acquisition unit 110 to perform API communication for transmitting log information included in (or related to the request) in the request transmission. , Acquires unstructured log information L from service application SA. Further, the acquisition unit 110 causes API communication to transmit a part or all of the log information when the service server SS stores the log information in the storage unit of the own server as a result of receiving the request transmitted from the service application SA. Is performed to acquire the unstructured log information L from the service server SS. The process of acquiring the unstructured log information L by the acquisition unit 110 in this way may be referred to as a REST API.

The acquisition unit 110 may be realized by an open source having a message queue function such as Apache Kafka.

A plurality of acquisition units 110 may be provided, a dedicated acquisition unit 110 may be provided for each service server SS, or one acquisition unit 110 may be provided with unstructured log information L from the plurality of service server SSs. It may be the one to acquire.

The grouping unit 130 groups the unstructured log information L based on at least the service identification information among the information contained in the unstructured log information L stored in the data lake 120 and outputs the unstructured log information L to the distributed processing device 200. .. The grouping unit 130 may compress the grouped unstructured log information L. The grouping process by the grouping unit 130 will be described later.

[Unstructured log information]
FIG. 3 is a diagram showing an example of unstructured log information L. The unstructured log information L includes at least information for identifying a service (hereinafter, service identification information LE). If the unstructured log information L does not include the service identification information LE, the acquisition unit 110 identifies the service server SS that is the output source of the unstructured log information L, and adds the service identification information LE. You may.

In the following description, the service identification information LE will be described as including Project, which is a major classification of service types, and Dataset, which is a classification classification having a finer particle size than Project. FIG. 4 is a diagram schematically showing the grouping process by the grouping unit 130. For example, if the Project included in the service identification information LE is Project1 “shopping”, the Dataset is Dataset10 “top page”, Dataset11 “special page”, Dataset12 “product introduction page”, Dataset13 “purchase procedure page”, etc. Contains information that distinguishes the characteristics of web pages and the content of services.

The components and the order of the components included in the unstructured log information L can be set by the service provider, and the components may be added, deleted, or the order may be changed at any time. Further, unstructured log information L having a different configuration may be output for each web page according to the type and characteristics of the web page provided by the service server SS.

The unstructured log information L is transmitted to the information processing system 1 from the browser of the service application SA or the end user terminal T, for example, by using the POST method of HTML. The unstructured log information L may be the same as the log information transmitted from the service application SA to the service server SS, or a part of the log information transmitted from the service application SA to the service server SS is used. May be done. The acquisition unit 110 acquires the unstructured log information L as a JSON (JavaScript (registered trademark) Object Notation) format log file as shown in the figure and stores it in the data lake 120.

[Overall configuration 2]
Returning to FIG. 2, the distributed processing device 200 is realized by distributed storage represented by, for example, Hadoop (registered trademark), and divides unstructured metadata into a plurality of data. It is a device that processes each divided data in parallel. The distribution processing device 200 is an example of a “distribution processing unit”.

The aggregation / analysis device 300 is a device for realizing a data warehouse construction environment represented by Hive, for example, and aggregates or inquires about the unstructured log information L stored in the data lake 120. Or analyze. The aggregation / analysis device 300 receives an HQL (Hibernate Query Language) transmitted by the service provider terminal D. HQL is an query language that can be described by a convention similar to SQL (Structured Query Language), which is an query language for managing and operating a relational database. The distributed processing device 200 and the aggregation / analysis device 300 are examples of the “collection / analysis unit”.

The access authority management device 400 manages whether or not the service provider may access the log information of the service server SS that provides other services. The service provider sets in advance in the access authority management device 400 what may or may not be disclosed to other service providers among the log information of the service server SS managed by the service provider. The access authority management device 400 is an example of the “access authority management unit”.

[Grouping process]
FIG. 5 is a diagram for explaining the grouping process by the grouping unit 130. The grouping process by the grouping unit 130 is composed of, for example, four stepwise conversion processes.

As the first conversion, the grouping unit 130 serializes the unstructured log information L and further binarizes the unstructured log information L. The grouping unit 130 files the processing result of the first conversion in units of, for example, a predetermined first predetermined time (for example, 1 [min]) ((1) in FIG. 5).

Next, when the processing result of the first conversion becomes the first predetermined number (for example, 10 [pieces]), that is, when the log information of 10 [min] is collected, the grouping unit 130 makes a second. Start the conversion. For example, the grouping unit 130 classifies the processing result of the first conversion for each service and converts the classification result into an aggregated file ((2) in FIG. 5).

Next, when the processing result of the second conversion becomes the second predetermined number (for example, 6 [pieces]), that is, when the log information of 60 [min] is collected, the grouping unit 130 makes a third. Start the conversion. For example, the grouping unit 130 aggregates the processing results of the second conversion into one file ((3) in FIG. 5).

Next, as the fourth conversion, the grouping unit 130 compresses the processing result of the third conversion and sets the access authority. The grouping unit 130 converts the processing result of the third conversion into, for example, an MDS (Multiple-Dimension-Spread) format file by the compression processing in the fourth conversion ((4) in FIG. 5). MDS is open source that can achieve schemaless compression. The compression process in the fourth conversion refers to a schema such as an ORC format file (a file format suitable for use in Hive) and an LZ4 format file (a file compressed by a compression algorithm called the LZ4 algorithm). It may be converted by the required method. When converting in the ORC format or the LZ4 format, the grouping unit 130 refers to the schema information SDB in which the schema name and configuration of each service are stored.

[Use scene]
FIG. 6 is a diagram for explaining a usage scene of the information processing system 1 by a service provider. The service provider P who manages the service A thinks that, for example, "wants to compare and refer to the browsing result on Thursday and the browsing result on Friday of the service B", and explains that the information processing system 1 is used.

First, the aggregation / analysis device 300 transmits an HQL request of the service provider P, "I want to compare and refer to the browsing result on Thursday and the browsing result on Friday of the service B" from the service provider terminal D (step S1). Next, the aggregation / analysis device 300 determines whether or not the data referred to by the HQL request is data that may be referred to by the service provider P, that is, whether or not the service provider P has access authority. The access authority management device 400 is inquired (step S2), and it is determined whether or not to perform the processing after step S4 according to the inquiry result (step S3) of the access authority. For the sake of simplicity, it is assumed that the service provider P is determined to have access authority in steps S2 and S3.

Next, the aggregation / analysis device 300 interprets the received HQL request and transmits it to the distribution processing device 200 (step S4). Next, the distributed processing apparatus 200 accesses the data stored in the data lake 120 based on the interpretation result of the HQL request received in step S4 (step S5), and the data lake 120 accesses the data accessed in step S3. Is transmitted to the distributed processing device 200 (step S6), whereby the distributed processing device 200 and the aggregation / analysis device 300 transmit a response corresponding to the interpretation result of the HQL request received in step S2 to the service provider terminal D (step S6). Steps S7 and S8).

As described above, the service provider P transmits an HQL request to the information processing system 1 via the service provider terminal D to refer to the log information in the service B different from the service A managed by the service provider P. Can be done. The distributed processing device 200 omits the processes corresponding to steps S5 and S6 when the process of accumulating the data stored in the data lake 120 under the distributed file system of its own device has already been completed. May be done.

[Processing flow log collection]
Hereinafter, the processing flow of the information processing system 1 will be described. FIG. 7 is a timing chart showing an example of the processing flow until the unstructured log information L related to the service is collected by the information processing system 1.

First, the service application SA or the service server SS outputs the usage history by the end user as a log (step S10). Next, the acquisition unit 110 of the information processing apparatus 100 acquires the log information output in step S10 in the format of unstructured log information L (step S12), converts it into a predetermined format (step S14). , Stored in the data lake 120 (step S16). Next, the distributed processing device 200 performs distributed processing on the unstructured log information L stored in the data lake 120 (step S18). Since the details of the processes corresponding to steps S14 and S18 will be described later, the description thereof is omitted here. This is the end of the description of the processing of this timing chart.

[Refer to processing flow log]
FIG. 8 is a timing chart showing an example of the processing flow until the unstructured log information L collected by the information processing system 1 is referred to.

First, the service provider terminal D transmits an HQL request reflecting the content that the service provider wants to refer to to the information processing system 1 (step S20). Next, the aggregation / analysis device 300 of the information processing system 1 receives the HQL request transmitted in step S20 (step S22), and determines whether or not the service provider has access authority (step S24).

If the service provider does not have the access authority, the aggregation / analysis device 300 transmits error information indicating that the service provider does not have the reference authority to the service provider terminal D, and ends the process (step S26). When the service provider has access authority, the distributed processing device 200 generates a response to the HQL request received in step S22 (step S28), and transmits the processing result to the service provider terminal D (step S30). This is the end of the description of the processing of this timing chart.

[Processing flow log conversion]
FIG. 9 is a timing chart showing an example of a processing flow in which the unstructured log information L collected by the information processing system 1 is converted. The flowchart shown in FIG. 9 corresponds to steps S14 and S18 of FIG. 7. Further, the flowchart shown in FIG. 9 corresponds to an example of the processing flow shown in FIG.

First, the acquisition unit 110 acquires the unstructured log information L (step S100). Next, the acquisition unit 110 performs the first conversion (for example, serialization or binarization) of the acquired log information (step S102).

Next, the distributed processing apparatus 200 determines whether or not the second conversion condition (for example, the processing result of the first conversion has reached the first predetermined number) is satisfied (step S104). If it is not determined that the condition of the second conversion condition is satisfied, the distribution processing apparatus 200 returns the processing to step S100. When it is determined that the condition of the second conversion condition is satisfied, the distribution processing device 200 classifies the processing result of the second conversion processing (for example, the processing result of the first conversion for each service) into a file in which the classification results are aggregated. The conversion process) is performed (step S106).

Next, the distributed processing apparatus 200 determines whether or not the third conversion condition (for example, the processing result of the second conversion has reached the second predetermined number) is satisfied (step S108). If it is not determined that the condition of the third conversion condition is satisfied, the distribution processing apparatus 200 returns the processing to step S100. When it is determined that the condition of the third conversion condition is satisfied, the distribution processing device 200 performs a third conversion process (for example, a process of aggregating a predetermined number of second conversion processing results into one file) ( Step S110).

Next, the distributed processing apparatus 200 determines whether or not the condition for the fourth conversion (for example, a predetermined processing time is reached) is satisfied (step S112). If it is not determined that the condition of the fourth conversion condition is satisfied, the distribution processing apparatus 200 returns the processing to step S100. When it is determined that the condition of the fourth conversion condition is satisfied, the distribution processing apparatus 200 performs a fourth conversion process (for example, a process of archiving the processing result of the third conversion) (step S114). This is the end of the description of the processing of this flowchart.

As described above, the information processing system 1 of the present embodiment has an acquisition unit 110 that acquires unstructured log information L regarding the usage status of a plurality of service servers SS-1 to SS-N, and an unstructured unit 110. The grouping unit 130 that groups the information included in the organization log information L based on at least the service identification information LE and the time stamp and stores it in the data lake 120, and the unstructured log information L that stores the information in the data lake 120. The distributed processing device 200 for distributed processing and the access authority management device 400 for managing the access authority of the service provider to the grouped unstructured log information L stored in the data lake 120 are grouped and distributed processed. By providing an aggregation / analysis device 300 that accepts the designation of reference conditions when the service provider refers to the unstructured log information L in a format such as HQL, the unstructured log can be logged from a plurality of service servers SS. The information L can be preferably collected, and the service provider can refer to the log information of another service.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 Information processing system 100 Information processing device 110 Acquisition unit 130 Grouping unit 200 Distributed processing device 300 Aggregation / analysis device 400 Access authority management device SS service server

Claims (5)

A log collection unit that acquires unstructured log information related to the usage status of a plurality of service servers, groups the unstructured log information based on at least service identification information and a time stamp, and stores it in a storage unit.
A distributed processing unit that performs distributed processing of the unstructured log information stored in the storage unit,
An access authority management unit that manages user access rights to the grouped unstructured log information stored in the storage unit, and
An aggregation / analysis unit that accepts the specification of reference conditions when the user refers to the unstructured log information that has been grouped and distributed.
An information processing system equipped with. An acquisition unit that acquires unstructured log information related to the service provision of multiple service servers,
A grouping unit that groups the information included in the unstructured log information based on at least the service identification information and stores it in the storage unit.
Information processing device equipped with. The grouping unit groups the unstructured log information stepwise by a predetermined number or a predetermined time based on the time stamp included in the unstructured log information, and the grouped unstructured log. The information is compressed and stored in the storage unit.
The information processing device according to claim 2. The computer
Get unstructured log information about the usage status of multiple service servers
Of the information included in the unstructured log information, the information is grouped based on at least the service identification information and the time stamp and stored in the storage unit.
Information processing method. On the computer
Get unstructured log information about the usage status of multiple service servers
Of the information included in the unstructured log information, a process of grouping the information based on at least the service identification information and the time stamp and storing the information in the storage unit is performed.
program.

Images