CN114138608A - Distributed full-link service monitoring system - Google Patents
Distributed full-link service monitoring system Download PDFInfo
- Publication number
- CN114138608A CN114138608A CN202111480604.7A CN202111480604A CN114138608A CN 114138608 A CN114138608 A CN 114138608A CN 202111480604 A CN202111480604 A CN 202111480604A CN 114138608 A CN114138608 A CN 114138608A
- Authority
- CN
- China
- Prior art keywords
- module
- monitoring
- alarm
- monitoring system
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a distributed full link service monitoring system, which comprises: the index collection module is used for collecting index data; the index collection module adopts a probe method and an sdk method; when the probe is started, introducing probe environment parameters and transmitting data to a core processing module; the sdk index collection method is used for adding corresponding annotations to the monitoring method; the alarm module is used for butting the core processing module, sending an alarm push and forwarding the alarm push to different alarm channels; the core processing module is used for interacting with other modules and providing analysis processing capacity; the core processing module comprises a remote calling unit, a link tracking unit, an index management unit, an analysis unit and a query unit. The invention overcomes the defects that open-source products can only support monitoring of remote calls such as http, rpc and the like and cannot support method levels in a process.
Description
Technical Field
The invention relates to the technical field of full-link service monitoring systems, in particular to a distributed full-link service monitoring system.
Background
With the increasing popularity of micro-service architectures, it becomes common for cross-calls between services to occur in a system, and when a business process involves cross-calls between multiple services, a service call link is formed, which is often referred to as a service call link. The method has the advantages that the whole business processing process can be packaged in different service processes, and the whole service is deployed in a distributed mode, so that the performance and the stability of the whole service are improved, which is the most core value of a micro-service architecture, but with the advantages brought, the micro-service also brings some disadvantages, such as distributed transaction, CAP (common CAP) problems and full link monitoring which needs to be mainly solved by the research.
When a service involves multiple services, calls between the services are generated, different services are usually encapsulated in different processes, and interaction is required through a specific protocol and a specific data format, and currently, popular Http call modes of spring group system and rpc call mode of dubbo system are available. When the service call link comprises more than 2 service nodes, once problems such as abnormal call or slow service call occur in the call process, the problems are quite difficult to be solved. Because different services are provided by different development teams or even different companies, related debugging codes cannot be uniformly added into codes of the services, different teams are often required to be mutually configured and jointly transformed, which consumes very large manpower and material resources and even influences the normal operation of the online services. Based on the above background, there is a distributed full-link service monitoring system, which monitors all service calls and method calls of the entire service call link, and if parameters and time are called, all service calls on the entire link are arranged in tandem in the call sequence as a chain. The global analysis can be carried out on the whole link, and the analysis can also be carried out on local nodes in the link so as to quickly locate the position where the problem occurs.
At present, open-source products with a plurality of links for monitoring, such as skywalk, zipkin, pinpoint and the like, in the market are all popular full-link monitoring products at present. These products basically provide functions of link monitoring, monitoring storage, UI query and the like, but open source products usually make some sacrifice in terms of related depth in order to meet various usage scenarios as much as possible. The existing open source product can only support monitoring of remote calls such as http and rpc, and cannot support method level in a process.
Disclosure of Invention
Aiming at the problems, the invention provides a distributed full-link service monitoring system, which overcomes the defects that open-source products can only support monitoring of remote calls such as http, rpc and the like and cannot support method levels in a process.
The technical scheme adopted by the invention is as follows:
a distributed full link service monitoring system, comprising: the index collection module is used for collecting index data; the index collection module adopts a probe method and an sdk method; when the probe is started, introducing probe environment parameters and transmitting data to a core processing module; the sdk index collection method is used for adding corresponding annotations to the monitoring method;
the alarm module is used for butting the core processing module, sending an alarm push and forwarding the alarm push to different alarm channels;
the core processing module is used for interacting with other modules and providing analysis processing capacity; the core processing module comprises a remote calling unit, a link tracking unit, an index management unit, an analysis unit and a query unit; the remote calling unit is used for receiving the collected data of the index collection module;
and the data storage module is used for storing the monitoring information.
The invention provides calling monitoring of method levels in a process, so that link monitoring is more comprehensive, abnormal services can be positioned in time, a specific method called in the services can be quickly positioned, problems are positioned more accurately, and better stability and reliability can be brought to certain systems with higher requirements on online service processing. The invention provides two monitoring acquisition modes of the probe and sdk, improves the usability and operability of the whole monitoring system by using the probe, and meets the monitoring requirements of users in different scenes by complementing the probe mode with sdk that the monitoring logic and the method level cannot be customized. Meanwhile, the invention provides two modes of intercepting intrusion and bytecode intrusion, improves the flexibility of index data acquisition, and a user can customize data acquisition logic according to actual requirements and meet special requirements in specific scenes.
Optionally, the system further comprises a UI visualization module, wherein the UI visualization module is used for providing various charts and visually checking the calling condition of the system. The existing open source products have the defects that visual UI statistics and analysis capability are not rich enough, and only simple link query is usually supported; the abnormal alarm module has poor expansion capability and cannot be expanded by self definition; poor flexibility, unable to deal with special scenes for individual transformation and the like. The UI self-defining module provided by the invention has the advantages that besides the default provided query function, the visual configuration is provided to generate the monitoring analysis panel, the monitoring analysis capability is improved, and the request calling condition is observed more intuitively; the invention overcomes the defects that the visual UI counting and analyzing capability is not rich enough and only simple link query is supported generally.
Optionally, the analysis unit is configured to analyze the monitoring data, push alarm information to the alarm module, and provide an analysis interface for the UI visualization module.
Optionally, the query unit is configured to provide a relevant query interface for the UI visualization module.
Optionally, the alarm module is further configured with an expansion interface, and the expansion interface is used for completing docking with other alarm channels.
Optionally, the warning channel includes a mail, a nail and a short message. The invention provides abundant expansion interfaces through the alarm module, and can be accessed to a specific alarm channel, such as nails and the like, only by realizing according to the interface standard; the problem that the abnormal alarm module is poor in expansion capability and cannot be expanded by self-definition is solved; poor flexibility and incapability of coping with the personalized reconstruction of special scenes.
Optionally, the monitoring process of the link tracking unit includes the following steps:
when the request reaches a certain service;
a spanId is generated for the request, and the spanId with the service is passed over with the request as the parent spanId for the current service.
The invention also discloses an operation method for the distributed full link service monitoring system to call the link monitoring at the method level, which comprises the following steps:
1) sdk packages were introduced in the project; the sdk packet provides a number of @ tracers and is the @ TraceMethod annotation;
2) generating an agent class and an agent method for the class and the method added with the @ Tracer and @ TracMethod notes during system packaging;
3) when in actual calling, the agent method of the agent class is called first, and the monitoring logic and the calling of the original interface are completed in the agent method;
4) when the call is finished, the span Id parameter is transmitted to the next agent method, and the agent method pushes the monitoring information to the core module for processing. The method is different from the traditional link monitoring system, the distributed full link monitoring system provides calling link monitoring at the method level in the process, the monitoring at the method level provided by the distributed full link monitoring is mainly completed in a coding mode, sdk packages are required to be introduced into related projects when the monitoring function is realized, in order to minimize code intrusion, a series of comments are provided in sdk packages, and the monitoring logic can be completed only by adding the related comments to the method to be monitored and performing simple configuration.
Sdk, the first annotation provided in the package is @ Tracer, which can only be decorated on Class, and indicates that the Class is the Class that needs to be enhanced by link monitoring, and finally, a dynamic proxy mechanism is used to generate proxy classes for all classes to which the @ Tracer annotation is added.
The second annotation is @ TraceMethod, and the methods added with the annotations generate agent methods, and monitoring related logic such as traceId and spanId generation, request condition recording, request execution time recording and the like are encapsulated in the agent methods.
During system packaging, generating an agent class and an agent method for the class and the method added with the @ Tracer and @ TraceMethod annotation, and during actual calling, calling the agent method of the agent class first, and completing monitoring logic and calling of an original interface in the agent method. And at the end of the call, pass parameters such as the spanId into the next proxy method.
(III) advantageous effects
1. The invention provides calling monitoring of method levels in a process, so that link monitoring is more comprehensive, abnormal services can be positioned in time, a specific method called in the services can be quickly positioned, problems are positioned more accurately, and better stability and reliability can be brought to certain systems with higher requirements on online service processing.
2. The UI self-defining module provided by the invention has the advantages that a visual configuration is provided to generate a monitoring analysis panel besides the default provided query function, the monitoring analysis capability is improved, and the request calling condition is more intuitively observed. The invention provides the UI self-defining module, so that a user can self-define and construct the monitoring panel according to the self service requirement, the monitoring panel is more in line with the actual requirement and more intuitive, and the user can directly obtain the monitoring index concerned by the user.
3. The invention provides two monitoring and collecting modes of agent and sdk, the usability and operability of the whole monitoring system are improved by using the agent, sdk is used for complementing the monitoring mode that the monitoring logic and the method level cannot be customized, and the monitoring requirements of users under different scenes are met.
Description of the drawings:
fig. 1 is a schematic structural diagram of a distributed full-link service monitoring system according to embodiment 1 of the present invention;
fig. 2 is a logical structure diagram of an index collection module of a distributed full-link service monitoring system according to an embodiment 2 of the present invention;
FIG. 3 is a flowchart of a method of an alarm module of a distributed full link service monitoring system according to an embodiment 2 of the present invention;
fig. 4 is a logical structure diagram of a link tracking unit of the distributed full-link service monitoring system according to the embodiment 2 of the present invention;
fig. 5 is a logic structure diagram of method-level call link monitoring of a distributed full link service monitoring system according to embodiment 3 of the present invention.
The specific implementation mode is as follows:
the present invention will be described in detail below with reference to the accompanying drawings.
The sdk method of the present invention refers to a software development kit.
Example 1
As shown in fig. 1, the technical solution adopted by the present invention is as follows:
a distributed full link service monitoring system, comprising: the system comprises an index collection module, an alarm module, a core processing module, a data storage module and a UI visualization module, wherein the index collection module, the alarm module, the core processing module, the data storage module and the UI visualization module are used for collecting index data; the index collection module adopts a probe method and an sdk method; when the probe is started, introducing probe environment parameters and transmitting data to a core processing module; the sdk index collection method is used for adding corresponding annotations to the monitoring method; the alarm module is used for butting the core processing module, sending an alarm push and forwarding the alarm push to different alarm channels; the core processing module is used for interacting with other modules and providing analysis processing capacity; the core processing module comprises a remote calling unit, a link tracking unit, an index management unit, an analysis unit and a query unit; the remote calling unit is used for receiving the collected data of the index collection module; and the data storage module is used for storing the monitoring information.
Example 2
A distributed full link service monitoring system, comprising: the system comprises an index collection module, an alarm module, a core processing module, a data storage module and a UI visualization module, wherein the index collection module is used for collecting index data; the index collection module adopts a probe method and an sdk method; when the probe is started, introducing probe environment parameters and transmitting data to a core processing module; the sdk index collection method is used for adding corresponding annotations to the monitoring method; the alarm module is used for butting the core processing module, sending an alarm push and forwarding the alarm push to different alarm channels; the core processing module is used for interacting with other modules and providing analysis processing capacity; the core processing module comprises a remote calling unit, a link tracking unit, an index management unit, an analysis unit and a query unit; the remote calling unit is used for receiving the collected data of the index collection module; and the data storage module is used for storing the monitoring information.
In the embodiment, the index collection module collects index data and pushes the data to the core processing module; the remote calling module of the core processing module receives the data, analyzes the monitoring data through the analysis unit and pushes the alarm information to the alarm module; during which the analyzed monitoring data is stored on a data storage module.
In a specific implementation process, the data storage module refers to a data storage system supported by a distributed full-link monitoring system, and includes mainstream relational data MySQL, a NoSQL data database elastic search, and the like, and a user can select different data storage systems according to actual data volume and use conditions.
As shown in fig. 2, in the specific implementation process, when the probe mode is used, only the environment parameters related to the probe need to be added when the project is started, and no code is needed, the probe automatically acquires the call information according to the service call network, and persists the call information through the distributed full-link core module. The code intrusion mode mainly refers to an sdk mode, a related sdk packet needs to be introduced into a project, a certain degree of intrusion is caused to codes, in order to reduce code intrusion, a sdk packet provides annotation related annotations, when the method is used, the corresponding annotations are only needed to be added to related methods, and redundant codes do not need to be written. sdk is mainly used to make up for the weakness of the probe method that cannot realize the in-process method level monitoring.
The UI visualization module is used for providing various charts and visually checking the calling condition of the system. The analysis unit is used for analyzing the monitoring data, pushing alarm information to the alarm module and providing an analysis interface for the UI visualization module. The query unit is used for providing a relevant query interface for the UI visualization module. And the alarm module is also provided with an expansion interface which is used for completing the butt joint with other alarm channels. The alarm channel comprises mails, nails and short messages.
As shown in fig. 3, in the specific implementation process, the distributed full-link monitoring system supports multiple warning channels such as mails, nails, short messages and the like by default, and when these channels are used, the messages can be validated only by configuring the abnormal warning module of the distributed link monitoring system, each channel provides a specific configuration item, for example, the mails need to configure information such as mailbox servers, user names, passwords and the like, and the configuration supports immediate validation. In addition to the default alarm channel, the distributed full-link monitoring system also provides an extension interface for alarm, and for the condition that the default alarm channel cannot meet the requirement, a user can realize the extension of the alarm channel in a self-defined mode according to the extension interface, the self-defined extension package supports dynamic loading, the whole system does not need to be repackaged or started, the loading of the self-defined package is completed in real time through a JVM dynamic loading mechanism, and the system takes effect immediately.
As shown in fig. 4, in the implementation, the link monitoring concatenates the entire service invocation chain through a traceId, and a service invocation link has only one unique traceId, and when a request reaches a certain service, a spanId is generated for the request, and the spanId with the service is transmitted along with the request as the parent spanId of the current service. With the traceId and the spanId, the entire link invocation service can be composed into one complete link tree.
Example 3
As shown in fig. 5, the present invention also discloses an operation method of a distributed full link service monitoring system for method-level call link monitoring, which comprises the following steps:
1) sdk packages were introduced in the project; the sdk packet provides a number of @ tracers and is the @ TraceMethod annotation;
2) generating an agent class and an agent method for the class and the method added with the @ Tracer and @ TracMethod notes during system packaging;
3) when in actual calling, the agent method of the agent class is called first, and the monitoring logic and the calling of the original interface are completed in the agent method;
4) when the call is finished, the span Id parameter is transmitted to the next agent method, and the agent method pushes the monitoring information to the core module for processing.
In the specific implementation process, different from a traditional link monitoring system, the distributed full link monitoring system provides calling link monitoring at a method level in a process, the monitoring at the method level provided by the distributed full link monitoring is mainly completed in a coding mode, sdk packages need to be introduced into related items when a monitoring function is realized, in order to minimize code intrusion, a series of annotations are provided in sdk packages, and monitoring logic can be completed only by adding the related annotations to the method to be monitored and performing simple configuration.
Sdk, the first annotation provided in the package is @ Tracer, which can only be decorated on Class, and indicates that the Class is the Class that needs to be enhanced by link monitoring, and finally, a dynamic proxy mechanism is used to generate proxy classes for all classes to which the @ Tracer annotation is added.
The second annotation is @ TraceMethod, and the methods added with the annotations generate agent methods, and monitoring related logic such as traceId and spanId generation, request condition recording, request execution time recording and the like are encapsulated in the agent methods.
During system packaging, generating an agent class and an agent method for the class and the method added with the @ Tracer and @ TraceMethod annotation, and during actual calling, calling the agent method of the agent class first, and completing monitoring logic and calling of an original interface in the agent method. And at the end of the call, pass parameters such as the spanId into the next proxy method.
The above description is only for the preferred embodiment of the present invention and is not intended to limit the scope of the present invention, and all equivalent structural changes made by using the contents of the present specification and the drawings can be directly or indirectly applied to other related technical fields and are included in the scope of the present invention.
Claims (8)
1. A distributed full link service monitoring system, comprising:
the index collection module is used for collecting index data; the index collection module adopts a probe method and an sdk method; when the probe is started, introducing probe environment parameters and transmitting data to a core processing module; the sdk index collection method is used for adding corresponding annotations to the monitoring method;
the alarm module is used for butting the core processing module, sending an alarm push and forwarding the alarm push to different alarm channels;
the core processing module is used for interacting with other modules and providing analysis processing capacity; the core processing module comprises a remote calling unit, a link tracking unit, an index management unit, an analysis unit and a query unit; the remote calling unit is used for receiving the collected data of the index collection module;
and the data storage module is used for storing the monitoring information.
2. The distributed full link service monitoring system of claim 1 further comprising a UI visualization module for providing various charts for visually viewing system calls.
3. The distributed full link service monitoring system of claim 2, wherein the analysis unit is configured to analyze the monitoring data, push the alarm information to the alarm module, and provide an analysis interface for the UI visualization module.
4. The distributed full link service monitoring system of claim 2 wherein the query unit is configured to provide a relevant query interface for the UI visualization module.
5. The distributed full link service monitoring system according to claim 1, 2, 3 or 4, wherein an expansion interface is further configured on the alarm module, and the expansion interface is used for completing interfacing with other alarm channels.
6. The distributed full link service monitoring system of claim 1 or 2 or 3 or 4 wherein the alert channel includes mail, nails, and text messages.
7. The distributed full link service monitoring system according to claim 1 or 2 or 3 or 4, wherein the monitoring flow of the link tracking unit comprises the steps of:
when the request reaches a certain service;
a spanId is generated for the request, and the spanId with the service is passed over with the request as the parent spanId for the current service.
8. An operation method for a distributed full link service monitoring system to invoke link monitoring at a method level, comprising the steps of:
1) sdk packages were introduced in the project; the sdk packet provides a number of @ tracers and is the @ TraceMethod annotation;
2) generating an agent class and an agent method for the class and the method added with the @ Tracer and @ TracMethod notes during system packaging;
3) when in actual calling, the agent method of the agent class is called first, and the monitoring logic and the calling of the original interface are completed in the agent method;
4) when the call is finished, the span Id parameter is transmitted to the next agent method, and the agent method pushes the monitoring information to the core module for processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111480604.7A CN114138608A (en) | 2021-12-07 | 2021-12-07 | Distributed full-link service monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111480604.7A CN114138608A (en) | 2021-12-07 | 2021-12-07 | Distributed full-link service monitoring system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114138608A true CN114138608A (en) | 2022-03-04 |
Family
ID=80384730
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111480604.7A Pending CN114138608A (en) | 2021-12-07 | 2021-12-07 | Distributed full-link service monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114138608A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118055039A (en) * | 2024-04-11 | 2024-05-17 | 明阳产业技术研究院(沈阳)有限公司 | Network equipment management method and system based on telemetry data |
-
2021
- 2021-12-07 CN CN202111480604.7A patent/CN114138608A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118055039A (en) * | 2024-04-11 | 2024-05-17 | 明阳产业技术研究院(沈阳)有限公司 | Network equipment management method and system based on telemetry data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11269859B1 (en) | Correlating different types of data of a distributed ledger system | |
| US10528454B1 (en) | Intelligent automation of computer software testing log aggregation, analysis, and error remediation | |
| US11836148B1 (en) | Data source correlation user interface | |
| CN107273280B (en) | Log processing method and device, electronic equipment and storage medium | |
| US9697104B2 (en) | End-to end tracing and logging | |
| CN107562556B (en) | Failure recovery method, recovery device and storage medium | |
| US8667147B2 (en) | Monitoring related content requests | |
| US11507562B1 (en) | Associating data from different nodes of a distributed ledger system | |
| CN102196478B (en) | Diagnosis method and system for failure of network management system | |
| WO2015039046A1 (en) | Data flow exploration | |
| CN101404696A (en) | Visible service flow tracing method and system, service flow processing system | |
| CN106354647B (en) | One kind applies detection method and server | |
| US11875275B1 (en) | Custom time series models in computer analytics systems | |
| US11449488B2 (en) | System and method for processing logs | |
| US11860858B1 (en) | Decoding distributed ledger transaction records | |
| US11816479B2 (en) | System and method for implementing a code audit tool | |
| CN104320793B (en) | A kind of Automated testing method of cell phone short messages and system | |
| WO2023273218A1 (en) | Json packet checking method and json packet checking apparatus | |
| CN113127108A (en) | Service request processing method and device, storage medium and electronic equipment | |
| CN113836014A (en) | Interface testing method and device, electronic equipment and storage medium | |
| CN114138608A (en) | Distributed full-link service monitoring system | |
| CN109286608B (en) | Call chain processing method and device, storage device and computer device | |
| CN116991929A (en) | Micro-service system based on big hospital data | |
| CN110147294A (en) | Acquisition methods, device, terminal and the computer readable storage medium of Debugging message | |
| CN113419738A (en) | Interface document generation method and device and interface management equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |