CN113722139B - Data request method with high request efficiency and difficult data loss - Google Patents
Data request method with high request efficiency and difficult data loss Download PDFInfo
- Publication number
- CN113722139B CN113722139B CN202111000125.0A CN202111000125A CN113722139B CN 113722139 B CN113722139 B CN 113722139B CN 202111000125 A CN202111000125 A CN 202111000125A CN 113722139 B CN113722139 B CN 113722139B
- Authority
- CN
- China
- Prior art keywords
- retry
- exception
- request
- data
- spring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0745—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in an input/output transactions management context
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
- Retry When Errors Occur (AREA)
Abstract
The invention discloses a data request method with high request efficiency and difficult data loss, an electronic device for executing the method and a computer readable storage medium for storing the method. Aiming at the aspect of exception handling, the invention is divided into I/O exception, business logic exception, other exception (non-procedural exception) except the I/O exception, and other multi-level dimensionalities, for the business logic exception, SPRING RETRY is used for handling, and for the I/O exception, the bottom layer exception is handled by adopting an HttpClient retry mechanism, and the problem of network layer exception can be solved more bottom layer and more efficiently by utilizing a HttpRequestRetryHandler exception retry interface, thereby reducing program loss. And, in the case that the successful call and the non-procedural exception are not realized through the httpfile retry and SPRING RETRY retry, the network request is packaged into the object store so as to complete the network request by the next retrying, thereby avoiding the loss of data.
Description
Technical Field
The present invention relates to the field of data interaction technologies, and in particular, to a data request method, an electronic device, and a readable storage medium, where the data request method is efficient and data is not easy to be lost.
Background
In a data platform/warehouse, a large amount of data of various industry latitudes is often required for data integration, analysis and modeling, so that powerful decision support is provided, and therefore, it is necessary to continuously acquire data from a third party data source. Among them, calling through an interface form is common, and how to ensure that one piece of precious data can be efficiently and not easily lost in a network request is a problem that needs to be studied and solved.
To increase the probability of successful request when network is abnormal, a retry mechanism needs to be pre-made, and the request is retried after the request call fails. At present, the retry components such as SPRING RETRY, guava Retrying and the like are generally adopted to realize the abnormal retry processing, so that a very excellent abnormal retry processing mechanism is realized, but IOException abnormality which does not relate to business logic is thrown to the business layer retry processing, and unnecessary extra program loss is caused. In addition, the problems that the data acquisition fails due to factors such as unstable short-term network or downtime of a data source side server, data unavailability and the like exceeding retry times, data loss is caused, and later investigation is difficult are solved.
Accordingly, there is a need to provide a new data request method to solve the above-mentioned problems in the prior art.
Disclosure of Invention
The invention aims to provide a data request method, electronic equipment and a readable storage medium, wherein the data request method is high in request efficiency and difficult to lose data.
In order to achieve the above object, the present invention provides a data request method, including the steps of:
S1, initiating a network request to call an interface to acquire data;
S2, acquiring an interface calling result, if the interface calling result is successfully called, entering a step S5, otherwise, entering a step S3;
S3, determining an exception type corresponding to the network request, if the exception type is I/O exception, adopting an HttpClient retry machine to initiate a retry request, if the exception type is business logic exception, adopting a SPRING RETRY retry mechanism to initiate a retry request, and entering a step S4 after the HttpClient retry and/or SPRING RETRY retry is not successfully invoked and reaches a predefined retry threshold, or the exception type is I/O exception and other exceptions except business logic exception;
S4, packaging the network request into objects and storing;
s5, ending.
Preferably, the retry threshold includes a first retry threshold and a second retry threshold, in step S3, it is first determined whether the anomaly type is an I/O anomaly, if so, the httptest retry mechanism is adopted to initiate a retry request until the first retry threshold is successfully invoked and entered step S5 or exceeded; if the first retry threshold is exceeded or the abnormality type is not an I/O abnormality, judging whether the abnormality type is a business logic abnormality, if so, initiating a retry request by adopting the SPRING RETRY retry mechanism until the step S5 is successfully invoked or the second retry threshold is exceeded; if the second retry threshold is exceeded or the anomaly type is not a business logic anomaly, step S4 is entered.
Specifically, before step S1, the method further includes the steps of: s0, constructing HttpClient, and defining HttpRequestRetryHandler interfaces; and defining SPRING RETRY retry templates, and setting corresponding retry strategies and backoff strategies.
Preferably, httplient is structured in a single instance mode and timeout time is set and Http connection pool is enabled using PoolingHttpClientConnectionManager to implement connection multiplexing.
Preferably, a single-threaded pool task that scans for empty connections periodically is started, and abnormal or empty connections are closed.
Preferably, execution addShutdownHook instructs to close the httpfile object and to close the single-threaded pool for scanning for empty connections when the JVM is closed.
Preferably, before step S5 after step S4, the method further comprises: the network request is re-executed based on a Java reflection mechanism.
To achieve the above object, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor executing the computer program to perform the data requesting method as described above.
To achieve the above object, the present invention provides a computer-readable storage medium storing a computer program executable by a processor to perform the data request method as described above.
Compared with the prior art, the invention is divided into I/O abnormality, business logic abnormality, other abnormality (non-procedural abnormality) and other multi-level dimensions such as I/O abnormality, business logic abnormality, etc. aiming at the abnormality processing aspect, SPRING RETRY is used for processing the business logic abnormality, and the bottom abnormality such as the I/O abnormality is processed by adopting an HttpClient retry mechanism, and the problem of network level abnormality can be solved more bottom and more efficiently by utilizing HttpRequestRetryHandler abnormality retry interfaces, thereby reducing program loss. And, in the case that the successful call and the non-procedural exception are not realized through the httpfile retry and SPRING RETRY retry, the network request is packaged into the object store so as to complete the network request by the next retrying, thereby avoiding the loss of data.
Drawings
FIG. 1 is a flow chart of a data request method according to an embodiment of the invention.
Fig. 2 is a block diagram showing the constitution of an electronic device according to an embodiment of the present invention.
Detailed Description
For a detailed description of the contents, construction features, achieved objects and effects of the present invention, a technical solution of the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The data request method provided by the invention comprises the following steps:
S0, constructing HttpClient, and defining HttpRequestRetryHandler interfaces; and defining SPRING RETRY retry templates, and setting corresponding retry strategies and backoff strategies. Among other things, how to construct httpfile and implement HttpRequestRetryHandler interfaces, and the definition and setup of SPRING RETRY are implemented by the prior art.
S1, initiating a network request to call an interface to acquire data.
S2, obtaining an interface calling result, if the interface calling result is successfully called, entering a step S5, otherwise, entering a step S3.
S3, determining an exception type corresponding to the network request, if the exception type is I/O exception (exception caused by I/O error), adopting an HttpClient retry machine to initiate a retry request, if the exception type is business logic exception (exception not conforming to business rules), adopting a SPRING RETRY retry mechanism to initiate a retry request, and if the retry request is not successfully invoked and reaches a predefined retry threshold through HttpClient retry and/or SPRING RETRY retry, or if the exception type is I/O exception and other exceptions except the business logic exception, entering step S4.
S4, packaging the network request into objects and storing the objects in a storage medium such as a database or a message queue.
S5, ending.
How to determine the exception type corresponding to the network request is the prior art may be implemented by, for example, instanceof operators, which are not described herein.
The data request method of the present invention will be described in detail below by taking an embodiment as an example with reference to fig. 1 of the accompanying drawings.
Referring to fig. 1, fig. 1 shows an embodiment of a data request method of the present invention. As shown in fig. 1, in step S3, it is first determined whether the exception type is an I/O exception, if so, the httpcalient retry mechanism is adopted to initiate a retry request, after each retry, it is determined again whether the call is successful, if so, step S5 can be directly entered to end the current program; if the call is not successful, further judging whether the first retry threshold (retry times) is exceeded, if the first retry threshold is not exceeded, initiating a retry request again by adopting the httpgent retry mechanism until the call is successful, and entering step S5 or exceeding the first retry threshold. If the abnormal type is not the I/O abnormality or exceeds the first retry threshold, further judging whether the abnormal type is the business logic abnormality, if so, adopting SPRING RETRY retry machine to initiate a retry request, and after each retry, determining whether to call successfully again, if so, directly entering step S5 to end the current program; if the call is not successful, further judging whether the second retry threshold (retry times) is exceeded, if the second retry threshold is not exceeded, initiating a retry request again by adopting SPRING RETRY retry mechanism until the call is successful, and entering step S5 or exceeding the second retry threshold. If the anomaly type is not a business logic anomaly or exceeds the second retry threshold, the process proceeds directly to step S4 to make a persistent network request (the network request is packaged as an object and stored).
In this embodiment, before step S5 after step S4, further comprising: the network request is re-executed based on the Java reflection mechanism. Since the network request is persistently stored in step S4, the network request can be executed again after the abnormality cause is checked and the abnormality is eliminated, without losing any piece of data. The persistent information in step S4 includes basic information required by Java reflection, such as parameters, parameter types, method names, class names, and the like, and further includes status and exception information of the network request, so that whether the network request is successfully executed or not and reasons of execution failure can be obtained in the later stage, and the later stage investigation is facilitated.
Further, in the process of executing the data request method, a single-thread pool task for periodically scanning the empty connection is also started, and the abnormal or idle connection is periodically scanned to clear the abnormal or idle connection, so that more available connections are vacated. When the JVM (Java Virtual Machine ) is turned off, a addShutdownHook instruction (adding a closing hook) is executed to close the HttpClient object and to close the method for scanning the single thread pool of the empty connection, so as to implement memory cleaning, resource reclamation, and the like, and make room.
In this embodiment, when httpllient is used to send post and get requests, httpllient instances need to be instantiated, and then the httpllient. Execution () method needs to be called, each time httpllient is instantiated, the httpllient instances cannot be shared, which is disadvantageous for a large number of request processes, and httpllient class thread security is considered, in this embodiment httpllient is constructed in a single instance mode. Because the Http connection requires three handshakes, the overhead is high, in the embodiment, the timeout time is set when the Http connection is constructed, the PoolingHttpClientConnectionManager is used for starting the Http connection pool to realize connection multiplexing, the overhead of creating the TCP connection is reduced, the performance is improved, and the efficiency of network request is further improved. Meanwhile, connection activity detection is forbidden when the connection is fetched each time, and Keep-Alive mode, so that the overhead of establishing TCP connection is reduced, the performance is improved, and the efficiency of network request is improved.
Referring next to fig. 2, an electronic device 100 according to the present invention includes a processor 10, a memory 20, and a computer program 30. The memory 20 stores one or more computer programs 30, such as a data request program, which when executed by the processor 10 causes the processor 10 to implement the data request method described above.
The memory 20 may be any form of random access memory, read only memory, flash memory or other similar device integrated into an electronic device. The processor 10 may be a central processing unit or other programmable general purpose or special purpose microprocessor, digital signal processor, programmable controller, special integrated circuit or other similar device or combination of such devices.
Accordingly, the present invention also relates to a computer readable storage medium storing a computer program which, when executed by the processor 10, performs the data requesting method in the above-described embodiment. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, executable files or in some intermediate form, etc. The computer readable storage medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a Read Only Memory (ROM), a Random Access Memory (RAM), and so forth.
In summary, the invention is divided into I/O exception, business logic exception, and other exception (non-procedural exception) except I/O exception and business logic exception, etc. multi-level dimension, for business logic exception, SPRING RETRY is used to process, for I/O exception, the bottom layer exception is processed by using httpClient retry mechanism, httpRequestRetryHandler exception retry interface is used to solve problem of network layer exception more bottom layer and more high efficiency, and program loss is reduced. And, in the case that the successful call and the non-procedural exception are not realized through the httpfile retry and SPRING RETRY retry, the network request is packaged into the object store so as to complete the network request by the next retrying, thereby avoiding the loss of data. That is, the retry and spam are initiated with as little loss as possible, ensuring that each network request can be performed with trace and with efficiency.
The foregoing disclosure is only illustrative of the preferred embodiments of the present invention and is not to be construed as limiting the scope of the invention, which is defined by the appended claims.
Claims (8)
1. A data request method with high request efficiency and difficult data loss is characterized by comprising the following steps:
S1, initiating a network request to call an interface to acquire data;
S2, acquiring an interface calling result, if the interface calling result is successfully called, entering a step S5, otherwise, entering a step S3;
S3, determining an exception type corresponding to the network request, if the exception type is I/O exception, adopting an HttpClient retry machine to initiate a retry request, if the exception type is business logic exception, adopting a SPRING RETRY retry mechanism to initiate a retry request, and entering a step S4 after the HttpClient retry and/or SPRING RETRY retry is not successfully invoked and reaches a predefined retry threshold, or the exception type is I/O exception and other exceptions except business logic exception;
S4, packaging the network request into objects and storing;
S5, ending;
In step S3, first, whether the anomaly type is an I/O anomaly is judged, if yes, a retry request is initiated by adopting the httplient retry mechanism until the first retry threshold is successfully invoked and entered in step S5 or exceeded; if the first retry threshold is exceeded or the abnormality type is not an I/O abnormality, judging whether the abnormality type is a business logic abnormality, if so, initiating a retry request by adopting the SPRING RETRY retry mechanism until the step S5 is successfully invoked or the second retry threshold is exceeded; if the second retry threshold is exceeded or the anomaly type is not a business logic anomaly, step S4 is entered.
2. The data request method according to claim 1, further comprising, prior to step S1:
s0, constructing HttpClient, and defining HttpRequestRetryHandler interfaces; and defining SPRING RETRY retry templates, and setting corresponding retry strategies and backoff strategies.
3. The data request method of claim 2, wherein HttpClient is structured in single instance mode and a timeout time is set, and wherein connection multiplexing is implemented using PoolingHttpClientConnectionManager enable Http connection pool.
4. The data request method of claim 1, wherein a single-threaded pool task that scans for empty connections is turned on periodically and an abnormal or empty connection is turned off.
5. A data request method as defined in claim 4, wherein execution addShutdownHook instructions close the HttpClient object and close a single-threaded pool for scanning for empty connections when the JVM is closed.
6. The data request method according to claim 1, further comprising, after step S4 and before step S5:
The network request is re-executed based on a Java reflection mechanism.
7. An electronic device, comprising:
A processor, a memory and a computer program stored in the memory and configured to be executed by the processor, when executing the computer program, performing the data request method of any of claims 1 to 6.
8. A computer readable storage medium storing a computer program executable by a processor to perform the data request method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111000125.0A CN113722139B (en) | 2021-08-27 | 2021-08-27 | Data request method with high request efficiency and difficult data loss |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111000125.0A CN113722139B (en) | 2021-08-27 | 2021-08-27 | Data request method with high request efficiency and difficult data loss |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113722139A CN113722139A (en) | 2021-11-30 |
| CN113722139B true CN113722139B (en) | 2024-08-20 |
Family
ID=78678723
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111000125.0A Active CN113722139B (en) | 2021-08-27 | 2021-08-27 | Data request method with high request efficiency and difficult data loss |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113722139B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102446123A (en) * | 2010-10-09 | 2012-05-09 | 杭州华三通信技术有限公司 | Method and device for processing SCSI sensing data |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3157794B2 (en) * | 1998-12-15 | 2001-04-16 | 茨城日本電気株式会社 | Peripheral control processor |
| JP3498651B2 (en) * | 1999-11-02 | 2004-02-16 | 日本電気株式会社 | Disk control device, disk device, disk control system, retry frequency control method, and recording medium |
| US20070260754A1 (en) * | 2006-04-13 | 2007-11-08 | Irish John D | Hardware Assisted Exception for Software Miss Handling of an I/O Address Translation Cache Miss |
| US8381030B2 (en) * | 2009-12-23 | 2013-02-19 | Sap Ag | Business methods retry optimization |
| US10003528B2 (en) * | 2016-10-14 | 2018-06-19 | Oracle International Corporation | Scalable framework for managing switch link failovers in a switch fabric network |
| US10698941B2 (en) * | 2017-01-06 | 2020-06-30 | Oracle International Corporation | ZFS block-level deduplication at cloud scale |
| CN109376029B (en) * | 2018-09-27 | 2021-11-19 | 郑州云海信息技术有限公司 | Processing method and processing system for SCSI hard disk abnormal overtime |
| CN110475166A (en) * | 2019-08-19 | 2019-11-19 | 深圳市双翼科技股份有限公司 | A kind of data exception investigation method and system of passive optical network |
| CN110750213A (en) * | 2019-09-09 | 2020-02-04 | 华为技术有限公司 | Hard disk management method and device |
-
2021
- 2021-08-27 CN CN202111000125.0A patent/CN113722139B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102446123A (en) * | 2010-10-09 | 2012-05-09 | 杭州华三通信技术有限公司 | Method and device for processing SCSI sensing data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113722139A (en) | 2021-11-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113568686B (en) | Asynchronous processing method and device for Lua language, computer equipment and storage medium | |
| US7823170B2 (en) | Queued asynchronous remote function call dependency management | |
| US9158606B2 (en) | Failure repetition avoidance in data processing | |
| CN109471711B (en) | Task processing method and device | |
| CN110278192A (en) | Method, apparatus, computer equipment and the readable storage medium storing program for executing of extranet access Intranet | |
| US9880897B2 (en) | Failure mode identification and reporting | |
| CN114880194B (en) | Service abnormity monitoring method and device, electronic equipment and computer storage medium | |
| US12373746B2 (en) | Data processing | |
| CN115499493B (en) | Asynchronous transaction processing method, device, storage medium and computer equipment | |
| CN113722139B (en) | Data request method with high request efficiency and difficult data loss | |
| CN113127172A (en) | Task execution method and device, electronic equipment and storage medium | |
| CN115134167B (en) | A vulnerability scanning method, device, equipment and storage medium | |
| CN112379993A (en) | Robot process automation processing system, method and device | |
| CN113535439B (en) | Service request processing method, device, device and storage medium | |
| CN111913804A (en) | Pre-visit report generation method and device, electronic equipment and storage medium | |
| CN113312055B (en) | A method, device and equipment for delivering small programs | |
| CN110908820B (en) | Asynchronous task execution method, device, equipment and storage medium | |
| CN111949297A (en) | Block chain intelligent contract upgrading method and device and electronic equipment | |
| WO2023185137A1 (en) | Task management method and apparatus, and device and storage medium | |
| CN114090112B (en) | Method and device for loading configuration file, storage medium and electronic device | |
| CN114610413A (en) | Method, device, equipment and storage medium for executing synchronous and asynchronous tasks based on Java | |
| CN113127209A (en) | Cloud computing execution scheduling method and device, electronic equipment and storage medium | |
| CA2381739A1 (en) | Adaptive spin latches | |
| CN112416704B (en) | A method and apparatus for detecting system faults | |
| CN116991630B (en) | Retry execution method, retry execution device, electronic device, and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240725 Address after: 523000 room 1301, unit 2, building 4, Tian'an Digital City, No. 1, Huangjin Road, Nancheng street, Dongguan City, Guangdong Province Applicant after: Dongguan Mengda Group Co.,Ltd. Country or region after: China Applicant after: Dongguan Mengda Data Technology Co.,Ltd. Address before: 523000 room 1301, unit 2, building 4, Tian'an Digital City, No. 1, Huangjin Road, Nancheng street, Dongguan City, Guangdong Province Applicant before: Dongguan Mengda Group Co.,Ltd. Country or region before: China |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |