CN113722139A - Data request method with high request efficiency and data loss prevention - Google Patents

Data request method with high request efficiency and data loss prevention Download PDF

Info

Publication number
CN113722139A
CN113722139A CN202111000125.0A CN202111000125A CN113722139A CN 113722139 A CN113722139 A CN 113722139A CN 202111000125 A CN202111000125 A CN 202111000125A CN 113722139 A CN113722139 A CN 113722139A
Authority
CN
China
Prior art keywords
retry
exception
request
data
abnormal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111000125.0A
Other languages
Chinese (zh)
Inventor
陈浩强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dongguan Mengda Group Co ltd
Original Assignee
Dongguan Mengda Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dongguan Mengda Group Co ltd filed Critical Dongguan Mengda Group Co ltd
Priority to CN202111000125.0A priority Critical patent/CN113722139A/en
Publication of CN113722139A publication Critical patent/CN113722139A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0793Remedial or corrective actions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0745Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in an input/output transactions management context

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Retry When Errors Occur (AREA)

Abstract

The invention discloses a data request method with high request efficiency and difficult data loss, electronic equipment for executing the method and a computer-readable storage medium for storing the method. Aiming at the aspect of exception handling, the exception handling method is divided into multiple levels of dimensions such as I/O exception, service logic exception and other exceptions (non-procedural exception) except the I/O exception and the service logic exception, for the service logic exception, Spring Retry is used for handling, for the bottom level exception of the I/O exception, an HttpClient Retry mechanism is used for handling, the problem of the network level exception can be solved more bottom level and more effectively by using an HttpRequestRetryHandler exception Retry interface, and the program loss is reduced. And if successful calling is not realized through the HttpClient Retry and the Spring Retry and the non-procedural exception condition is not realized, the network request is packaged into an object for storage so as to be convenient for re-triggering and completing the network request next time, and the loss of data is avoided.

Description

Data request method with high request efficiency and data loss prevention
Technical Field
The present invention relates to the field of data interaction technologies, and in particular, to a data requesting method, an electronic device, and a readable storage medium with high requesting efficiency and low data loss probability.
Background
In a data platform/warehouse, a large amount of data of various industry latitudes are often needed for data integration, analysis and modeling, so that powerful decision support is provided, and therefore, it is extremely necessary to continuously acquire data from a third-party data source. The calling is common through an interface form, and how to ensure that one piece of precious data can be efficiently and easily lost in a network request is a problem which needs to be researched and solved urgently.
In order to improve the probability of successful request when the network is abnormal, a retry mechanism needs to be established in advance, and the request is retried after the request call fails. At present, Retry components such as Spring Retry, Guava Retry and the like are usually adopted to implement exception Retry processing, so that a very excellent exception Retry processing mechanism is implemented, but an IOException exception not involving service logic is also thrown to service level Retry processing, which results in unnecessary extra program loss. In addition, the data acquisition fails due to the fact that factors such as short-term network instability, data source server downtime and data not ready exceed retry times, and the problems of data loss, large difficulty in later-stage troubleshooting and the like are caused.
Therefore, it is desirable to provide a new data request method to solve the above problems in the prior art.
Disclosure of Invention
The invention aims to provide a data request method, an electronic device and a readable storage medium which have high request efficiency and are not easy to lose data.
In order to achieve the above object, the present invention provides a data requesting method, comprising the steps of:
s1, initiating a network request to call an interface to acquire data;
s2, obtaining the interface calling result, if the calling is successful, entering the step S5, otherwise, entering the step S3;
s3, determining the abnormal type corresponding to the network request, if the abnormal type is I/O abnormal, initiating a Retry request by adopting an HttpClient Retry mechanism, if the abnormal type is service logic abnormal, initiating a Retry request by adopting a Spring Retry mechanism, if the abnormal type is not successfully called by the HttpClient Retry and/or the Spring Retry and reaches a predefined Retry threshold value, or, if the abnormal type is other abnormal except I/O abnormal and service logic abnormal, entering the step S4;
s4, packaging the network request into an object and storing the object;
and S5, ending.
Preferably, the retry threshold includes a first retry threshold and a second retry threshold, in step S3, first determining whether the exception type is an I/O exception, if the exception type is an I/O exception, initiating a retry request by using the http valid retry mechanism until the successful call enters step S5 or the first retry threshold is exceeded; if the first Retry threshold is exceeded or the exception type is not an I/O exception, determining whether the exception type is a service logic exception, if the exception type is a service logic exception, initiating a Retry request by using the Spring Retry mechanism until the step S5 is successfully invoked or the second Retry threshold is exceeded; if the second retry threshold is exceeded or the exception type is not a business logic exception, the process proceeds to step S4.
Specifically, before step S1, the method further includes the steps of: s0, constructing an HttpClient, and defining an HttpRequestRetryHandler interface; and defining a Spring Retry template, and setting a corresponding Retry strategy and a back-off strategy.
Preferably, the Http valid is constructed in a singleton mode and a timeout time is set, and the Http connection manager is enabled by PoolinghttplientConnectionManager to implement connection multiplexing.
Preferably, the single thread pool task of the timing scanning null connection is started, and the abnormal or idle connection is closed.
Preferably, executing the addShutdownHook instruction closes the http valid object and closes the single threaded pool for scanning for empty connections when the JVM is closed.
Preferably, after the step S4 and before the step S5, the method further includes: re-executing the network request based on a Java reflection mechanism.
To achieve the above object, the present invention provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to perform the data request method as described above.
To achieve the above object, the present invention provides a computer-readable storage medium storing a computer program executable by a processor to perform the data requesting method as described above.
Compared with the prior art, the invention is divided into multiple levels of dimensions such as I/O exception, service logic exception, other exceptions (non-procedural exception) except the I/O exception and the service logic exception aiming at the aspect of exception handling, for the service logic exception, Spring Retry is used for handling, for the bottom-level exception of the I/O exception, an HttpClient Retry mechanism is used for handling, the problem of the network-level exception can be solved more bottom-level and more efficiently by using an HttpRequestRetryHandler exception Retry interface, and the program loss is reduced. And if successful calling is not realized through the HttpClient Retry and the Spring Retry and the non-procedural exception condition is not realized, the network request is packaged into an object for storage so as to be convenient for re-triggering and completing the network request next time, and the loss of data is avoided.
Drawings
Fig. 1 is a flowchart illustrating a data request method according to an embodiment of the present invention.
Fig. 2 is a block diagram of a component structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to explain the contents, structural features, objects and effects of the present invention in detail, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data request method provided by the invention comprises the following steps:
s0, constructing an HttpClient, and defining an HttpRequestRetryHandler interface; and defining a Spring Retry template, and setting a corresponding Retry strategy and a back-off strategy. How to construct the HttpClient and realize the HttpRequestRetryHandler interface, and the definition and setting of Spring Retry are realized by the prior art.
And S1, initiating a network request to call an interface to acquire data.
S2, obtaining the interface calling result, if the calling is successful, entering the step S5, otherwise, entering the step S3.
S3, determining the abnormal type corresponding to the network request, if the abnormal type is I/O abnormal (IOException abnormal, abnormal caused when I/O error occurs), adopting an HttpClient Retry mechanism to initiate a Retry request, if the abnormal type is service logic abnormal (abnormal not conforming to service rule), adopting a Spring Retry mechanism to initiate a Retry request, and entering step S4 if the abnormal type is other abnormal except I/O abnormal and service logic abnormal, and the Retry request is not successfully called by HttpClient Retry and/or Spring Retry and reaches a predefined Retry threshold value or the abnormal type is I/O abnormal and service logic abnormal.
S4, packaging the network request into an object and storing the object in a storage medium such as a database or a message queue.
And S5, ending.
How to determine the exception type corresponding to the network request is the prior art, and may be implemented by, for example, an instanceof operator, which is not described herein again.
The data request method of the present invention is described in detail below with reference to fig. 1 of the drawings and taking an embodiment as an example.
Referring to fig. 1, fig. 1 shows an embodiment of a data request method according to the present invention. As shown in fig. 1, in step S3, first, it is determined whether the exception type is an I/O exception, if the I/O exception is detected, a retry request is initiated by using an http valid retry mechanism, after each retry, it is determined whether the call is successful again, and if the call is successful, the process may directly proceed to step S5 to end the current program; if the call is not successfully called, further determining whether the call exceeds a predefined first retry threshold (retry number), and if the call does not exceed the predefined first retry threshold, initiating a retry request again by using the http valid retry mechanism until the call is successfully called to step S5 or the retry threshold is exceeded. If the abnormal type is not I/O abnormal or exceeds the first Retry threshold, further judging whether the abnormal type is business logic abnormal, if the abnormal type is business logic abnormal, initiating a Retry request by adopting a Spring Retry mechanism, after retrying once, determining whether the calling is successful again, and if the calling is successful, directly entering step S5 to finish the current program; if the call is not successfully called, further determining whether the call exceeds a predefined second Retry threshold (Retry number), and if the call does not exceed the second Retry threshold, initiating a Retry request again by using a Spring Retry mechanism until the call is successfully called to step S5 or the Retry threshold is exceeded. If the exception type is not a business logic exception or exceeds the second retry threshold, the process proceeds directly to step S4 to perform a persistent network request (package the network request into an object and store it).
In this embodiment, after step S4 and before step S5, the method further includes: the network request is re-executed based on the Java reflection mechanism. Since the network request is persistently stored in step S4, after the cause of the abnormality is checked and the abnormality is eliminated, the network request can be executed again without losing any piece of data. The persistent information in step S4 includes basic information required by Java reflection, such as parameters, parameter types, method names, class names, etc., and further includes status and exception information of the network request, so that whether the network request is executed successfully and the reason for the execution failure can be known in the later stage, which is convenient for later stage investigation.
Further, in the process of executing the data request method, a single thread pool task for scanning the empty connection at a fixed time is started, and abnormal or idle connections are scanned at a fixed time period to clear the abnormal or idle connections and vacate more available connections. When a JVM (Java Virtual Machine) is closed, an addShutdownHook instruction is executed (a closing hook is added) to close an httpparent object and close a single thread pool for scanning an empty connection, so as to realize memory cleaning, resource recovery and the like and vacate a space.
In this embodiment, when sending a post request and a get request using http policy, the http policy instance needs to be instantiated, and then the http policy.execute () method is called, so that it takes a long time to instantiate http policy each time, and the http policy instance cannot be shared, which is not beneficial to a large amount of request processing. In this embodiment, timeout time is set when constructing Http valid, and the Http connection manager is used to enable the Http connection pool to implement connection multiplexing, thereby reducing the overhead of creating TCP connections and improving performance, and further improving the efficiency of network requests. Meanwhile, the connection activity detection is forbidden when the reused connection is taken each time, and the Keep-Alive mode is kept, so that the overhead of creating the TCP connection is reduced, the performance is improved, and the network request efficiency is improved.
Referring next to fig. 2, an electronic device 100 according to the present invention includes a processor 10, a memory 20, and a computer program 30. The memory 20 stores one or more computer programs 30, such as data request programs, which when executed by the processor 10 cause the processor 10 to implement the data request method described above.
The memory 20 may be any type of random access memory, read only memory, flash memory or the like integrated into the electronic device. The processor 10 may be a central processing unit or other programmable general purpose or special purpose microprocessor, digital signal processor, programmable controller, special integrated circuit or other similar device or combination of devices.
Accordingly, the present invention also relates to a computer-readable storage medium, which stores a computer program, and when the computer program is executed by the processor 10, the data request method in the above-mentioned embodiment is completed. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U.S. disk, removable hard disk, magnetic disk, optical disk, computer memory, Read Only Memory (ROM), Random Access Memory (RAM), or the like.
In summary, the present invention is divided into multiple levels of dimensions, such as I/O exception, service logic exception, and other exceptions (non-procedural exception) except the I/O exception and the service logic exception, for the service logic exception, Spring Retry is used to process the service logic exception, and for the bottom-level exception, such as the I/O exception, http Retry mechanism is used to process the bottom-level exception, and the problem of the network-level exception can be solved more bottom-level and more efficiently by using http Retry interface, thereby reducing program loss. And if successful calling is not realized through the HttpClient Retry and the Spring Retry and the non-procedural exception condition is not realized, the network request is packaged into an object for storage so as to be convenient for re-triggering and completing the network request next time, and the loss of data is avoided. That is, retry and bottom entry are initiated with the smallest possible loss, so as to ensure that each network request can be executed in a traceable manner, and to ensure efficiency.
The above disclosure is only a preferred embodiment of the present invention, and should not be taken as limiting the scope of the invention, so that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

Claims (9)

1. A data request method with high request efficiency and difficult data loss is characterized by comprising the following steps:
s1, initiating a network request to call an interface to acquire data;
s2, obtaining the interface calling result, if the calling is successful, entering the step S5, otherwise, entering the step S3;
s3, determining the abnormal type corresponding to the network request, if the abnormal type is I/O abnormal, initiating a Retry request by adopting an HttpClient Retry mechanism, if the abnormal type is service logic abnormal, initiating a Retry request by adopting a Spring Retry mechanism, if the abnormal type is not successfully called by the HttpClient Retry and/or the Spring Retry and reaches a predefined Retry threshold value, or, if the abnormal type is other abnormal except I/O abnormal and service logic abnormal, entering the step S4;
s4, packaging the network request into an object and storing the object;
and S5, ending.
2. The data requesting method according to claim 1, wherein the retry threshold includes a first retry threshold and a second retry threshold, and in step S3, it is first determined whether the exception type is an I/O exception, and if the exception type is an I/O exception, a retry request is initiated by using the http valid retry mechanism until a successful call enters step S5 or the first retry threshold is exceeded; if the first Retry threshold is exceeded or the exception type is not an I/O exception, determining whether the exception type is a service logic exception, if the exception type is a service logic exception, initiating a Retry request by using the Spring Retry mechanism until the step S5 is successfully invoked or the second Retry threshold is exceeded; if the second retry threshold is exceeded or the exception type is not a business logic exception, the process proceeds to step S4.
3. The data requesting method of claim 1, further comprising, before step S1:
s0, constructing an HttpClient, and defining an HttpRequestRetryHandler interface; and defining a Spring Retry template, and setting a corresponding Retry strategy and a back-off strategy.
4. The data request method of claim 3, wherein the Http valid is constructed in a singleton mode and a timeout time is set, and connection multiplexing is implemented using a poolinchtplientconnectionmanager to enable Http connection pool.
5. The data request method of claim 1, wherein the single thread pool task that periodically scans for empty connections is turned on and the exception or idle connection is turned off.
6. The data request method of claim 5, wherein executing the addShutdownHook instruction closes the http parent object and closes the single thread pool for scanning for empty connections when the JVM is closed.
7. The data requesting method of claim 2, further comprising, after the step S4 and before the step S5:
re-executing the network request based on a Java reflection mechanism.
8. An electronic device, comprising:
a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, when executing the computer program, performing the data request method of any of claims 1 to 7.
9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program executable by a processor to perform the data request method according to any one of claims 1 to 7.
CN202111000125.0A 2021-08-27 2021-08-27 Data request method with high request efficiency and data loss prevention Pending CN113722139A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111000125.0A CN113722139A (en) 2021-08-27 2021-08-27 Data request method with high request efficiency and data loss prevention

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111000125.0A CN113722139A (en) 2021-08-27 2021-08-27 Data request method with high request efficiency and data loss prevention

Publications (1)

Publication Number Publication Date
CN113722139A true CN113722139A (en) 2021-11-30

Family

ID=78678723

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111000125.0A Pending CN113722139A (en) 2021-08-27 2021-08-27 Data request method with high request efficiency and data loss prevention

Country Status (1)

Country Link
CN (1) CN113722139A (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000181810A (en) * 1998-12-15 2000-06-30 Nec Ibaraki Ltd Peripheral control processor
JP2001135042A (en) * 1999-11-02 2001-05-18 Nec Corp Disk controller, disk device, disk control system, control method for the number of times of retry and recording medium
CN101055546A (en) * 2006-04-13 2007-10-17 国际商业机器公司 Method and system for processing an I/O address translation cache miss
US20110154096A1 (en) * 2009-12-23 2011-06-23 Sap Ag Business Methods Retry Optimization
CN102446123A (en) * 2010-10-09 2012-05-09 杭州华三通信技术有限公司 Method and device for processing SCSI sensing data
US20180109443A1 (en) * 2016-10-14 2018-04-19 Oracle International Corporation Scalable framework for managing switch link failovers in a switch fabric network
CN109376029A (en) * 2018-09-27 2019-02-22 郑州云海信息技术有限公司 A kind of processing method and processing system that SCSI hard disk is extremely overtime
CN110100228A (en) * 2017-01-06 2019-08-06 甲骨文国际公司 Utilize the efficient delta backup and recovery of the file system hierarchy structure of cloud object storage
CN110475166A (en) * 2019-08-19 2019-11-19 深圳市双翼科技股份有限公司 A kind of data exception investigation method and system of passive optical network
CN110750213A (en) * 2019-09-09 2020-02-04 华为技术有限公司 Hard disk management method and device

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000181810A (en) * 1998-12-15 2000-06-30 Nec Ibaraki Ltd Peripheral control processor
JP2001135042A (en) * 1999-11-02 2001-05-18 Nec Corp Disk controller, disk device, disk control system, control method for the number of times of retry and recording medium
CN101055546A (en) * 2006-04-13 2007-10-17 国际商业机器公司 Method and system for processing an I/O address translation cache miss
US20110154096A1 (en) * 2009-12-23 2011-06-23 Sap Ag Business Methods Retry Optimization
CN102446123A (en) * 2010-10-09 2012-05-09 杭州华三通信技术有限公司 Method and device for processing SCSI sensing data
US20180109443A1 (en) * 2016-10-14 2018-04-19 Oracle International Corporation Scalable framework for managing switch link failovers in a switch fabric network
CN110100228A (en) * 2017-01-06 2019-08-06 甲骨文国际公司 Utilize the efficient delta backup and recovery of the file system hierarchy structure of cloud object storage
CN109376029A (en) * 2018-09-27 2019-02-22 郑州云海信息技术有限公司 A kind of processing method and processing system that SCSI hard disk is extremely overtime
CN110475166A (en) * 2019-08-19 2019-11-19 深圳市双翼科技股份有限公司 A kind of data exception investigation method and system of passive optical network
CN110750213A (en) * 2019-09-09 2020-02-04 华为技术有限公司 Hard disk management method and device

Similar Documents

Publication Publication Date Title
US8074116B2 (en) Exception raised notification
US9158606B2 (en) Failure repetition avoidance in data processing
US9495201B2 (en) Management of bottlenecks in database systems
CN113568686B (en) Asynchronous processing method and device for Lua language, computer equipment and storage medium
CN108519920B (en) Scheduling retry method and device
EP0532334A2 (en) Error recovery in an information processing system
CN113157411B (en) Celery-based reliable configurable task system and device
CN110633133A (en) Task processing method and device and computer readable storage medium
US20160034337A1 (en) Failure Mode Identification and Reporting
WO2020253045A1 (en) Configured supplementary processing method and device for data of which forwarding has abnormality, and readable storage medium
CN113961338A (en) Management system and management method of dynamic thread pool and thread task processing method
CN113722139A (en) Data request method with high request efficiency and data loss prevention
CN111913804A (en) Pre-visit report generation method and device, electronic equipment and storage medium
CN113364857A (en) Service data processing method and device and server
WO2023185137A1 (en) Task management method and apparatus, and device and storage medium
CN114816662A (en) Container arrangement method and system applied to Kubernetes
CN115934272A (en) Online batch task processing method and device
CN115499493A (en) Asynchronous transaction processing method and device, storage medium and computer equipment
CN111078442A (en) Fault-tolerant retry method and device and terminal
CN112328408A (en) Data processing method, device, system, equipment and storage medium
CN110543349B (en) Application starting acceleration method and device and computer readable storage medium
US6976107B2 (en) Adaptive spin latches
CN115469934A (en) Configuration file loading method, device and equipment
CN111563000A (en) File generation method, intelligent terminal and storage medium
CN111538491A (en) Data event processing method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination