CN115757918A

CN115757918A - Data collection method, system, device and medium

Info

Publication number: CN115757918A
Application number: CN202211468555.XA
Authority: CN
Inventors: 贺宁; 魏程琛
Original assignee: Chongqing Unisinsight Technology Co Ltd
Current assignee: Chongqing Unisinsight Technology Co Ltd
Priority date: 2022-11-22
Filing date: 2022-11-22
Publication date: 2023-03-07

Abstract

The application provides a data collection method, a system, a device and a medium, wherein the method comprises the following steps: determining a retry error list and a number of retries, the retry error list including at least one of a network failure and a request failure; reading the retention time of data in a collection data table, and initiating a transfer request before the retention time is reached; marking the data as collected data which is not transferred according to the transfer request; acquiring the address of the collected data, and downloading the collected data according to the retriable error list, the retried times and the address of the collected data; and after the collection data is downloaded successfully, the collection data is transferred to a preset persistent file to store the collection data for a long time. The method and the device have the advantages that on one hand, the requirement of a user for storing the collected data for a long time can be met, the collected data cannot be lost, on the other hand, the collected data can be accurately stored, and the accuracy and the efficiency of data collection are improved.

Description

Data collection method, system, device and medium

Technical Field

The present application relates to the field of video monitoring or image processing, and in particular, to a data collection method, system, device, and medium.

Background

At present, with the continuous development of urbanization, the continuous increase of video monitoring position, the picture data that the video monitoring trade was gathered are more and more, lead to video image's database bigger and bigger, but because hardware construction's is with high costs, the cycle length, so many places all adopt the storage strategy that shortens data retention period or data cover the oldest data completely. However, some data users consider it relatively important to perform business processing on them, such as: however, since the collected data and the automatically collected data are stored together, they are easily covered or deleted after the storage period, and further, the collected data cannot be stored for a long time.

Content of application

In view of the above-mentioned shortcomings of the prior art, the present application provides a data collection method, system, device and medium to solve the above-mentioned technical problem that the collected data cannot be saved for a long time.

In a first aspect, the present application provides a data collection method, including:

determining a retriable error list and a number of retries, the retriable error list including at least one of a network failure and a request failure;

reading the retention time of data in a collection data table, and initiating a transfer request before the retention time is reached;

marking the data as collection data which is not transferred according to the transfer request;

acquiring the address of the collected data, and downloading the collected data according to the retry error list, the retry times and the address of the collected data;

and after the collection data is downloaded successfully, the collection data is transferred to a preset persistent file to store the collection data for a long time.

In a possible implementation manner, the marking the data as non-transferred collection data according to the transfer request includes:

and acquiring the execution state of the data according to the unloading request, acquiring the value range of a mark field of the data when the execution state of the data is in an un-called state, changing the value of the mark field according to the value range of the mark field, and setting the data state to be in an un-unloading state according to the changed value of the mark field so as to mark the data as the collection data which is not unloaded.

In one possible implementation, obtaining an address of the collection data, and downloading the collection data according to the retriable error list, the retry number, and the address of the collection data includes:

judging whether the address of the collected data is an HTTP address or not;

when the address of the collected data is an HTTP address, identifying the address as an absolute path, and downloading the collected data according to the retriable error list, the retriable times and the absolute path;

when the address of the collected data is not an HTTP address, identifying the address as a relative path, converting the relative path into the absolute path, and downloading the collected data according to the retriable error list, the retried times and the absolute path.

In a possible implementation manner, the identifying, when the address of the favorite data is not an HTTP address, that the address is a relative path, converting the relative path into the absolute path, and downloading the favorite data according to the retriable error list, the retry number, and the absolute path includes:

when the address of the collected data is not an HTTP address, extracting a storage mark to be identified from the address of the collected data;

matching the storage mark to be identified with an identified storage mark stored locally, wherein the matching is successful, and setting the cloud storage address corresponding to the identified storage mark as the cloud storage address corresponding to the storage mark to be identified according to the corresponding relation between the identified storage mark and the cloud storage address;

and downloading the collection data according to the retriable error list, the retried times and the cloud storage address corresponding to the storage mark to be identified.

In a possible implementation manner, after the collection data is successfully downloaded, the method of transferring the collection data to a preset persistent file to store the collection data for a long time includes:

after the collection data is downloaded successfully, the collection data is transferred to a preset persistent file;

and acquiring a storage address of the collected data in the persistent file, taking the storage address as a new address of the collected data, and replacing the new address of the collected data with the address of the collected data.

In one possible implementation, after obtaining the address of the favorite data and downloading the favorite data according to the retriable error list, the retry number, and the address of the favorite data, the method further includes:

if the collected data is not downloaded successfully, acquiring a downloading error according to an error information field in the return, and counting the downloading times;

judging whether the download error is in the retry error list or not, and simultaneously judging whether the download times are not more than the retry times or not;

and when the download error is in the retriable error list and the download times are not more than the retry times, downloading the collected data again until the download error is not in the retriable error list or the download times are more than the retry times, and stopping downloading the collected data.

In a possible implementation manner, the data volume of the collection data is obtained, if the data volume of the collection data is larger than a preset data volume threshold, a thumbnail of the collection data is generated, and the collection data and the thumbnail of the collection data are uploaded to a preset backup server.

In a second aspect, the present application further provides a data collection system, the system comprising:

a determining module for determining a retriable error list and a retry number, the retriable error list including at least one of a network failure and a request failure;

the retention time reading module is used for reading the retention time of the data in the collection data table and initiating a unloading request before the retention time is reached;

the collected data marking module is used for marking the collected data which are not transferred according to the transfer request;

the collected data downloading module is used for acquiring the address of the collected data and downloading the collected data according to the retry error list, the retry times and the address of the collected data;

and the collected data dump module is used for dumping the collected data into a preset persistent file after the collected data is successfully downloaded so as to store the collected data for a long time.

In a third aspect, the present application also provides an electronic device comprising a processor, a memory, and a communication bus;

the communication bus is used for connecting the processor and the memory;

the processor is configured to execute the computer program stored in the memory to implement the data collection method according to any one of the embodiments.

In a fourth aspect, the present application further provides a computer-readable storage medium having stored thereon a computer program for causing a computer to execute the data collection method according to any one of the embodiments described above.

The beneficial effect of this application: compared with the traditional method, the method has the advantages that on one hand, the requirement of a user for storing the collected data for a long time can be met, the collected data cannot be lost, the safety of data collection is improved, on the other hand, the collected data can be accurately stored, and the accuracy and the efficiency of data collection are improved.

Drawings

FIG. 1 is a schematic diagram of an application environment of a data collection method implemented in an embodiment of the present application;

FIG. 2 is a flow chart illustrating a data collection method provided in an embodiment of the present application;

FIG. 3 is a flow diagram provided in an embodiment of the present application for identifying a collection data address;

FIG. 4 is another flow diagram provided in an embodiment of the present application for identifying a collection data address;

FIG. 5 is a flow chart of downloading favorite data provided in an embodiment of the present application;

FIG. 6 is a complete flow diagram of a method of data collection provided in an embodiment of the present application;

FIG. 7 is a block diagram of a data collection system provided in an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application is provided by way of specific examples, and other advantages and effects of the present application will be readily apparent to those skilled in the art from the disclosure herein. The present application is capable of other and different embodiments and its several details are capable of modifications and/or changes in various respects, all without departing from the spirit of the present application. It should be noted that the features in the following embodiments and examples may be combined with each other without conflict.

It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present application, and the drawings only show the components related to the present application and are not drawn according to the number, shape and size of the components in actual implementation, and the type, number and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.

In the following description, numerous details are set forth to provide a more thorough explanation of the embodiments of the present application, however, it will be apparent to one skilled in the art that the embodiments of the present application may be practiced without these specific details, and in other embodiments, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the embodiments of the present application.

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the implementation method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, blockchains, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (CV) is a science for researching how to make a machine see, and further means that a camera and a Computer are used for replacing human eyes to perform machine Vision such as identification, tracking and measurement on a target, and further image processing is performed, so that the Computer processing becomes an image more suitable for human eyes to observe or is transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technologies, virtual reality, augmented reality, synchronous positioning, map construction, and other technologies, and also include common biometric technologies such as face recognition and fingerprint recognition.

The application provides a data collection method, and relates to the technical fields of artificial intelligence, machine learning and the like.

Please refer to fig. 1, which is a schematic diagram of an application environment of an implementation environment of a data collection method according to an embodiment of the present application. As shown in fig. 1, the enforcement environment application network architecture may include a server 01 (server cluster) and a monitoring terminal cluster. The monitoring terminal cluster may include one or more monitoring terminals, and the number of the monitoring terminals is not limited herein. As shown in fig. 1, the monitoring terminal 100a, the monitoring terminal 100b, the monitoring terminal 100c, \8230, and the monitoring terminal 100n may be specifically included. As shown in fig. 1, a monitoring terminal 100a, a monitoring terminal 100b, and a monitoring terminal 100c, \8230, and a monitoring terminal 100n may be respectively connected to the server 10 through a network, so that each monitoring terminal may interact data with the server 10 through the network connection. The specific connection manner of the network connection is not limited herein, and for example, the connection may be directly or indirectly performed through a wired communication manner, or may be directly or indirectly performed through a wireless communication manner.

As shown in fig. 1, the server 01 in the embodiment of the present application may be a server corresponding to a monitoring terminal. The server 01 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. For convenience of understanding, the monitoring terminal can collect data by collecting monitoring videos and sending the monitoring videos to the server 01. The data collection method can be performed in any device such as a server, a server cluster or a cloud computing service cluster. For example, the server has a data collection function, for example, the server collects data of the acquired image to be measured.

Referring to fig. 2, fig. 2 is a schematic flow chart of a data collection method according to an embodiment of the present application, which is detailed as follows:

step S201, determining a retry error list and retry times, wherein the retry error list comprises at least one of network failure and request failure;

step S202, reading the retention time of the data in the collection data table, and initiating a transfer request before the retention time is reached;

the retention time is set by the user or default of the system, and is not limited herein.

Wherein different retention times may be set for different data. For convenience of explanation, the following are exemplified:

the collection data table comprises data A, data B and data C, the retention time of the data A is set to be 20 days, the retention time of the data B is set to be 30 days, the retention time of the data C is set to be 50 days, a unloading request of the data A is initiated before the data A arrives 20 days, an unloading request of the data B is initiated before the data B arrives 30 days, and an unloading request of the data C is initiated before the data C arrives 50 days.

The data A, the data B and the data C are any one of picture data, video data, audio data and character data. For convenience of explanation, the following are exemplified: the data a and the data B are picture data, and the data C is video data. Alternatively, the data a is video data, the data B is picture data, and the data C is picture data.

The collection data table is a data table for storing collected data. The collected data includes, but is not limited to, picture data, video data, audio data, and text data.

Step S203, marking the data as collection data which is not transferred according to the transfer request;

wherein, the marking the data as the collection data which is not transferred according to the transfer request comprises:

and acquiring the data state of the data, setting the data state to be a non-transfer state according to the transfer request, and marking the data as the collection data which is not transferred.

And setting the state field of the collection data as an un-unloading field according to the unloading request, thereby setting the data state as an un-unloading state.

Wherein the status field is used to describe the status of the data. The status field is 0 to indicate that the data state is not transferred, the status field is 1 to indicate that the data state is transferred, the status field is 2 to indicate that the transfer fails, and the status field is 3 to indicate that retry is required.

Step S204, acquiring the address of the collected data, and downloading the collected data according to the retriable error list, the retriable times and the address of the collected data;

and S205, after the collection data is downloaded successfully, the collection data is transferred to a preset persistent file to store the collection data for a long time.

After the collection data is downloaded successfully, the collection data is transferred to a preset persistent file to store the collection data for a long time, and the method comprises the following steps:

The method comprises the steps of storing collected data, wherein the address of the collected data is used before the collected data is transferred and stored, and the new address of the collected data is used after the collected data is transferred and stored, so that the collected data cannot be checked by a user, and the user experience is improved.

And the address of the data is a cloud storage address.

Wherein, the data comprises any one or the combination of picture data, video data, audio data and character data.

The path of the persistent file may be set by the user, or may be a default of the system, which is not limited herein.

In this embodiment, after the collection data is successfully downloaded, the collection data is transferred to a preset persistent file to store the collection data for a long time.

Referring to fig. 3, fig. 3 is a flowchart of identifying a collection data address according to an embodiment of the present application, which is detailed as follows:

step S301, judging whether the address of the collected data is an HTTP address;

step S302, when the address of the collected data is an HTTP address, identifying the address as an absolute path, and downloading the collected data according to the retriable error list, the retriable times and the absolute path;

step S303, when the address of the collected data is not an HTTP address, identifying the address as a relative path, converting the relative path into the absolute path, and downloading the collected data according to the retry error list, the retry times and the absolute path.

In the embodiment, the collection data is directly downloaded through the absolute path, so that the invalid downloading amount and the invalid downloading times of the collection data are reduced, and the network resources consumed by blindly downloading the collection data are reduced, thereby improving the success rate and the efficiency of the collection data downloading.

Referring to fig. 4, fig. 4 is another flowchart illustrating the identifying of the collected data address according to an embodiment of the present application, which is detailed as follows:

step S401, when the address of the collected data is not an HTTP address, extracting a storage mark to be identified from the address of the collected data;

wherein the storage flag to be identified is extracted before a back slash or a forward slash in the address of the collection data.

The storage flag is a user-defined or system default, and is not limited herein.

Step S402, matching the storage mark to be identified with an identified storage mark stored locally, setting the cloud storage address corresponding to the identified storage mark as the cloud storage address corresponding to the storage mark to be identified according to the corresponding relation between the identified storage mark and the cloud storage address after the matching is successful;

and storing the corresponding relation between the identified storage mark and the cloud storage address locally.

And matching the storage mark to be identified with an identified storage mark stored locally, and if the matching is successful, setting the cloud storage address corresponding to the identified storage mark matched successfully as the cloud storage address corresponding to the storage mark to be identified.

For convenience of explanation, the following are exemplified:

the identified storage mark 1, the identified storage mark 2 and the identified storage mark 3 are stored locally, and the cloud storage addresses corresponding to the identified storage mark 1, the identified storage mark 2 and the identified storage mark 3 are stored locally.

And respectively matching the storage mark to be identified with the identified storage mark 1, the identified storage mark 2 and the identified storage mark 3.

And if the storage mark to be identified is successfully matched with the identified storage mark 1, setting the cloud storage address corresponding to the identified storage mark 1 as the cloud storage address corresponding to the storage mark to be identified.

And if the storage mark to be identified is successfully matched with the identified storage mark 2, setting the cloud storage address corresponding to the identified storage mark 2 as the cloud storage address corresponding to the storage mark to be identified.

And if the storage mark to be identified is successfully matched with the identified storage mark 3, setting the cloud storage address corresponding to the identified storage mark 3 as the cloud storage address corresponding to the storage mark to be identified.

Step S403, downloading the collected data according to the retriable error list, the retrial number, and the cloud storage address corresponding to the storage flag to be identified.

In the embodiment, the collection data is downloaded through the cloud storage address, so that the invalid downloading amount and the invalid downloading times of the collection data are reduced, and the network resources consumed by blindly downloading the collection data are reduced, so that the success rate and the efficiency of the collection data downloading are improved.

Referring to fig. 5, fig. 5 is a flowchart illustrating downloading favorite data according to an embodiment of the present application, which is detailed as follows:

step S501, if the collected data is not successfully downloaded, acquiring a downloading error according to an error information field in the return, and counting the downloading times;

step S502, judging whether the download error is in the retry error list, and judging whether the download times are not more than the retry times;

step S503, when the download error is in the retriable error list and the download number is not greater than the retry number, downloading the favorite data again until the download error is not in the retriable error list or the download number is greater than the retry number, and stopping downloading the favorite data.

In this embodiment, by defining the retriable error list and the retried times, the collection data can be downloaded according to the network condition, so that the network resources consumed by blindly downloading the collection data are reduced, and the downloading efficiency of the collection data is improved.

Referring to fig. 6, fig. 6 is a flowchart illustrating a data collection method according to an embodiment of the present application, which is detailed as follows:

for convenience of description, the following is specifically mentioned as an example of the picture dump:

starting, a user initiates a collection request, puts data of the collection request into a warehouse at the first time, judges whether the warehousing is successful, and if so, returns success; if not, returning to failure, and ending;

starting a picture unloading process;

and assembling the data with the status fields of 0 and 3 into a list, and initiating the unloading of the data when the length of the list is greater than 0. The status field is used to describe the status of the data. A status field of 0 indicates that the data state is not transferred, a status field of 1 indicates that the data state is transferred, a status field of 2 indicates that the transfer fails, and a status field of 3 indicates that retry is pending.

Judging whether the state field is not transferred;

judging whether the address of the picture is an absolute path or not; if yes, downloading the file; and if not, the address of the spliced picture is an absolute path.

Judging whether the downloading is successful; if yes, saving the file and writing the file back to the database, and ending; if not, it is determined whether the error is retriable, and the retry is enabled, and the retry count in the database is incremented by 1.

And judging whether the downloading times are greater than the retry times, if not, downloading the file, if so, marking the status field as unloading failure, and ending.

For convenience of explanation, the following are exemplified:

1. when a user finds data needing to be collected in the automatically collected data, a collection request is initiated, after the collection request is initiated, the data is added into a collection data table, the retention time of the data in the collection data table is read, a transfer request is initiated before the retention time is reached, and the status field of the data is marked to be 0 according to the transfer request so as to mark the data as the collection data in a non-transfer state;

2. judging whether the address of the collected data is an absolute path or not, wherein the judging method comprises the following steps: and judging whether http characters exist in the first 28 characters of the address in sequence, and whether the colon + the double-slash + [0 ] 256] number + [0 ] 256] number + the colon + [0-65535] number exists, if so, the address is considered to be an absolute path, and if not, the address is considered to be a relative path.

3. When the address of the collection data is an absolute path, the collection data is directly downloaded, after the collection data is successfully downloaded, the collection data is stored in a localized persistent file of a specified path, and the position of the stored file is stored in a collection data table. If the file downloading fails, judging whether the downloading error is in the retry error list, if the downloading error is in the retry error list, judging whether the retry frequency reaches the upper limit, if so, marking the status field as 2 to indicate the unloading failure, if not, marking the status field as 3 to indicate the retrying, and adding 1 to the downloading frequency.

5. If the path is not an absolute path, the path is a relative path, the path is converted into the absolute path, the conversion method is to take a storage mark before the first "/" of the path, judge whether a cloud storage address corresponding to the storage mark exists, if so, directly download the collected data through the cloud storage address, and the download process is consistent with the absolute path download process.

In the embodiment, by defining the retriable error list and the retriable times, the collected data can be downloaded according to the network condition, so that the network resource consumed by blindly downloading the collected data is reduced, and the downloading efficiency of the collected data is improved.

Referring to FIG. 7, FIG. 7 is a block diagram of a data collection system provided in an embodiment of the present application, it should be noted that the data collection system 700 shown in FIG. 7 is only an example, and should not bring any limitation to the function and the scope of the embodiment of the present application, and the following is detailed as follows:

data collection system 700 includes:

a determining module 701, configured to determine a retriable error list and a retry number, where the retriable error list includes at least one of a network failure and a request failure;

a retention time reading module 702, configured to read a retention time of data in a collection data table, and initiate a copy-over request before the retention time arrives;

a collected data marking module 703, configured to mark, according to the dump request, the collected data as collected data that is not dumped;

a collected data downloading module 704, configured to obtain an address of the collected data, and download the collected data according to the retry error list, the retry number, and the address of the collected data;

the collection data unloading module 705 is configured to unload the collection data into a preset persistent file after the collection data is successfully downloaded, so as to store the collection data for a long time.

In this embodiment, the system is substantially provided with a plurality of modules for executing the method in the above embodiment, and specific functions and technical effects may be obtained by referring to the above method embodiment, which is not described herein again.

Referring to fig. 8, fig. 8 is a schematic structural diagram of an electronic device provided in an embodiment of the present application, and it should be noted that the electronic device 800 shown in fig. 8 is only an example, and should not bring any limitation to the function and the application range of the embodiment of the present application.

As shown in fig. 8, an electronic device 800 includes a processor 801, a memory 802, and a communication bus 803;

the communication bus 803 is used to connect the processor 801 and the memory 802;

the processor 801 is adapted to execute a computer program stored in the memory 802 to implement the method as described in one or more of the above embodiments.

Embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, the computer program being configured to cause a computer to execute the method according to any one of the above embodiments.

Embodiments of the present application also provide a non-transitory readable storage medium, where one or more modules (programs) are stored in the storage medium, and when the one or more modules are applied to a device, the device may execute instructions (instructions) included in an embodiment of the present application.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The above embodiments are merely illustrative of the principles and utilities of the present application and are not intended to limit the application. Any person skilled in the art can modify or change the above-described embodiments without departing from the spirit and scope of the present application. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical concepts disclosed in the present application shall be covered by the claims of the present application.

Claims

1. A method of data collection, the method comprising:

determining a retry error list and a number of retries, the retry error list including at least one of a network failure and a request failure;

and after the collection data is downloaded successfully, the collection data is transferred to a preset persistent file so as to store the collection data for a long time.

2. The method of claim 1, wherein said marking said data as uncapped collection data according to said load request comprises:

and acquiring the execution state of the data according to the unloading request, acquiring the value range of a mark field of the data when the execution state of the data is in an un-invoked state, changing the value of the mark field according to the value range of the mark field, and setting the data state to be in an un-unloading state according to the changed value of the mark field so as to mark the data as the collection data which is not unloaded.

3. The method of claim 1, wherein obtaining the address of the collection data, downloading the collection data based on the retriable error list, the number of retries, and the address of the collection data, comprises:

judging whether the address of the collected data is an HTTP address or not;

4. The method of claim 3, wherein said identifying said address as a relative path when said address of said favorite data is not an HTTP address, converting said relative path to said absolute path, downloading said favorite data based on said retry number and said absolute path of said retriable error list comprises:

5. The method of claim 1, wherein after the successful downloading of the collection data, the unloading of the collection data into a preset persistent file for long-term storage of the collection data comprises:

6. The method of claim 1, wherein after obtaining the address of the favorite data, downloading the favorite data based on the retriable error list, the number of retries, and the address of the favorite data, the method further comprises:

7. The method of any one of claims 1 to 6, wherein after said downloading said collection data is successful, unloading said collection data into a preset persistent file for long-term storage of said collection data, said method further comprising:

and acquiring the data volume of the collected data, if the data volume of the collected data is larger than a preset data volume threshold value, generating a thumbnail of the collected data, and uploading the collected data and the thumbnail of the collected data to a preset backup server.

8. A data collection system, the system comprising:

a determining module for determining a retriable error list and a number of retries, the retriable error list including at least one of a network failure and a request failure;

and the collection data unloading module is used for unloading the collection data to a preset persistent file after the collection data is successfully downloaded, so as to store the collection data for a long time.

9. An electronic device comprising a processor, a memory, and a communication bus;

the communication bus is used for connecting the processor and the memory;

the processor is configured to execute a computer program stored in the memory to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, having stored thereon a computer program for causing a computer to perform the method of any one of claims 1-7.