CN110909055A - Data processing method, data processing device, electronic equipment and storage medium - Google Patents
Data processing method, data processing device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110909055A CN110909055A CN201911129851.5A CN201911129851A CN110909055A CN 110909055 A CN110909055 A CN 110909055A CN 201911129851 A CN201911129851 A CN 201911129851A CN 110909055 A CN110909055 A CN 110909055A
- Authority
- CN
- China
- Prior art keywords
- file
- abnormal
- format conversion
- processed
- priority
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention relates to the field of communication, and discloses a data processing method, a data processing device, electronic equipment and a storage medium. The data processing method is used for processing the abnormal data extracted in the ETL process and comprises the following steps: acquiring abnormal data to be processed, wherein the abnormal data comprises at least one abnormal file; determining the priority of each abnormal file in the abnormal data; selecting an abnormal file with the highest priority as a target file, and performing format conversion on the target file; if the format conversion is successful, sending a file obtained after the format conversion; if the format conversion fails, judging whether the processed times of the target file reach a first preset threshold value, if so, discarding the target file, and if not, updating the processed times of the target file and reducing the priority of the target file, so that the efficiency of performing format conversion on data during ETL is improved, and computer hardware resources are saved.
Description
Technical Field
The embodiment of the invention relates to the field of communication, in particular to a data processing method.
Background
ETL (extract-Transformation-Loading) is a process of Loading data of a business system into a data warehouse after Extraction and format conversion, and aims to integrate scattered, disordered and non-uniform data in an enterprise to provide an analysis basis for enterprise decision making.
However, the inventors found that at least the following problems exist in the prior art: in the process of converting the format of the data, the data which is failed to be converted is repeatedly processed for many times, so that the efficiency is low and the hardware resources of the computer are wasted.
Disclosure of Invention
The embodiment of the invention aims to provide a data processing method, a data processing device, electronic equipment and a storage medium, so that the efficiency of format conversion of data in ETL is improved, and computer hardware resources are saved.
In order to solve the above technical problem, an embodiment of the present invention provides a data processing method, including: acquiring abnormal data to be processed, wherein the abnormal data comprises at least one abnormal file; determining the priority of each abnormal file in the abnormal data; selecting an abnormal file with the highest priority as a target file, and performing format conversion on the target file; if the format conversion is successful, sending a file obtained after the format conversion; if the format conversion fails, judging whether the processed times of the target file reach a first preset threshold value, if so, discarding the target file, and if not, updating the processed times of the target file and reducing the priority of the target file.
An embodiment of the present invention further provides a data processing apparatus, including: the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring abnormal data to be processed, and the abnormal data comprises at least one abnormal file; the determining module is used for determining the priority of each abnormal file in the abnormal data; the format conversion module is used for selecting the abnormal file with the highest priority as a target file and carrying out format conversion on the target file; the sending module is used for sending the file obtained after the format conversion when the format conversion is successful; and the judging module is used for judging whether the processed times of the target file reach a first preset threshold value or not when the format conversion fails, discarding the target file if the processed times of the target file reach the first preset threshold value, and updating the processed times of the target file and reducing the priority of the target file if the processed times of the target file do not reach the first preset threshold value.
An embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data processing method.
Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which when executed by a processor implements the above-described data processing method.
Compared with the prior art, in the data processing process, the priority of each abnormal file in the abnormal data is determined, the abnormal data is processed according to the priority sequence, the priority of the abnormal file is reduced after each processing failure, and the abnormal file with low success rate is processed through hysteresis processing, so that the data processing efficiency is improved; and discarding the abnormal files with the processed times reaching the preset threshold, and stopping resource investment on the abnormal files with the processing success rate lower than the lower limit value, so that the hardware resources of the computer are saved.
In addition, the determining the priority of each exception file of the exception data includes: if the abnormal file is not marked with the priority, determining the priority of the abnormal file according to the parameters of the abnormal file; and if the abnormal file is marked with the priority, acquiring the marked priority of the abnormal file. And for the abnormal files without marked priorities, setting the priorities of the abnormal files according to the parameters of the abnormal files, so that the priority setting of the abnormal files has rationality.
In addition, the parameters of the exception file include: any one or any combination of file update time, file size, and file type. And determining the priority of the abnormal file according to the file updating time, the file size or the file type, so that the set priority of the abnormal file is more reasonable.
In addition, the data processing method further includes: acquiring a file to be processed, and performing format conversion on the file to be processed; if the format conversion is successful, sending a file obtained after the format conversion; if the format conversion fails, judging whether the processed times of the target reach a second preset threshold value, if so, marking the file to be processed as the abnormal data to be processed, and if not, updating the processed times of the file to be processed. With the arrangement, the processing method for normal data during ETL is provided, and files with the processed times reaching the threshold value in the normal data are classified into abnormal data, so that the processing efficiency is prevented from being lowered due to repeated processing of files with conversion failure in the processing process.
In addition, the file obtained after the format conversion is sent includes: sending the file obtained after the format conversion to a virtual file system through a receiving interface on the virtual file system, and sending the file obtained after the format conversion through a sending interface of the virtual file system; wherein the receive interface and the transmit interface use different communication protocols. According to the arrangement, the file obtained after format conversion is sent through the virtual file system, and the communication protocol is switched through the receiving interface and the sending interface on the virtual file system, so that code rewriting and supplement are not needed, and the labor cost is saved.
In addition, the receiving interface and the sending interface use different communication protocols, including: the transmit interface and the receive interface use different application layer protocols. The virtual file system supports switching of application layer protocols without code rewriting and supplementing, and labor cost is saved.
In addition, the receiving interface and the sending interface use different communication protocols, including: the transmit interface and the receive interface use different network layer protocols. The virtual file system supports switching of network layer protocols without code rewriting and supplementing, and labor cost is saved.
Drawings
One or more embodiments are illustrated by the corresponding figures in the drawings, which are not meant to be limiting.
FIG. 1 is a schematic flow diagram of a data processing method according to a first embodiment of the invention;
FIG. 2 is a schematic flow chart of a data processing method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of a multi-thread execution multi-pipeline according to a second embodiment of the present invention;
FIG. 4 is a schematic flow chart diagram of a data processing method according to a third embodiment of the invention;
FIG. 5 is a schematic configuration diagram of a data processing apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
In the communication service industry, after a gateway or a set top box is installed for a user, in order to better provide services for the user and accurately and quickly solve the problem of the user, a soft probe plug-in is usually built in the gateway or the set top box of the user for collecting running state information of the gateway or the set top box, and then the running state information is reported to a soft probe platform, so that data often has multiple platform requirements, and in order to reduce transmission pressure, the format is simplified, for example, JSON-format data is converted into data with a separator of "|". The whole process of extracting data through the soft probe plug-in, performing format conversion on the data and sending the converted data to the soft probe platform is an ETL (extract-Transformation-Loading) process.
The first embodiment of the invention relates to a data processing method, which is used for processing abnormal data extracted in an ETL process, and the core of the embodiment lies in acquiring abnormal data to be processed, wherein the abnormal data comprises at least one abnormal file; determining the priority of each abnormal file in the abnormal data; selecting an abnormal file with the highest priority as a target file, and performing format conversion on the target file; if the format conversion is successful, sending a file obtained after the format conversion; if the format conversion fails, judging whether the processed times of the target file reach a first preset threshold value, if so, discarding the target file, and if not, updating the processed times of the target file and reducing the priority of the target file, so that the efficiency of performing format conversion on data during ETL is improved, and computer hardware resources are saved. The following describes implementation details of the data processing method of the present embodiment in detail, and the following is only provided for easy understanding and is not necessary for implementing the present embodiment.
A flow chart of the data processing method in the present embodiment is schematically shown in fig. 1:
step 101: obtaining exception data to be processed, wherein the exception data comprises at least one exception file.
Specifically, the abnormal data to be processed is generally data in which the number of format conversion failures in the ETL process reaches a preset threshold (generally regarded as 1 by default).
Step 102: and determining the priority of each abnormal file in the abnormal data.
Specifically, determining the priority of each exception file of the exception data comprises: if the abnormal file is not marked with the priority, determining the priority of the abnormal file according to the parameters of the abnormal file; and if the abnormal file is marked with the priority, acquiring the marked priority of the abnormal file.
Wherein, the parameters of the abnormal file comprise: any one or any combination of file update time, file size, and file type.
Preferably, for the abnormal file without marked priority, the priority is determined according to the file updating time of the abnormal file. Generally, the earlier the file update time is, the higher the priority of the abnormal file which is longer than the current time is, and after each processing of the abnormal file, the file update time is updated, and the priority of the abnormal file is changed along with the change of the file update time. The priority of the abnormal file is determined in this way, and the scheduling performance can be optimized.
Step 103: and selecting the abnormal file with the highest priority as a target file, and performing format conversion on the target file.
Specifically, in this step, the target File is converted from the JSON format to the custom format of the soft probe platform, and in practical applications, the target File whose original format is other than the JSON format may also be subjected to format conversion, such as File, Zip, Jar, Tar, and GZIP formats.
Step 104: and judging whether the format conversion is successful, if so, entering the step 105, and if not, entering the step 106.
Step 105: and sending the file obtained after the format conversion.
Specifically, the file obtained after format conversion is sent to the soft probe platform.
Step 106: and judging whether the processed times of the target file reach a first preset threshold value or not, if so, entering a step 107, and if not, entering a step 108.
Step 107: and discarding the target file.
Specifically, for an abnormal file whose processed times reach a first preset threshold, discarding (or marking as an unmanageable file) is performed, because the more the abnormal file is processed, that is, the more the processing failure times, the greater the possibility that the abnormal file has a problem is, and by discarding the abnormal file whose processed times reach the threshold, the resource investment on the file with the problem is reduced.
Step 108: and updating the processed times of the target file and reducing the priority of the target file.
Specifically, since the more the number of times the exception file has been processed indicates that the exception file has a higher possibility of being problematic, by lowering the priority of the exception file after each processing failure, the exception file with a lower possibility of being problematic is preferentially acquired in step 103, the success rate of completing the processing is relatively high, and thus the processing efficiency can be improved.
In the present embodiment, it is preferable that the process returns to step 101 after step 108 is executed.
Compared with the prior art, in the data processing process, the priority of each abnormal file in the abnormal data is determined, the abnormal data is processed according to the priority sequence, the priority of the abnormal file is reduced after each processing failure, and the abnormal file with low success rate is processed through hysteresis processing, so that the data processing efficiency is improved; and discarding the abnormal files with the processed times reaching the preset threshold, and stopping resource investment on the abnormal files with the processing success rate lower than the lower limit value, so that the hardware resources of the computer are saved.
A second embodiment of the present invention relates to a data processing method. The second embodiment is substantially the same as the first embodiment except that the second embodiment further provides a method of processing source data, including: acquiring a file to be processed, and performing format conversion on the file to be processed; if the format conversion is successful, sending a file obtained after the format conversion; if the format conversion fails, judging whether the processed times of the target reach a second preset threshold value, if so, marking the file to be processed as the abnormal data to be processed, and if not, updating the processed times of the file to be processed. .
A flow diagram of the data processing method in this embodiment is shown in fig. 2, and specifically includes:
step 201: and acquiring a file to be processed, and performing format conversion on the file to be processed.
Specifically, the file to be processed is obtained, namely, the file to be processed is extracted from the source data to be processed, and format conversion is performed on the file to be processed.
Step 202: and judging whether the format conversion is successful.
Step 203: and sending the file obtained after the format conversion.
Specifically, the file obtained after format conversion is sent to the soft probe platform.
Step 204: and judging whether the processed times of the file to be processed reach a second preset threshold value or not.
Step 205: and marking the file to be processed as the abnormal data to be processed.
Step 206: and updating the processed times of the file to be processed.
Specifically, in steps 204 to 206, the file to be processed, which has been processed to the second preset threshold, is marked as the abnormal data to be processed, that is, the file is processed by the circuit unit responsible for processing the source data, and the data is divided into the circuit units responsible for processing the abnormal data by the marking. Therefore, the circuit unit responsible for processing the source data can be ensured, other normal files cannot be processed in a lagging mode due to the fact that a certain abnormal file is repeated, and processing efficiency is improved.
In the present embodiment, it is preferable that the process returns to step 201 after step 206 is executed.
Step 207: obtaining exception data to be processed, wherein the exception data comprises at least one exception file.
Step 208: and determining the priority of each abnormal file in the abnormal data.
Step 209: and selecting the abnormal file with the highest priority as a target file, and performing format conversion on the target file.
Step 210: and judging whether the format conversion is successful, if so, entering the step 211, and if not, entering the step 212.
Step 211: and sending the file obtained after the format conversion.
Step 212: and judging whether the processed times of the target file reach a first preset threshold value, if so, entering a step 213, and if not, entering a step 214.
Step 213: and discarding the target file.
Step 214: and updating the processed times of the target file and reducing the priority of the target file.
In this embodiment, it is preferable that the process returns to step 207 after step 214 is executed.
It should be noted that, in this embodiment, steps 201 to 202, i.e. the step of performing format conversion on the file to be processed (hereinafter referred to as step a), step 203, i.e. the step of sending the file with successful format conversion (hereinafter referred to as step B), and steps 204 to 205, i.e. the step of processing the file with the processed number of times reaching the threshold value (hereinafter referred to as step C), and step 206, i.e. the step of processing the file with the format conversion failure and the processed number of times not reaching the threshold value (hereinafter referred to as step D), when steps a, B, C and D are executed, pipeline technology is adopted, four circuit units respectively responsible for steps A, B, C and D are provided in the central processing unit CPU, and when data processing is performed, the four circuit units respectively and simultaneously control and execute the respectively responsible steps (for example, the circuit unit responsible for step a finishes processing the file 1, the file 1 is processed by the circuit unit responsible for step B, and C, while the circuit unit in charge of the step B processes the file 1, the circuit unit in charge of the step A processes the file 2, and so on), so that the steps A, B, C and D can be completed in one CPU clock cycle, and the operation speed of the CPU is improved.
Similarly, in this embodiment, steps 207 to 210 are a step of obtaining a target file and performing format conversion on the target file (hereinafter referred to as step E), step 211 is a step of sending the target file after format conversion (hereinafter referred to as step F), steps 212 to 213 are a step of processing the file whose processed times reaches the threshold (hereinafter referred to as step G), and step 214 is a step of processing the file whose format conversion fails and whose processed times does not reach the threshold (hereinafter referred to as step H), where step F and step B may be handled by the same circuit unit of the CPU, and when steps E, F, G and H are executed, a pipeline technique is adopted, and specific details are the same as those of steps A, B, C and D, which are not described herein again.
Further, in this embodiment, as shown in fig. 3, there is no restriction on the precedence relationship between step 201 to step 206 (excluding step 203), i.e. the normal pipeline, step 207 to step 214 (excluding step 211), i.e. the abnormal pipeline, and step 203 and step 211, i.e. the transmission pipeline, and in practical applications, the above steps can be executed simultaneously by multiple threads.
Compared with the prior art, the embodiment of the invention provides the processing method of the normal data during ETL, and the files with the processing times reaching the threshold value in the normal data are classified into the abnormal data, so that the processing efficiency is prevented from being lowered due to repeated processing of the files with failed format conversion in the processing process.
A third embodiment of the present invention relates to a data processing method. The third embodiment is substantially the same as the first embodiment, except that the third embodiment transmits a file obtained by format conversion, and includes: and sending the file obtained after the format conversion to a virtual file system through a receiving interface on the virtual file system, and sending the file obtained after the format conversion through a sending interface of the virtual file system, wherein the receiving interface and the sending interface use different communication protocols.
As shown in fig. 4, the data processing method in the present embodiment specifically includes:
step 301: obtaining exception data to be processed, wherein the exception data comprises at least one exception file.
Step 302: and determining the priority of each abnormal file in the abnormal data.
Step 303: and selecting the abnormal file with the highest priority as a target file, and performing format conversion on the target file.
Step 304: it is determined whether the format conversion was successful, and if so, step 305 is entered, and if not, step 306 is entered.
Step 305: and transmitting the file obtained after the format conversion through a virtual file system.
Specifically, the file obtained after the format conversion is sent to the virtual file system through a receiving interface on the virtual file system, and the file obtained after the format conversion is sent through a sending interface of the virtual file system. The receiving interface and the sending interface use different communication protocols, including: the transmit interface and the receive interface use different application layer protocols. Application layer protocols supported by the receiving interface and the transmitting interface include FTP (File Transfer Protocol), FTPs (encrypted FTP), SFTP (secure File Transfer Protocol), HTTP (hypertext Transfer Protocol), and HTTPs (hypertext Transfer security Protocol over secure session layer). For example, the application layer protocol used by the receiving interface to receive the file is FTP, and since the application layer protocol used by the soft probe platform is SFTP, the virtual file system receives the file through the receiving interface. The protocol seamless switching can be realized through a transmitting interface supporting SFTP.
In addition, the receiving interface and the sending interface use different communication protocols, and the method further comprises the following steps: the transmit interface and the receive interface use different network layer protocols. The network protocols supported by the transmission interface include IPv4 (Internet Protocol Version 4) and IPv6 (Internet Protocol Version 6, Internet Protocol Version 6). And the seamless switching of the network layer protocol can be realized by transmitting the file through the virtual file system.
Step 306: and judging whether the processed times of the target file reach a first preset threshold value, if so, entering a step 307, and if not, entering a step 308.
Step 307: and discarding the target file.
Step 308: and updating the processed times of the target file and reducing the priority of the target file.
Note that, the same improvement of the step of transmitting the format-converted file in the present embodiment applies to step 203 in the second embodiment.
In the present embodiment, it is preferable that the process returns to step 301 after step 308 is executed.
Compared with the prior art, the embodiment of the invention sets the file obtained after the virtual file system sends the format conversion, and switches the communication protocol through the receiving interface and the sending interface on the virtual file system without rewriting and supplementing codes, thereby saving the labor cost.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the same logical relationship is included, which are all within the protection scope of the present patent; it is within the scope of the patent to add insignificant modifications to the algorithms or processes or to introduce insignificant design changes to the core design without changing the algorithms or processes.
A fourth embodiment of the present invention relates to a network slice selection device including: the specific structure of the acquiring module 401, the determining module 402, the format converting module 403, the sending module 404, and the determining module 405 is as shown in fig. 5:
an obtaining module 401, configured to obtain exception data to be processed, where the exception data includes at least one exception file;
a determining module 402, configured to determine a priority of each exception file in the exception data;
a format conversion module 403, configured to select an abnormal file with the highest priority as a target file, and perform format conversion on the target file;
a sending module 404, configured to send a file obtained after format conversion when the format conversion is successful;
a determining module 405, configured to determine, when the format conversion fails, whether the processed number of times of the target file reaches a first preset threshold, if so, discard the target file, and if not, update the processed number of times of the target file and reduce the priority of the target file.
It should be understood that this embodiment is an example of the apparatus corresponding to the first embodiment, and may be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the first embodiment.
It should be noted that each module referred to in this embodiment is a logical module, and in practical applications, one logical unit may be one physical unit, may be a part of one physical unit, and may be implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, elements that are not so closely related to solving the technical problems proposed by the present invention are not introduced in the present embodiment, but this does not indicate that other elements are not present in the present embodiment.
A fifth embodiment of the present invention relates to an electronic device, as shown in fig. 6, including at least one processor 501; and a memory 502 communicatively coupled to the at least one processor 501; the memory 502 stores instructions executable by the at least one processor 501, and the instructions are executed by the at least one processor 501, so that the at least one processor 501 can execute the data processing method embodiments described above.
The memory 502 and the processor 501 are coupled by a bus, which may include any number of interconnected buses and bridges that couple one or more of the various circuits of the processor 501 and the memory 502 together. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 501 is transmitted over a wireless medium through an antenna, which further receives the data and transmits the data to the processor 501.
The processor 501 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 502 may be used to store data used by processor 501 in performing operations.
The embodiment of the invention also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the data processing method embodiment when being executed by a processor.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (10)
1. A data processing method, comprising:
acquiring abnormal data to be processed, wherein the abnormal data comprises at least one abnormal file;
determining the priority of each abnormal file in the abnormal data;
selecting an abnormal file with the highest priority as a target file, and performing format conversion on the target file;
if the format conversion is successful, sending a file obtained after the format conversion;
if the format conversion fails, judging whether the processed times of the target file reach a first preset threshold value, if so, discarding the target file, and if not, updating the processed times of the target file and reducing the priority of the target file.
2. The data processing method of claim 1, wherein the determining the priority of each exception file of the exception data comprises:
if the abnormal file is not marked with the priority, determining the priority of the abnormal file according to the parameters of the abnormal file;
and if the abnormal file is marked with the priority, acquiring the marked priority of the abnormal file.
3. The data processing method of claim 1, wherein the parameters of the exception file include: any one or any combination of file update time, file size, and file type.
4. The data processing method of claim 1, further comprising:
acquiring a file to be processed, and performing format conversion on the file to be processed;
if the format conversion is successful, sending a file obtained after the format conversion;
if the format conversion fails, judging whether the processed times of the target reach a second preset threshold value, if so, marking the file to be processed as the abnormal data to be processed, and if not, updating the processed times of the file to be processed.
5. The data processing method according to claim 1 or 4, wherein the sending the format-converted file comprises:
sending the file obtained after the format conversion to a virtual file system through a receiving interface on the virtual file system, and sending the file obtained after the format conversion through a sending interface of the virtual file system;
wherein the receive interface and the transmit interface use different communication protocols.
6. The data processing method of claim 5, wherein the receiving interface and the sending interface use different communication protocols, comprising:
the transmit interface and the receive interface use different application layer protocols.
7. The data processing method of claim 5, wherein the receiving interface and the sending interface use different communication protocols, comprising:
the transmit interface and the receive interface use different network layer protocols.
8. A data processing apparatus, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring abnormal data to be processed, and the abnormal data comprises at least one abnormal file;
the determining module is used for determining the priority of each abnormal file in the abnormal data;
the format conversion module is used for selecting the abnormal file with the highest priority as a target file and carrying out format conversion on the target file;
the sending module is used for sending the file obtained after the format conversion when the format conversion is successful;
and the judging module is used for judging whether the processed times of the target file reach a first preset threshold value or not when the format conversion fails, discarding the target file if the processed times of the target file reach the first preset threshold value, and updating the processed times of the target file and reducing the priority of the target file if the processed times of the target file do not reach the first preset threshold value.
9. An electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing method of any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911129851.5A CN110909055A (en) | 2019-11-18 | 2019-11-18 | Data processing method, data processing device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911129851.5A CN110909055A (en) | 2019-11-18 | 2019-11-18 | Data processing method, data processing device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110909055A true CN110909055A (en) | 2020-03-24 |
Family
ID=69817795
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911129851.5A Pending CN110909055A (en) | 2019-11-18 | 2019-11-18 | Data processing method, data processing device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110909055A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113918770A (en) * | 2021-12-16 | 2022-01-11 | 上海冰鉴信息科技有限公司 | Method and device for converting character string and time field |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0332351A2 (en) * | 1988-03-09 | 1989-09-13 | Fujitsu Limited | Direct memory access controller |
| CN102479113A (en) * | 2010-11-30 | 2012-05-30 | 中国移动通信集团黑龙江有限公司 | Abnormal self-adapting processing method and system |
| CN106933872A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method and device that cloud storage service is accessed by traditional file systemses interface |
| CN110347735A (en) * | 2019-06-17 | 2019-10-18 | 平安普惠企业管理有限公司 | The data forwarding of configurationization mends processing method, device and readable storage medium storing program for executing extremely |
-
2019
- 2019-11-18 CN CN201911129851.5A patent/CN110909055A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0332351A2 (en) * | 1988-03-09 | 1989-09-13 | Fujitsu Limited | Direct memory access controller |
| CN102479113A (en) * | 2010-11-30 | 2012-05-30 | 中国移动通信集团黑龙江有限公司 | Abnormal self-adapting processing method and system |
| CN106933872A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method and device that cloud storage service is accessed by traditional file systemses interface |
| CN110347735A (en) * | 2019-06-17 | 2019-10-18 | 平安普惠企业管理有限公司 | The data forwarding of configurationization mends processing method, device and readable storage medium storing program for executing extremely |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113918770A (en) * | 2021-12-16 | 2022-01-11 | 上海冰鉴信息科技有限公司 | Method and device for converting character string and time field |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220394316A1 (en) | Message sending method and device, readable medium and electronic device | |
| CN110120882B (en) | Policy driving method and device | |
| US8898311B2 (en) | Data communication method and information processing device | |
| US11595964B2 (en) | Method and device for determining information domain value in DCI | |
| CN105814543A (en) | Power balancing to increase workload density and improve energy efficiency | |
| US10200240B2 (en) | Method for configuring an M2M system | |
| CN105052076B (en) | Network element management system and network element management method based on cloud computing | |
| US10033546B2 (en) | Method and system for reprogramming | |
| CN104378234A (en) | Cross-data-center data transmission processing method and system | |
| CN109582335A (en) | It is a kind of without interrupt storage cluster node online upgrading method, device and equipment | |
| US20190053169A1 (en) | Method for dynamic adaptation of a data rate | |
| WO2013169422A1 (en) | Communications management | |
| CN107925652B (en) | Fixed conversion method and device of transmission protocol based on message protocol identification | |
| US20170168473A1 (en) | Signalling of specification information on an industrial device | |
| CN110909055A (en) | Data processing method, data processing device, electronic equipment and storage medium | |
| CN106341433A (en) | Service system, signal server and intercom server | |
| CN112202896A (en) | Edge calculation method, frame, terminal and storage medium | |
| CN111315026B (en) | Channel selection method, device, gateway and computer readable storage medium | |
| CN103368862B (en) | Load balance dispatching method and load balance dispatching device | |
| CN102428673B (en) | For distributing control circuit and the method for orthogonal sequence | |
| CN112181594A (en) | Virtual machine live migration method, device, equipment and storage medium | |
| US20220019871A1 (en) | Method for Adapting a Software Application Executed in a Gateway | |
| CN105187244A (en) | Access management system of digital communication equipment supporting multiple management modes | |
| CN107147694B (en) | Information processing method and device | |
| CN111240867A (en) | Information communication system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200324 |