CN112685481A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN112685481A CN112685481A CN201910988064.XA CN201910988064A CN112685481A CN 112685481 A CN112685481 A CN 112685481A CN 201910988064 A CN201910988064 A CN 201910988064A CN 112685481 A CN112685481 A CN 112685481A
- Authority
- CN
- China
- Prior art keywords
- data
- export
- mode
- import
- calling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 100
- 230000008676 import Effects 0.000 claims abstract description 63
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000004590 computer program Methods 0.000 claims description 10
- 230000001360 synchronised effect Effects 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000007667 floating Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 21
- 230000008569 process Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 238000009795 derivation Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method and device, and relates to the technical field of computers. One specific implementation mode of the method comprises the steps of obtaining data to be processed, selecting a preset number of the data to be processed as sample data, and respectively calculating the processing time length of each sample data to obtain the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time. Therefore, the implementation mode of the invention can solve the problem that flexible data import and export aiming at different scenes cannot be realized in the prior art.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
In most enterprise-level applications, existing data with the same format is entered into a database, and the entry method can be roughly divided into two types: 1) manual entry, 2) data import. Then, extracting the data with the same format in the database is generally completed by data export. The manual entry mode is generally directed at a library writing process with a small data volume, and when the data volume is large, the data entry is completed by adopting an import mode. The data in the database is generally obtained by way of derivation.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
most of the existing import and export schemes basically adopt one mode in implementation, the data volume is generally estimated before the scheme is determined, and different implementations are selected according to the data volume. The data import or export amount is variable along with different scenes, and fixing a certain import and export scheme can reduce the flexibility of application, increase the resource consumption of a computer if the certain import and export scheme is fixed, and cause memory overflow to crash the computer if the certain import and export scheme is fixed.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method and apparatus, which can solve the problem in the prior art that flexible data import and export cannot be implemented for different scenes.
In order to achieve the above object, according to an aspect of the embodiments of the present invention, a data processing method is provided, including obtaining data to be processed, optionally selecting a preset number of data to be processed as sample data, and respectively calculating a processing time length of each sample data to obtain an average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time.
Optionally, based on the total processing duration, selecting a corresponding data importing and exporting manner for the data to be processed, including:
judging whether the total processing time length is smaller than a first threshold preset in a first time level range, and if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold preset in a second time level range, if so, selecting an asynchronous import and export mode, and otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit floating value; the second threshold in the second time level range is the sum of the second time level threshold and the second time level user tolerance float value.
Optionally, the asynchronous import mode includes:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data in a third-party database by calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database to acquire the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
Optionally, the asynchronous multithread import mode includes:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data to a third-party database in a multithreading mode by calling an implementation interface;
the asynchronous multithread export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database in a multithreading mode to obtain the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
In addition, according to an aspect of the embodiments of the present invention, there is provided a data processing apparatus, including a calculating module, configured to obtain data to be processed, optionally select a preset number of data to be processed as sample data, and calculate a processing duration of each sample data, respectively, to obtain an average processing duration of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and the processing module is used for selecting corresponding data import and export modes for the data to be processed based on the total processing time length.
Optionally, the processing module selects, based on the total processing duration, a corresponding data importing and exporting manner for the data to be processed, including:
judging whether the total processing time length is smaller than a first threshold preset in a first time level range, and if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold preset in a second time level range, if so, selecting an asynchronous import and export mode, and otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit floating value; the second threshold in the second time level range is the sum of the second time level threshold and the second time level user tolerance float value.
Optionally, the asynchronous import mode includes:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data in a third-party database by calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database to acquire the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
Optionally, the asynchronous multithread import mode includes:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data to a third-party database in a multithreading mode by calling an implementation interface;
the asynchronous multithread export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database in a multithreading mode to obtain the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any of the data processing embodiments described above.
According to another aspect of the embodiments of the present invention, there is also provided a computer readable medium, on which a computer program is stored, which when executed by a processor, implements the method according to any of the above-mentioned data processing-based embodiments.
One embodiment of the above invention has the following advantages or benefits: the method comprises the steps of respectively calculating the processing time length of each sample data by acquiring data to be processed and optionally selecting a preset number of data to be processed as the sample data to obtain the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time. Therefore, the invention can realize rapid and efficient data import and export.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a data processing method according to a first embodiment of the present invention
Fig. 2 is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention;
fig. 3 is a schematic diagram of a main flow of a data processing method according to a third embodiment of the present invention;
fig. 4 is a schematic diagram of a main flow of a data processing method according to a fourth embodiment of the present invention;
fig. 5 is a schematic diagram of a main flow of a data processing method according to a fifth embodiment of the present invention;
fig. 6 is a schematic diagram of a main flow of a data processing method according to a sixth embodiment of the present invention;
FIG. 7 is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention;
FIG. 8 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 9 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data processing method according to a first embodiment of the present invention, which may include:
step S101, data to be processed is obtained, and a preset number of data to be processed is selected as sample data.
In the examples, n optionally requiring treatmentsampleThe bar data, as sample data. The sample data volume is less than or equal to the total data to be processed and less than a preset sample number threshold. Preferably, nsample≤ndata&&nsample<5。
Step S102, the processing time length of each sample data is calculated respectively to obtain the average processing time length of the sample data.
In the embodiment, the processing time length of each sample data is calculated respectively, and then the processing time length of the average sample data is calculated
Step S103, obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data.
In an embodiment, the average processing duration is
Multiplied by the total amount of data to be processed ndataTo obtain a total processing time Ttotal。
And step S104, selecting corresponding data import and export modes for the data to be processed based on the total processing duration.
Preferably, when selecting corresponding data importing and exporting modes for the data to be processed, it may be determined whether the total processing time is less than a first threshold preset in a first time level range, and if so, selecting a synchronous importing and exporting mode. Otherwise, judging whether the total processing time length is smaller than a second threshold preset in a second time level range, if so, selecting an asynchronous import and export mode, and otherwise, selecting an asynchronous multithreading import and export mode.
Wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance float value. The second threshold in the second time level range is the sum of the second time level threshold and the second time level user tolerance float value.
In an embodiment, the first threshold within the first time level range may be expressed as thresholda+floataWherein threshold isaIs a first time level threshold, floataIs a first time level user tolerance limit float value. Preferably, the first time level may be in the order of seconds.
The second threshold value in the second time scale range may be denoted as thresholdb+floatbWherein threshold isbIs a second time level threshold, floatbIs a second time level user tolerance limit float value. Preferably, the second time level may be on the order of hours.
Therefore, when T istotal<thresholda+floataAnd selecting a synchronous leading-in and leading-out mode. When threshold is reacheda+floata≤Ttotal<thresholdb+floatbAnd selecting asynchronous import and export modes. When T istotal≥thresholdb+floatbAnd then, selecting an asynchronous multithread import and export mode.
Further, the asynchronous import mode includes:
and receiving a calling parameter sent by the foreground application, sending the calling parameter to the message consumer in a queue mode, and then accessing the cloud storage by the message consumer through the task parameter so as to download data. And acquiring download data, and storing the download data in a third-party database by calling an implementation interface.
The asynchronous export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. And based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database to acquire the export data of the tasks. And analyzing the exported data into a preset format file and outputting the file.
In addition, the asynchronous multithread import mode includes:
and receiving a calling parameter sent by the foreground application, sending the calling parameter to the message consumer in a queue mode, and then accessing the cloud storage by the message consumer through the task parameter so as to download data. And acquiring download data, and storing the download data to a third-party database in a multithreading mode by calling an implementation interface.
The asynchronous multithread export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. And based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database in a multithreading mode to obtain the export data of the tasks. And analyzing the exported data into a preset format file and outputting the file.
Therefore, the invention provides a data processing method which can autonomously switch different import and export schemes in the dimension of data size and time. And the data size is a key factor influencing the import or export efficiency, the data processing time length is calculated in a sampling mode, and which scheme should be selected is determined on the basis of the processing time length, so that the import or export efficiency is ensured, and the use experience of a user is improved.
Fig. 2 is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention, which may include:
step S201, acquiring data to be processed, and optionally selecting a preset number of data to be processed as sample data.
Step S202, the processing time length of each sample data is respectively calculated to obtain the average processing time length of the sample data.
Step S203, obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data.
Step S204, determining whether the total processing time is less than a first threshold preset in a first time level range, if so, performing step S205, otherwise, performing step S206.
Wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance float value.
Step S205, a synchronous import/export method is selected.
Step S206, determining whether the total processing time is less than a second threshold preset in a second time level range, if yes, performing step S207, otherwise, performing step S208.
Wherein the second threshold value in the second time level range is the sum of the second time level threshold value and the second time level user tolerance float value.
Step S207, an asynchronous import/export mode is selected.
In an embodiment, the asynchronous import mode includes:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data; and acquiring download data, and storing the download data in a third-party database by calling an implementation interface.
The asynchronous export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. And based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database to acquire the export data of the tasks. And analyzing the exported data into a preset format file and outputting the file.
And S208, selecting an asynchronous multithread import and export mode.
In an embodiment, the asynchronous multithreading import mode includes:
and receiving a calling parameter sent by the foreground application, sending the calling parameter to the message consumer in a queue mode, and then accessing the cloud storage by the message consumer through the task parameter so as to download data. And acquiring download data, and storing the download data to a third-party database in a multithreading mode by calling an implementation interface.
The asynchronous multithread export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. And based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database in a multithreading mode to obtain the export data of the tasks. And analyzing the exported data into a preset format file and outputting the file.
Fig. 3 is a schematic diagram of a main flow of a data processing method according to a third embodiment of the present invention, and a specific implementation process of an asynchronous import mode includes:
app a (namely foreground application) calls App b in an RPC mode, data are stored in cloud storage (cloud), and the foreground is responded after the two operations are completed, so that App a can operate other functions.
App b receives a call parameter (Task param) from App a, stores the call parameter into a database, sends the call parameter (Task param) to a message consumer MQ consumer in an MQ mode, and after receiving a message, the message consumer MQ consumer accesses the cloud storage (cluster) through the Task param and downloads the data in the cloud storage (cluster).
App b analyzes data downloaded from cloud storage (cloud), calls different implementation interfaces (BusinessLogicHandler) in App c through RPC, and stores the data in a database. In the process of calling App c, the state in the task table is changed at the same time.
Wherein, businessologichandler Class represents server registration for a managed code assembly that implements a business logic handler. The RPC is known as Remote Procedure Call, a request for services from a Remote computer program over a network.
Fig. 4 is a schematic diagram of a main flow of a data processing method according to a fourth embodiment of the present invention, and a specific implementation process of asynchronous derivation includes:
app a (namely foreground application) transmits information such as condition parameters in the export task to App b in an RPC mode, and stores the information such as the condition parameters in the database.
And (3) continuously scanning a Task table in real time by a Task Service in App b to obtain query parameters meeting conditions, and calling App c to acquire data in an RPC mode.
And a Data parser Data Intepenter in App b parses the Data into a required format file and outputs the file to a user.
Fig. 5 is a schematic diagram of a main flow of a data processing method according to a fifth embodiment of the present invention, and a specific implementation process of multithreaded asynchronous import:
app a (namely foreground application) calls App b in an RPC mode, data are stored in cloud storage (cloud), and the foreground is responded after the two operations are completed, so that App a can operate other functions.
App b receives a call parameter (Task param) from App a, stores the call parameter into a database, sends the call parameter (Task param) to a message consumer MQ consumer in an MQ mode, and after receiving a message, the message consumer MQ consumer accesses the cloud storage (cluster) through the Task param and downloads the data in the cloud storage (cluster).
App b analyzes data downloaded from cloud storage (cloud), calls different implementation interfaces (BusinessLogicHandler) in App c in a multithreading mode, and stores the data in a database. In the process of calling App c, the state in the task table is changed at the same time.
It can be seen that the multithreading asynchronous import mode establishes connection between different applications in an RPC mode, thereby realizing an asynchronous processing process. And storing the data to be processed on the cloud storage, and analyzing the data on the cloud storage in an asynchronous mode. And multithreading processing the analyzed data and storing the data into a database.
Fig. 6 is a schematic diagram of a main flow of a data processing method according to a sixth embodiment of the present invention, and a specific implementation process of multithread asynchronous derivation:
app a (namely foreground application) transmits information such as condition parameters in the export task to App b in an RPC mode, and stores the information such as the condition parameters in the database.
And a Task Service in the App b continuously scans a Task table in real time to obtain query parameters meeting conditions, and calls the App c to obtain data in a multithreading mode.
And a Data parser Data Intepenter in App b parses the Data into a required format file and outputs the file to a user.
Preferably, the sizes of the stored single data amounts are different, and there is a certain difference between different computer performances, so the thread number needs to be dynamically determined according to the actual calculation condition, and the thread number can be constrained according to the following formula:
wherein the threadwatingTimeWait time for a thread.
threadcpuTimeIs the thread CPU time duration.
NumsCPUThe number of the CPUs.
It can be seen that the multithreading asynchronous export scheme establishes connection between different applications in an RPC manner, thereby implementing an asynchronous processing procedure. And the data analyzer analyzes the data acquired according to the derived parameters in an asynchronous mode and summarizes and returns the analyzed data to the user.
In summary, the data processing method according to various embodiments of the present invention can adapt to the data volume that changes due to the change of the service, for example, the initial data volume is smaller, and as the data volume increases, corresponding data importing and exporting manners may be adopted to ensure the demand for service increase. Meanwhile, the data import and export modes can be dynamically changed when the variation range of the data size of the processed data is large.
Fig. 7 is a schematic diagram of main blocks of a data processing apparatus according to a first embodiment of the present invention, and as shown in fig. 7, the data processing apparatus 700 includes a calculation module 701 and a processing module 702. The calculation module 701 is configured to acquire data to be processed, select a preset number of data to be processed as sample data, and calculate a processing time length of each sample data respectively to obtain an average processing time length of the sample data; and obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data. The processing module 702 is configured to select a corresponding data importing and exporting manner for the data to be processed based on the total processing duration.
As another embodiment, the processing module 702 selects a corresponding data importing and exporting manner for the data to be processed based on the total processing time, including:
judging whether the total processing time length is smaller than a first threshold preset in a first time level range, and if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold preset in a second time level range, if so, selecting an asynchronous import and export mode, and otherwise, selecting an asynchronous multithreading import and export mode.
Wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance float value. The second threshold in the second time level range is the sum of the second time level threshold and the second time level user tolerance float value.
Preferably, the asynchronous import mode includes:
and receiving a calling parameter sent by the foreground application, sending the calling parameter to the message consumer in a queue mode, and then accessing the cloud storage by the message consumer through the task parameter so as to download data. And acquiring download data, and storing the download data in a third-party database by calling an implementation interface.
The asynchronous export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. And based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database to acquire the export data of the tasks. And analyzing the exported data into a preset format file and outputting the file.
Still preferably, the asynchronous multithread import mode includes:
and receiving a calling parameter sent by the foreground application, sending the calling parameter to the message consumer in a queue mode, and then accessing the cloud storage by the message consumer through the task parameter so as to download data. And acquiring download data, and storing the download data to a third-party database in a multithreading mode by calling an implementation interface.
The asynchronous multithread export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. And based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database in a multithreading mode to obtain the export data of the tasks. And analyzing the exported data into a preset format file and outputting the file.
It should be noted that the data processing method and the data processing apparatus according to the present invention have corresponding relation in the specific implementation contents, and therefore, the repeated contents are not described again.
Fig. 8 shows an exemplary system architecture 800 of a data processing method or data processing apparatus to which embodiments of the present invention may be applied.
As shown in fig. 8, the system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. The network 804 serves to provide a medium for communication links between the terminal devices 801, 802, 803 and the server 805. Network 804 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 801, 802, 803 to interact with a server 805 over a network 804 to receive or send messages or the like. The terminal devices 801, 802, 803 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 801, 802, 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 805 may be a server that provides various services, such as a back-office management server (for example only) that supports shopping-like websites browsed by users using the terminal devices 801, 802, 803. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the server 805, and accordingly, the data processing apparatus is generally disposed in the server 805.
It should be understood that the number of terminal devices, networks, and servers in fig. 8 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 9, shown is a block diagram of a computer system 900 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 9, the computer system 900 includes a Central Processing Unit (CPU)901 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM903, various programs and data necessary for the operation of the system 900 are also stored. The CPU901, ROM902, and RAM903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.
The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The above-described functions defined in the system of the present invention are executed when the computer program is executed by a Central Processing Unit (CPU) 901.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a computation module and a processing module. Wherein the names of the modules do not in some cases constitute a limitation of the module itself.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring data to be processed, selecting a preset number of data to be processed as sample data, and respectively calculating the processing time length of each sample data to obtain the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time.
According to the technical scheme of the embodiment of the invention, the problem that flexible data import and export aiming at different scenes cannot be realized in the prior art can be solved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring data to be processed, selecting a preset number of data to be processed as sample data, and respectively calculating the processing time length of each sample data to obtain the average processing time length of the sample data;
obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data;
and selecting corresponding data import and export modes for the data to be processed based on the total processing time.
2. The method of claim 1, wherein selecting corresponding data import and export methods for the data to be processed based on the total processing duration comprises:
judging whether the total processing time length is smaller than a first threshold preset in a first time level range, and if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold preset in a second time level range, if so, selecting an asynchronous import and export mode, and otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit floating value; the second threshold in the second time level range is the sum of the second time level threshold and the second time level user tolerance float value.
3. The method of claim 2, wherein the asynchronous import mode comprises:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data in a third-party database by calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database to acquire the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
4. The method of claim 2, wherein the asynchronous multithreading import mode comprises:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data to a third-party database in a multithreading mode by calling an implementation interface;
the asynchronous multithread export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database in a multithreading mode to obtain the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
5. A data processing apparatus, comprising:
the calculation module is used for acquiring data to be processed, selecting a preset number of data to be processed as sample data, and calculating the processing time length of each sample data to obtain the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data;
and the processing module is used for selecting corresponding data import and export modes for the data to be processed based on the total processing time length.
6. The apparatus of claim 5, wherein the processing module selects a corresponding data importing and exporting manner for the data to be processed based on the total processing duration, and the method comprises:
judging whether the total processing time length is smaller than a first threshold preset in a first time level range, and if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold preset in a second time level range, if so, selecting an asynchronous import and export mode, and otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit floating value; the second threshold in the second time level range is the sum of the second time level threshold and the second time level user tolerance float value.
7. The apparatus of claim 6, wherein the asynchronous import mode comprises:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data in a third-party database by calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database to acquire the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
8. The apparatus of claim 6, wherein the asynchronous multithreading import mode comprises:
receiving a calling parameter sent by a foreground application, sending the calling parameter to a message consumer in a queue mode, and then accessing cloud storage by the message consumer through a task parameter to download data;
acquiring download data, and storing the download data to a third-party database in a multithreading mode by calling an implementation interface;
the asynchronous multithread export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
based on the task table, inquiring the tasks meeting the condition parameters, and calling a third-party database in a multithreading mode to obtain the export data of the tasks;
and analyzing the exported data into a preset format file and outputting the file.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-4.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910988064.XA CN112685481B (en) | 2019-10-17 | 2019-10-17 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910988064.XA CN112685481B (en) | 2019-10-17 | 2019-10-17 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112685481A true CN112685481A (en) | 2021-04-20 |
| CN112685481B CN112685481B (en) | 2023-11-03 |
Family
ID=75444740
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910988064.XA Active CN112685481B (en) | 2019-10-17 | 2019-10-17 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112685481B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282456A (en) * | 2021-05-24 | 2021-08-20 | 北京京东振世信息技术有限公司 | Data processing method and device |
| CN113704269A (en) * | 2021-09-02 | 2021-11-26 | 京东城市(北京)数字科技有限公司 | Data processing method, system, storage medium and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657123A (en) * | 2013-11-21 | 2015-05-27 | 航天信息股份有限公司 | Method for dynamically exporting files |
| CN105894163A (en) * | 2014-12-17 | 2016-08-24 | 中信天津外包服务有限公司 | One-stop full-process archive information management method |
| CN108228752A (en) * | 2017-12-21 | 2018-06-29 | 中国联合网络通信集团有限公司 | Data full dose deriving method, data distribution device and data export node |
| CN108346106A (en) * | 2018-02-23 | 2018-07-31 | 平安科技（深圳）有限公司 | Bill input method, system, optical character identification server and storage medium |
| CN108874894A (en) * | 2018-05-21 | 2018-11-23 | 平安科技（深圳）有限公司 | Crosstab deriving method, device, computer equipment and storage medium |
| CN110019293A (en) * | 2017-09-07 | 2019-07-16 | 北京国双科技有限公司 | List data deriving method and device based on impala database |
| CN110020646A (en) * | 2019-04-16 | 2019-07-16 | 恒生电子股份有限公司 | Archive method, apparatus, electronic equipment and storage medium |
-
2019
- 2019-10-17 CN CN201910988064.XA patent/CN112685481B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657123A (en) * | 2013-11-21 | 2015-05-27 | 航天信息股份有限公司 | Method for dynamically exporting files |
| CN105894163A (en) * | 2014-12-17 | 2016-08-24 | 中信天津外包服务有限公司 | One-stop full-process archive information management method |
| CN110019293A (en) * | 2017-09-07 | 2019-07-16 | 北京国双科技有限公司 | List data deriving method and device based on impala database |
| CN108228752A (en) * | 2017-12-21 | 2018-06-29 | 中国联合网络通信集团有限公司 | Data full dose deriving method, data distribution device and data export node |
| CN108346106A (en) * | 2018-02-23 | 2018-07-31 | 平安科技（深圳）有限公司 | Bill input method, system, optical character identification server and storage medium |
| WO2019161615A1 (en) * | 2018-02-23 | 2019-08-29 | 平安科技（深圳）有限公司 | Bill entry method, system, optical character recognition server and storage medium |
| CN108874894A (en) * | 2018-05-21 | 2018-11-23 | 平安科技（深圳）有限公司 | Crosstab deriving method, device, computer equipment and storage medium |
| CN110020646A (en) * | 2019-04-16 | 2019-07-16 | 恒生电子股份有限公司 | Archive method, apparatus, electronic equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282456A (en) * | 2021-05-24 | 2021-08-20 | 北京京东振世信息技术有限公司 | Data processing method and device |
| CN113282456B (en) * | 2021-05-24 | 2023-09-22 | 北京京东振世信息技术有限公司 | Data processing method and device |
| CN113704269A (en) * | 2021-09-02 | 2021-11-26 | 京东城市(北京)数字科技有限公司 | Data processing method, system, storage medium and electronic equipment |
| CN113704269B (en) * | 2021-09-02 | 2024-04-16 | 京东城市(北京)数字科技有限公司 | Data processing method, system, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112685481B (en) | 2023-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112527649A (en) | Test case generation method and device | |
| CN110706093A (en) | Accounting processing method and device | |
| CN111478781B (en) | Message broadcasting method and device | |
| CN112379982B (en) | Task processing method, device, electronic equipment and computer readable storage medium | |
| CN113010405A (en) | Application program testing method and device | |
| CN112596920A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN112685481B (en) | Data processing method and device | |
| CN111881329A (en) | Account balance management method and system | |
| CN114979295A (en) | Gateway management method and device | |
| CN112084042A (en) | Message processing method and device | |
| CN112398669A (en) | Hadoop deployment method and device | |
| CN113742389A (en) | Service processing method and device | |
| CN111062572B (en) | Task allocation method and device | |
| CN112667368A (en) | Task data processing method and device | |
| CN113779122B (en) | Method and device for exporting data | |
| CN114924937A (en) | Batch task processing method and device, electronic equipment and computer readable medium | |
| CN113127561B (en) | Method and device for generating service single number, electronic equipment and storage medium | |
| CN116861397A (en) | Request processing method, device, electronic equipment and computer readable medium | |
| CN113779018A (en) | Data processing method and device | |
| CN112784139A (en) | Query method, query device, electronic equipment and computer readable medium | |
| CN113572704A (en) | Information processing method, production end, consumption end and server | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| CN112860447A (en) | Interaction method and system between different applications | |
| CN113722193A (en) | Method and device for detecting page abnormity | |
| CN113626176A (en) | Service request processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |