CN112685481B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN112685481B CN112685481B CN201910988064.XA CN201910988064A CN112685481B CN 112685481 B CN112685481 B CN 112685481B CN 201910988064 A CN201910988064 A CN 201910988064A CN 112685481 B CN112685481 B CN 112685481B
- Authority
- CN
- China
- Prior art keywords
- data
- time length
- mode
- task
- processing time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 101
- 230000008676 import Effects 0.000 claims abstract description 60
- 238000000034 method Methods 0.000 claims abstract description 33
- 238000004590 computer program Methods 0.000 claims description 10
- 230000001360 synchronised effect Effects 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 21
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a data processing method and device, and relates to the technical field of computers. One specific embodiment of the method comprises the steps of obtaining data to be processed, optionally taking a preset number of data to be processed as sample data, and respectively calculating the processing time length of each piece of sample data to obtain the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time. Therefore, the embodiment of the invention can solve the problem that flexible data import and export can not be realized aiming at different scenes in the prior art.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
In most enterprise-level applications, existing data of the same format is entered into a database, and can be broadly divided into two entry modes: 1) Manually entered, 2) data import. Then, the extraction of the data with the same format in the database is generally completed by a data export mode. The manual input mode is generally aimed at a database writing process with smaller data volume, and the data input mode is adopted to finish the data input when the data volume is larger. For obtaining data in the database, a export mode is generally adopted.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
most of the existing import and export schemes basically adopt one of the modes in terms of implementation, and generally, the data volume is estimated before the scheme is determined, and different implementations are selected according to the data volume. The amount of data import or export is variable along with different scenes, the flexibility of application is reduced by fixing a certain import and export scheme, the resource consumption of a computer is increased if the data is light, and the memory overflows to cause the computer to crash if the data is heavy.
Disclosure of Invention
In view of the above, embodiments of the present invention provide a data processing method and apparatus, which can solve the problem that flexible data import and export cannot be implemented for different scenarios in the prior art.
To achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a data processing method, including obtaining data to be processed, optionally a preset number of data to be processed as sample data, and calculating a processing time length of each sample data, respectively, to obtain an average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time.
Optionally, based on the total processing duration, selecting a corresponding data import and export mode for the data to be processed, including:
judging whether the total processing time length is smaller than a first threshold value preset in a first time level range, if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold value preset in a second time level range, if so, selecting an asynchronous import and export mode, otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit float value; the second threshold value within the second time level range is the sum of the second time level threshold value and the second time level user tolerance float value.
Optionally, the asynchronous introduction mode includes:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data in a third party database through calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
Optionally, the asynchronous multithreading import mode includes:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data to a third party database in a multithreading mode by calling an implementation interface;
the asynchronous multithreading export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database in a multithreading mode to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
In addition, according to an aspect of the embodiment of the present invention, there is provided a data processing apparatus, including a calculation module configured to obtain data to be processed, optionally a preset number of data to be processed as sample data, and calculate a processing duration of each sample data, respectively, so as to obtain an average processing duration of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and the processing module is used for selecting corresponding data import and export modes for the data to be processed based on the total processing time length.
Optionally, the processing module selects a corresponding data import and export mode for the data to be processed based on the total processing time, including:
judging whether the total processing time length is smaller than a first threshold value preset in a first time level range, if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold value preset in a second time level range, if so, selecting an asynchronous import and export mode, otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit float value; the second threshold value within the second time level range is the sum of the second time level threshold value and the second time level user tolerance float value.
Optionally, the asynchronous introduction mode includes:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data in a third party database through calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
Optionally, the asynchronous multithreading import mode includes:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data to a third party database in a multithreading mode by calling an implementation interface;
the asynchronous multithreading export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database in a multithreading mode to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
According to another aspect of an embodiment of the present invention, there is also provided an electronic device including:
one or more processors;
storage means for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of the data processing embodiments described above.
According to another aspect of an embodiment of the present invention, there is also provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method according to any of the above embodiments based on data processing.
One embodiment of the above invention has the following advantages or benefits: according to the invention, the data to be processed is obtained, a preset amount of data to be processed is selected as sample data, and the processing time length of each piece of sample data is calculated respectively so as to obtain the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time. Therefore, the invention can realize quick and efficient data import and export.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main flow of a data processing method according to a first embodiment of the present invention
FIG. 2 is a schematic diagram of the main flow of a data processing method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of the main flow of a data processing method according to a third embodiment of the present invention;
FIG. 4 is a schematic diagram of the main flow of a data processing method according to a fourth embodiment of the present invention;
FIG. 5 is a schematic diagram of the main flow of a data processing method according to a fifth embodiment of the present invention;
FIG. 6 is a schematic diagram of the main flow of a data processing method according to a sixth embodiment of the present invention;
FIG. 7 is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention;
FIG. 8 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 9 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of the main flow of a data processing method according to a first embodiment of the present invention, the data processing method may include:
step S101, obtaining data to be processed, and optionally obtaining a preset number of data to be processed as sample data.
In an embodiment, n, optionally in need of treatment sample Stripe data, as sample data. The sample data size is smaller than or equal to the total data to be processed and smaller than a preset sample size threshold. Preferably n sample ≤n data &&n sample <5。
Step S102, calculating processing time length of each sample data respectively to obtain average processing time length of the sample data.
In an embodiment, the processing time length of each sample data is calculated separately, and then the processing time length of the average sample data is calculated again
Step S103, obtaining the total processing duration of the data to be processed according to the average processing duration of the sample data.
In an embodiment, the processing duration will be averagedMultiplying by the total amount of data to be processed n data Obtaining the total processing time length T total 。
Step S104, selecting corresponding data import and export modes for the data to be processed based on the total processing time.
Preferably, when selecting a corresponding data import and export mode for the data to be processed, whether the total processing time length is smaller than a first threshold value preset in a first time level range or not can be judged, and if so, the synchronous import and export mode is selected. Otherwise, judging whether the total processing time length is smaller than a second threshold value preset in a second time level range, if so, selecting an asynchronous import and export mode, otherwise, selecting an asynchronous multithreading import and export mode.
Wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit float value. The second threshold value within the second time level range is the sum of the second time level threshold value and the second time level user tolerance float value.
In an embodiment, the first threshold value within the first time level range may be expressed as threshold a +float a Wherein threshold is a Is a first time level threshold, float a It is the first time level that the user tolerates the limit float value. Preferably, the first time level may be a second level.
The second threshold value within the second time level range may be expressed as threshold b +float b Wherein threshold is b Is a second time level threshold, float b It is the second time level that the user tolerates the limit float value. Preferably, the second time level may be an hour level.
Thus, when T total <threshold a +float a And selecting a synchronous import and export mode. When threshold is reached a +float a ≤T total <threshold b +float b And selecting an asynchronous import and export mode. When T is total ≥threshold b +float b And selecting an asynchronous multithreading import and export mode.
Further, the asynchronous importing method includes:
and receiving the calling parameters sent by the foreground application, and sending the calling parameters to the message consumer in a queue mode, so that the message consumer accesses the cloud storage through the task parameters to download the data. And obtaining the downloaded data, and storing the downloaded data in a third party database by calling an implementation interface.
The asynchronous export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. Based on the task table, inquiring the task meeting the condition parameters, and calling a third-party database to acquire the derived data of the task. And analyzing the derived data into a file with a preset format and outputting the file.
In addition, the asynchronous multithreading import mode comprises the following steps:
and receiving the calling parameters sent by the foreground application, and sending the calling parameters to the message consumer in a queue mode, so that the message consumer accesses the cloud storage through the task parameters to download the data. And acquiring the downloaded data, and storing the downloaded data to a third party database in a multithreading mode by calling an implementation interface.
The asynchronous multithreading export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. Based on the task table, inquiring the task meeting the condition parameters, and calling a third party database in a multithreading mode to acquire the derived data of the task. And analyzing the derived data into a file with a preset format and outputting the file.
Therefore, the invention provides a data processing method which can autonomously switch different import and export schemes in the dimension of the data size and the time. And the data size is a key factor influencing the importing or exporting efficiency, the data processing time length is calculated in a sampling mode, and what scheme should be selected is determined on the basis of the processing time length, so that the importing or exporting efficiency is ensured, and the use experience of a user is improved.
Fig. 2 is a schematic diagram of the main flow of a data processing method according to a second embodiment of the present invention, the data processing method may include:
in step S201, data to be processed is acquired, and optionally a preset number of data to be processed is used as sample data.
Step S202, calculating processing time length of each sample data respectively to obtain average processing time length of the sample data.
Step S203, obtaining the total processing duration of the data to be processed according to the average processing duration of the sample data.
Step S204, judging whether the total processing duration is smaller than a first threshold value preset in a first time level range, if yes, proceeding to step S205, otherwise proceeding to step S206.
Wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit float value.
In step S205, a synchronous import and export method is selected.
Step S206, judging whether the total processing duration is smaller than a second threshold value preset in a second time level range, if yes, proceeding to step S207, otherwise proceeding to step S208.
Wherein the second threshold value within the second time level range is the sum of the second time level threshold value and the second time level user tolerance limit float value.
Step S207, selecting asynchronous import and export modes.
In an embodiment, the asynchronous introduction mode includes:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data; and obtaining the downloaded data, and storing the downloaded data in a third party database by calling an implementation interface.
The asynchronous export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. Based on the task table, inquiring the task meeting the condition parameters, and calling a third-party database to acquire the derived data of the task. And analyzing the derived data into a file with a preset format and outputting the file.
Step S208, selecting an asynchronous multithreading import and export mode.
In an embodiment, the asynchronous multithreading import manner includes:
and receiving the calling parameters sent by the foreground application, and sending the calling parameters to the message consumer in a queue mode, so that the message consumer accesses the cloud storage through the task parameters to download the data. And acquiring the downloaded data, and storing the downloaded data to a third party database in a multithreading mode by calling an implementation interface.
The asynchronous multithreading export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. Based on the task table, inquiring the task meeting the condition parameters, and calling a third party database in a multithreading mode to acquire the derived data of the task. And analyzing the derived data into a file with a preset format and outputting the file.
FIG. 3 is a schematic diagram of a main flow of a data processing method according to a third embodiment of the present invention, and a specific implementation procedure of an asynchronous lead-in mode includes:
app a (i.e. foreground application) invokes App b in RPC mode, and stores the data to cloud storage (closed), and after the two operations are completed, the App a responds to the foreground, so that App a can operate other functions.
App b receives the call parameter (Task parameter) from App a, stores the call parameter (Task parameter) into database, sends the call parameter (Task parameter) to message consumer MQ consumer in MQ mode, and after receiving the message, the message consumer MQ consumer accesses cloud storage (closed) through Task parameter (Task parameter) to download data in cloud storage (closed).
App b parses the data downloaded from cloud storage (closed) and invokes a different implementation interface (businesslogicdandler) in App c through RPC, app c stores the data in the database. In the process of calling App c, the state in the task table is changed at the same time.
Where BusinessLogicHandler Class represents a server registration of a managed code program set that implements business logic handlers. RPC generic term Remote Procedure Call, remote procedure call, is a request for services from a remote computer program over a network.
FIG. 4 is a schematic diagram of the main flow of a data processing method according to a fourth embodiment of the present invention, and the implementation procedure of asynchronous derivation includes:
app a (i.e. foreground application) transmits information such as condition parameters in the export task to App b through RPC mode, and stores the information such as condition parameters in database.
And in the App b, task Service continuously scans the Task table in real time to obtain query parameters meeting the conditions, and the App c is called to acquire data in an RPC mode.
The Data parser Data Intpre in App b parses the Data into the required format file and outputs it to the user.
FIG. 5 is a schematic diagram of the main flow of a data processing method according to a fifth embodiment of the present invention, and a specific implementation procedure of multi-threaded asynchronous import:
app a (i.e. foreground application) invokes App b in RPC mode, and stores the data to cloud storage (closed), and after the two operations are completed, the App a responds to the foreground, so that App a can operate other functions.
App b receives the call parameter (Task parameter) from App a, stores the call parameter (Task parameter) into database, sends the call parameter (Task parameter) to message consumer MQ consumer in MQ mode, and after receiving the message, the message consumer MQ consumer accesses cloud storage (closed) through Task parameter (Task parameter) to download data in cloud storage (closed).
App b parses the data downloaded from the cloud storage (closed), and invokes different implementation interfaces (business logic handler) in App c in a multithreading manner, and App c stores the data in the database. In the process of calling App c, the state in the task table is changed at the same time.
It can be seen that the multithreading asynchronous import method establishes connection between different applications by the RPC method, thereby implementing an asynchronous processing procedure. And, store the data to be processed on the cloud storage, analyze the data on the cloud storage in an asynchronous way. And processing the analyzed data by multiple threads and storing the processed data into a database.
FIG. 6 is a schematic diagram of the main flow of a data processing method according to a sixth embodiment of the present invention, and a specific implementation procedure of multi-threaded asynchronous export:
app a (i.e. foreground application) transmits information such as condition parameters in the export task to App b through RPC mode, and stores the information such as condition parameters in database.
And in the App b, task Service continuously scans the Task table in real time to obtain query parameters meeting the conditions, and the App c is called in a multithreading mode to acquire data.
The Data parser Data Intpre in App b parses the Data into the required format file and outputs it to the user.
Preferably, the sizes of the stored single data amounts are inconsistent, and different computer performances have certain differences, so that the thread number needs to be dynamically determined according to actual calculation conditions, and the thread number can be constrained according to the following formula:
wherein, thread watingTime Waiting for a duration for the thread.
thread cpuTime Is the thread CPU duration.
Nums CPU The number of CPUs.
It can be seen that the multithreaded asynchronous export scheme establishes a connection between different applications by way of RPC, thereby implementing an asynchronous process. The data parser parses the data acquired according to the derived parameters in an asynchronous manner and aggregates the parsed data back to the user.
In summary, the data processing method according to various embodiments of the present invention can adapt to the data volume that changes due to the service change, for example, the initial data volume is smaller, and along with the increase of the data volume, corresponding data import and export modes can be adopted to ensure the service increase requirement. Meanwhile, the method can also ensure the dynamic transformation data import and export mode when the fluctuation of the data size variation range of the processed data is large.
Fig. 7 is a schematic diagram of main modules of a data processing apparatus according to a first embodiment of the present invention, and as shown in fig. 7, the data processing apparatus 700 includes a calculation module 701 and a processing module 702. The calculating module 701 is configured to obtain data to be processed, optionally, a preset number of data to be processed as sample data, and calculate a processing duration of each sample data, so as to obtain an average processing duration of the sample data; and obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data. The processing module 702 is configured to select a corresponding data import and export manner for the data to be processed based on the total processing duration.
As another embodiment, the processing module 702 selects, for the data to be processed, a corresponding data import and export manner based on the total processing duration, including:
judging whether the total processing time length is smaller than a first threshold value preset in a first time level range, if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold value preset in a second time level range, if so, selecting an asynchronous import and export mode, otherwise, selecting an asynchronous multithreading import and export mode.
Wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit float value. The second threshold value within the second time level range is the sum of the second time level threshold value and the second time level user tolerance float value.
Preferably, the asynchronous introduction mode includes:
and receiving the calling parameters sent by the foreground application, and sending the calling parameters to the message consumer in a queue mode, so that the message consumer accesses the cloud storage through the task parameters to download the data. And obtaining the downloaded data, and storing the downloaded data in a third party database by calling an implementation interface.
The asynchronous export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. Based on the task table, inquiring the task meeting the condition parameters, and calling a third-party database to acquire the derived data of the task. And analyzing the derived data into a file with a preset format and outputting the file.
Still preferably, the asynchronous multithreading import method includes:
and receiving the calling parameters sent by the foreground application, and sending the calling parameters to the message consumer in a queue mode, so that the message consumer accesses the cloud storage through the task parameters to download the data. And acquiring the downloaded data, and storing the downloaded data to a third party database in a multithreading mode by calling an implementation interface.
The asynchronous multithreading export mode comprises the following steps:
and receiving the condition parameters sent by the foreground application. Based on the task table, inquiring the task meeting the condition parameters, and calling a third party database in a multithreading mode to acquire the derived data of the task. And analyzing the derived data into a file with a preset format and outputting the file.
In the present invention, the data processing method and the data processing apparatus have a corresponding relationship in terms of implementation, and therefore, the description of the repeated contents is omitted.
Fig. 8 illustrates an exemplary system architecture 800 in which a data processing method or data processing apparatus of an embodiment of the present invention may be applied.
As shown in fig. 8, a system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. The network 804 serves as a medium for providing communication links between the terminal devices 801, 802, 803 and the server 805. The network 804 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 805 through the network 804 using the terminal devices 801, 802, 803 to receive or send messages or the like. Various communication client applications such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 801, 802, 803.
The terminal devices 801, 802, 803 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 805 may be a server providing various services, such as a background management server (by way of example only) that provides support for shopping-type websites browsed by users using the terminal devices 801, 802, 803. The background management server may analyze and process the received data such as the product information query request, and feedback the processing result (e.g., the target push information, the product information—only an example) to the terminal device.
It should be noted that, the data processing method provided by the embodiment of the present invention is generally executed by the server 805, and accordingly, the data processing apparatus is generally disposed in the server 805.
It should be understood that the number of terminal devices, networks and servers in fig. 8 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 9, there is illustrated a schematic diagram of a computer system 900 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 9 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 9, the computer system 900 includes a Central Processing Unit (CPU) 901, which can execute various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM903, various programs and data necessary for the operation of the system 900 are also stored. The CPU901, ROM902, and RAM903 are connected to each other through a bus 904. An input/output (I/O) interface 905 is also connected to the bus 904.
The following components are connected to the I/O interface 905: an input section 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 908 including a hard disk or the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 910 so that a computer program read out therefrom is installed into the storage section 908 as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 909 and/or installed from the removable medium 911. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 901.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a computing module and a processing module. The names of these modules do not constitute a limitation on the module itself in some cases.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: acquiring data to be processed, optionally selecting a preset number of data to be processed as sample data, and respectively calculating the processing time length of each piece of sample data to obtain the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data; and selecting corresponding data import and export modes for the data to be processed based on the total processing time.
According to the technical scheme provided by the embodiment of the invention, the problem that flexible data import and export cannot be realized aiming at different scenes in the prior art can be solved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method of data processing, comprising:
acquiring data to be processed, optionally selecting a preset number of data to be processed as sample data, and respectively calculating the processing time length of each piece of sample data to obtain the average processing time length of the sample data;
obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data;
based on the total processing time length, selecting a corresponding data import and export mode for the data to be processed; and determining to select a synchronous or asynchronous import and export mode according to whether the total processing time length is smaller than a first threshold value preset in a first time level range.
2. The method according to claim 1, wherein selecting respective data import and export modes for the data to be processed based on the total processing time length comprises:
judging whether the total processing time length is smaller than a first threshold value preset in a first time level range, if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold value preset in a second time level range, if so, selecting an asynchronous import and export mode, otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit float value; the second threshold value within the second time level range is the sum of the second time level threshold value and the second time level user tolerance float value.
3. The method according to claim 2, wherein the asynchronous introduction means comprises:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data in a third party database through calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
4. The method according to claim 2, wherein the asynchronous multithreading method comprises:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data to a third party database in a multithreading mode by calling an implementation interface;
the asynchronous multithreading export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database in a multithreading mode to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
5. A data processing apparatus, comprising:
the computing module is used for acquiring data to be processed, selecting a preset number of data to be processed as sample data, and respectively computing the processing time length of each piece of sample data so as to acquire the average processing time length of the sample data; obtaining the total processing time length of the data to be processed according to the average processing time length of the sample data;
the processing module is used for selecting corresponding data import and export modes for the data to be processed based on the total processing time length; and determining to select a synchronous or asynchronous import and export mode according to whether the total processing time length is smaller than a first threshold value preset in a first time level range.
6. The apparatus of claim 5, wherein the processing module selects a corresponding data import and export manner for the data to be processed based on a total processing time length, including:
judging whether the total processing time length is smaller than a first threshold value preset in a first time level range, if so, selecting a synchronous import and export mode; otherwise, judging whether the total processing time length is smaller than a second threshold value preset in a second time level range, if so, selecting an asynchronous import and export mode, otherwise, selecting an asynchronous multithreading import and export mode;
wherein the first threshold value in the first time level range is the sum of the first time level threshold value and the first time level user tolerance limit float value; the second threshold value within the second time level range is the sum of the second time level threshold value and the second time level user tolerance float value.
7. The apparatus of claim 6, wherein the asynchronous lead-in mode comprises:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data in a third party database through calling an implementation interface;
the asynchronous export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
8. The apparatus of claim 6, wherein the asynchronous multithreaded import comprises:
receiving call parameters sent by a foreground application, and sending the call parameters to a message consumer in a queue mode, so that the message consumer accesses cloud storage through task parameters to download data;
acquiring download data, and storing the download data to a third party database in a multithreading mode by calling an implementation interface;
the asynchronous multithreading export mode comprises the following steps:
receiving a condition parameter sent by a foreground application;
inquiring a task meeting the condition parameters based on the task table, and calling a third party database in a multithreading mode to acquire derived data of the task;
and analyzing the derived data into a file with a preset format and outputting the file.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-4.
10. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910988064.XA CN112685481B (en) | 2019-10-17 | 2019-10-17 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910988064.XA CN112685481B (en) | 2019-10-17 | 2019-10-17 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112685481A CN112685481A (en) | 2021-04-20 |
| CN112685481B true CN112685481B (en) | 2023-11-03 |
Family
ID=75444740
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910988064.XA Active CN112685481B (en) | 2019-10-17 | 2019-10-17 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112685481B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282456B (en) * | 2021-05-24 | 2023-09-22 | 北京京东振世信息技术有限公司 | Data processing method and device |
| CN113704269B (en) * | 2021-09-02 | 2024-04-16 | 京东城市(北京)数字科技有限公司 | Data processing method, system, storage medium and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657123A (en) * | 2013-11-21 | 2015-05-27 | 航天信息股份有限公司 | Method for dynamically exporting files |
| CN105894163A (en) * | 2014-12-17 | 2016-08-24 | 中信天津外包服务有限公司 | One-stop full-process archive information management method |
| CN108228752A (en) * | 2017-12-21 | 2018-06-29 | 中国联合网络通信集团有限公司 | Data full dose deriving method, data distribution device and data export node |
| CN108346106A (en) * | 2018-02-23 | 2018-07-31 | 平安科技（深圳）有限公司 | Bill input method, system, optical character identification server and storage medium |
| CN108874894A (en) * | 2018-05-21 | 2018-11-23 | 平安科技（深圳）有限公司 | Crosstab deriving method, device, computer equipment and storage medium |
| CN110019293A (en) * | 2017-09-07 | 2019-07-16 | 北京国双科技有限公司 | List data deriving method and device based on impala database |
| CN110020646A (en) * | 2019-04-16 | 2019-07-16 | 恒生电子股份有限公司 | Archive method, apparatus, electronic equipment and storage medium |
-
2019
- 2019-10-17 CN CN201910988064.XA patent/CN112685481B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657123A (en) * | 2013-11-21 | 2015-05-27 | 航天信息股份有限公司 | Method for dynamically exporting files |
| CN105894163A (en) * | 2014-12-17 | 2016-08-24 | 中信天津外包服务有限公司 | One-stop full-process archive information management method |
| CN110019293A (en) * | 2017-09-07 | 2019-07-16 | 北京国双科技有限公司 | List data deriving method and device based on impala database |
| CN108228752A (en) * | 2017-12-21 | 2018-06-29 | 中国联合网络通信集团有限公司 | Data full dose deriving method, data distribution device and data export node |
| CN108346106A (en) * | 2018-02-23 | 2018-07-31 | 平安科技（深圳）有限公司 | Bill input method, system, optical character identification server and storage medium |
| WO2019161615A1 (en) * | 2018-02-23 | 2019-08-29 | 平安科技（深圳）有限公司 | Bill entry method, system, optical character recognition server and storage medium |
| CN108874894A (en) * | 2018-05-21 | 2018-11-23 | 平安科技（深圳）有限公司 | Crosstab deriving method, device, computer equipment and storage medium |
| CN110020646A (en) * | 2019-04-16 | 2019-07-16 | 恒生电子股份有限公司 | Archive method, apparatus, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112685481A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108810047B (en) | Method and device for determining information push accuracy rate and server | |
| CN112527649A (en) | Test case generation method and device | |
| CN110706093A (en) | Accounting processing method and device | |
| CN110795147A (en) | Interface protocol file management method and device | |
| CN113076153B (en) | Interface calling method and device | |
| CN112685481B (en) | Data processing method and device | |
| CN112379982B (en) | Task processing method, device, electronic equipment and computer readable storage medium | |
| CN111062572B (en) | Task allocation method and device | |
| CN112084042B (en) | Message processing method and device | |
| CN111881329A (en) | Account balance management method and system | |
| CN112667368A (en) | Task data processing method and device | |
| CN109413212B (en) | Method and apparatus for processing requests | |
| CN111831503B (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN113096295B (en) | Numbering method, numbering device, electronic equipment and computer readable storage medium | |
| CN114924937A (en) | Batch task processing method and device, electronic equipment and computer readable medium | |
| CN113760487B (en) | Service processing method and device | |
| CN111176679B (en) | Application program updating method and device | |
| CN113127561B (en) | Method and device for generating service single number, electronic equipment and storage medium | |
| CN112784139A (en) | Query method, query device, electronic equipment and computer readable medium | |
| CN113779018A (en) | Data processing method and device | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| CN113722193A (en) | Method and device for detecting page abnormity | |
| CN112860447A (en) | Interaction method and system between different applications | |
| CN112783914A (en) | Statement optimization method and device | |
| CN110858240A (en) | Front-end module loading method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |