CN110569259A - Method and device for processing mass data - Google Patents
Method and device for processing mass data Download PDFInfo
- Publication number
- CN110569259A CN110569259A CN201910686474.9A CN201910686474A CN110569259A CN 110569259 A CN110569259 A CN 110569259A CN 201910686474 A CN201910686474 A CN 201910686474A CN 110569259 A CN110569259 A CN 110569259A
- Authority
- CN
- China
- Prior art keywords
- query
- data
- paging
- condition
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
the invention discloses a method and a device for processing mass data, wherein the method comprises the following steps: acquiring a data request, and generating a current data query condition according to the data request; executing a paging query, wherein the paging query comprises: performing paging query in the service table according to the current data query conditions and the paging set number, and writing query result data into a query result file; acquiring the page number which is currently inquired, and verifying whether the page number which is currently inquired reaches an early warning value or not; if the number of pages currently queried reaches an early warning value, generating a new data query condition according to the related information of the last piece of data in the query result data; and taking the new data query condition as the current data query condition, and repeatedly executing paging query until the query is finished. The invention controls the query depth by timely adjusting the query condition, avoids the problems of reduced query performance of the database, even unavailable overtime and the like caused by overlarge query depth, and meets the requirement of large-batch data processing.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing large-batch data.
background
the actual business scene of project development can have the requirement of data downloading, and under the condition that the data volume is not large (for example < 1 ten thousand), the data can be directly downloaded after being completely inquired, but when the data volume is more than 10 ten thousand, the problem of memory overflow caused by overlarge operation data object can occur in the direct downloading process. Although asynchronous paging downloading can solve the problem of large data volume of single downloading, under the condition that the data volume is more than 30 ten thousand, the query performance of the database is rapidly reduced due to deep query, and finally, the query is overtime, so that the database cannot be downloaded.
Disclosure of Invention
In order to solve the problems in the prior art, embodiments of the present invention provide a method and an apparatus for downloading large-batch data, so as to overcome the problems in the prior art that, due to an excessively large data download amount, a database is rapidly reduced in query performance due to deep query, so that the database cannot be downloaded due to an overtime deep query.
In order to solve one or more technical problems, the invention adopts the technical scheme that:
in one aspect, a method for processing large-batch data is provided, which includes the following steps:
Acquiring a data request, and generating a current data query condition according to the data request;
Executing a paging query, wherein the paging query comprises the following steps:
Performing paging query in a service table according to the current data query condition and the paging set number, and writing query result data into a query result file;
Acquiring the number of the pages which are inquired currently, and verifying whether the number of the pages which are inquired currently reaches a pre-alarm value or not;
if the number of pages currently queried reaches an early warning value, generating a new data query condition according to the related information of the last piece of data in the query result data;
And taking the new data query condition as the current data query condition, and repeatedly executing the paging query until the query is finished.
further, the generating a new data query condition according to the related information of the last piece of data in the query result data includes:
Acquiring a primary key value of a primary service table of the last data in the query result data, and taking the primary key value as an initial value of a primary key during query;
And adding the initial value of the primary key to the current data query condition to generate a new data query condition.
Further, before obtaining the number of pages currently being queried, the paging querying further includes:
And acquiring the number of the query results of the current page, verifying whether the number of the query results is less than the set number of the pages, if so, ending the process, and otherwise, acquiring the number of the pages which are queried currently.
Further, the method further comprises:
and if the paging number which is currently inquired does not reach the early warning value, continuing to carry out paging inquiry in a service table according to the current data inquiry condition and the paging set number.
further, the method further comprises:
And uploading the query result file to a file server, and updating a file path to a download task table so as to download the query result data according to the download task table.
In another aspect, an apparatus for processing large batch of data is provided, the apparatus comprising:
The first generation module is used for acquiring a data request and generating a current data query condition according to the data request;
the data query module is used for performing paging query in a service table according to the current data query condition and the paging set number;
The data writing module is used for writing the query result data into the query result file;
the first verification module is used for acquiring the number of the pages which are inquired currently and verifying whether the number of the pages which are inquired currently reaches an early warning value or not;
And the second generation module is used for generating a new data query condition according to the related information of the last piece of data in the query result data, and taking the new data query condition as the current data query condition.
Further, the second generating module includes:
A primary key obtaining unit, configured to obtain a primary key value of a primary service table of the last piece of data in the query result data, and use the primary key value as an initial value of a primary key during query;
and the condition generating unit is used for adding the initial value of the primary key to the current data query condition, generating a new data query condition and taking the new data query condition as the current data query condition.
further, the apparatus further comprises:
and the second verification module is used for acquiring the number of the query results of the current page and verifying whether the number of the query results is less than the set number of the pages.
Further, the data query module is further configured to:
and if the paging number which is currently inquired does not reach the early warning value, continuing to carry out paging inquiry in a service table according to the current data inquiry condition and the paging set number.
further, the apparatus further comprises:
The data uploading module is used for uploading the query result file to a file server;
And the path updating module is used for updating the file path to a downloading task table so as to download the query result data according to the downloading task table.
the technical scheme provided by the embodiment of the invention has the following beneficial effects:
1. according to the method and the device for processing the mass data, provided by the embodiment of the invention, the query depth is controlled by timely adjusting the query condition, the problems that the query performance of the database is reduced and even the database is unavailable due to overlarge depth in paging query downloading are avoided, and the requirements of processing such as mass data downloading of services are met;
2. according to the method and the device for processing the large-batch data, provided by the embodiment of the invention, the data is inquired in batches according to the pages, so that the overflow of the heap memory of the JVM (Java virtual machine) caused by large inquiry amount at a time can be effectively avoided.
drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow diagram illustrating a method of processing large volumes of data in accordance with an exemplary embodiment;
FIG. 2 is a flow diagram illustrating the generation of new data query conditions based on information about the last piece of data in the query result data, according to an example embodiment;
Fig. 3 is a schematic diagram illustrating a structure of a device for processing large volume data according to an exemplary embodiment.
Detailed Description
in order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The method and the device for processing the mass data provided by the embodiment of the invention are suitable for various service scenes, in particular suitable for scenes of inquiring and downloading the mass data (for example, the data volume is more than 30 ten thousand). For example, the business needs to download a sales data report for a certain period of the query, and the downloaded data amount is large (for example, close to 100 ten thousand). By adjusting the query depth in time, the embodiment of the invention avoids the problem that the database fails to query due to the overtime of deep query and finally fails to download due to overlarge download amount.
fig. 1 is a flow chart illustrating a method of processing a large volume of data according to an exemplary embodiment, and referring to fig. 1, the method includes the steps of:
s1: and acquiring a data request, and generating a current data query condition according to the data request.
Specifically, a data request sent by a user is acquired, and a current data query condition is generated according to the data request, wherein the data request includes a data query and download request (for example, a sales data report requesting to query and download a certain period of time). In the embodiment of the invention, the acquired data request is subjected to asynchronous processing. When the method is specifically implemented, the service type and the request parameters in the data request are stored in a database table, and a timing task is provided for processing the data request. Meanwhile, as a better implementation mode, the embodiment of the invention can also provide a download query page for a user to check the download progress and the processed files.
s2: a paging query is performed.
Specifically, at present, a method based on asynchronous paging downloading is generally adopted for query downloading of large-batch data. The asynchronous paging downloading method usually calculates the total amount of all data meeting the conditions according to the data query conditions, pages the data according to the set number of pages, and performs query processing according to the pages in batches. Although the asynchronous mode can effectively avoid the problem of connection overtime request caused by large data volume and long processing time, the running condition of the downloaded task can be dynamically adjusted, and the influence of the downloading function on normal business due to excessive consumption of system resources is reduced. And the heap memory overflow of the JVM (Java virtual machine) caused by large query amount per time can be effectively avoided by downloading in batches according to pages. However, the asynchronous paging downloading mode causes the problem that the database performance is reduced due to deep query, even causes the problem that the downloading is finally failed due to the query failure.
In order to solve the above problem, in the embodiment of the present invention, data processing logic (such as query logic) is processed, and an iterative paging query is performed according to the processed data processing logic. In specific implementation, the data query condition is processed, that is, a judgment on the primary key of the service table is newly added to the original data query condition (that is, a judgment condition is added to the original data query condition). As a better example, data query may be performed according to an original query condition, when a preset condition is triggered, an initial value of a primary key during query is assigned to generate a new data query condition, and then data query is continued according to the new data query condition, and such iteration is performed until query is completed. Wherein, it can be set that when the initial value of the primary key is null, the query condition does not work.
As further shown in fig. 1, the paging query specifically includes the following steps:
s2.1: and performing paging inquiry in a service table according to the current data inquiry condition and the paging set number, and writing inquiry result data into an inquiry result file.
specifically, all data meeting the conditions are retrieved from the service table according to the current data query conditions obtained in the above steps, the total number of the data is calculated, the data meeting the conditions are subjected to paging processing according to the number of paging set strips, paging query is performed according to the paging, and the query result data of each page is written into the query result file, wherein the query result file can be a csv file or a file in other formats, which is not limited herein.
In the embodiment of the invention, the set number of the paging is preset and is used for paging data. The number of the paging set pieces is the maximum number of data of each page after paging, for example, assuming that the number of the paging set pieces is 100, and the query result satisfies the data query condition of 1000, the data amount of each page is 100, and 10 pages in total. It should be noted that, in the embodiment of the present invention, the specific number of the set number of pages is not limited, and a user may set the number according to actual requirements.
s2.2: and acquiring the number of the pages which are inquired currently, and verifying whether the number of the pages which are inquired currently reaches an early warning value.
specifically, in the embodiment of the invention, the query page number is monitored in real time, so that the query depth is timely adjusted, and the problem that the database fails to query due to the fact that the deep query is overtime and finally fails to download due to overlarge download amount is solved. In specific implementation, an early warning value can be preset to give an early warning to the query depth. The query depth can be calculated by the number of currently queried pages and the number of set pages, that is, the current query depth is obtained by multiplying the number of currently queried pages by the number of set pages. For example, if the number of pages is set to 100, and the number of obtained pages currently being queried is 10, the current query depth is 1000.
if the query depth exceeds 1000, which may cause the database deep query to be over time, and the number of paging set entries is 100, the warning value may be set to 10. Namely, when the number of the pages which are acquired to be inquired currently is 10, judging that the number of the pages which are inquired currently reaches an early warning value. It should be noted that, in the embodiment of the present invention, specific numerical values of the warning values are not limited, and a user may set the warning values according to actual requirements.
s2.3: and if the number of the pages which are queried currently reaches an early warning value, generating a new data query condition according to the related information of the last piece of data in the query result data.
specifically, when the number of pages currently queried is monitored to reach an early warning value, the query depth of the next page is adjusted by adjusting the query condition of the integer data, namely, the query depth of the next page is reduced. In specific implementation, the related information of the last piece of data in the query result data can be acquired, and a new data query condition is generated again according to the related information.
S2.4: and taking the new data query condition as the current data query condition, and repeatedly executing the paging query until the query is finished.
specifically, the current data query condition is replaced by the new data query condition obtained in the above step, and the paging query is repeatedly executed until the query is finished, that is, the steps S2.1 to S2.4 are repeatedly executed until the query is finished.
fig. 2 is a flowchart illustrating a process of generating a new data query condition according to information related to a last piece of data in query result data according to an exemplary embodiment, and referring to fig. 3, as a preferred implementation manner, in the embodiment of the present invention, the generating a new data query condition according to information related to a last piece of data in the query result data includes:
S2.3.1: and acquiring a primary key value of a primary service table of the last data in the query result data, and taking the primary key value as an initial value of the primary key during query.
Specifically, when a new data query condition is generated according to the related information of the last piece of data in the query result data, the new data query condition may be regenerated through the primary key value of the primary service table of the last piece of data in the query result data. In specific implementation, a judgment on the primary key of the service table may be added to the original data query condition, that is, a judgment condition is added to the original data query condition, the added judgment condition may be an initial value of the primary key during query, and when the initial value of the primary key of the data query condition is null, the query condition does not work. The obtained primary key value of the last piece of data may be used as an initial value of the primary key in query.
s2.3.2: and adding the initial value of the primary key to the current data query condition to generate a new data query condition.
specifically, the parameter of the initial value of the primary key is added to the current data query condition (i.e., the original data query condition), and a new data query condition is generated. When the next page is queried, the data query is performed according to the new data query condition. Because the initial value of the primary key during query is added to the new data query condition, when paging query is subsequently performed in the service table, query will be started from the data with the primary key value equal to the initial value of the primary key, for example, the initial value of the primary key added to the new data query condition is 1000, and query will be started from the data with the primary key of 1000 in next query, so as to reduce the query depth in the database, avoid causing the problem of performance degradation of the database, and even cause the problem of download failure finally caused by query failure.
as a preferred implementation manner, in the embodiment of the present invention, before obtaining the number of pages currently being queried, the paging querying further includes:
And acquiring the number of the query results of the current query page, verifying whether the number of the query results is less than the set number of the pages, if so, ending the process, and otherwise, acquiring the number of the current query page.
specifically, in the embodiment of the present invention, the paging query process further includes verifying the number of query results currently querying the paging. In specific implementation, the number of the query results of the current query page is obtained, the number of the query results is compared with the set number of the page, if the number of the query results is less than the set number of the page, the current page is the last page to be queried corresponding to the data request, and at this time, the process can be directly ended, otherwise, the current page is not the last page to be queried corresponding to the data request, and at this time, the next process is entered to obtain the number of the page currently being queried.
It should be noted that, the above-mentioned determining whether the current page is the last page to be queried corresponding to the current data request by comparing whether the number of query results of the currently queried page is smaller than the set number of pages is based on that when the query result is the page query, the data is paged according to the set number of pages, if not, the number of query results should be equal to the set number of pages, and if the number of query results is smaller than the set number of pages, it indicates that the page is the last page. By increasing the verification of the number of the query results of the current query pages, the workload of database query can be reduced, and the pressure of database data processing can be reduced.
As a preferred implementation manner, in an embodiment of the present invention, the method further includes:
And if the paging number which is currently inquired does not reach the early warning value, continuing to carry out paging inquiry in a service table according to the current data inquiry condition and the paging set number.
specifically, if it is verified that the number of pages currently being queried does not reach the warning value, it is indicated that the database does not have the performance degradation and other problems caused by deep query, and at this time, in order to improve the query efficiency, a new data query condition may not be regenerated to replace the current data query condition, but the current data query condition generated before is continuously used to perform page query in the service table.
as a preferred implementation manner, in an embodiment of the present invention, the method further includes:
And uploading the query result file to a file server, and updating a file path to a download task table so as to download the query result data according to the download task table.
Specifically, in order to facilitate the user to perform subsequent processing such as downloading of the queried data, in the embodiment of the present invention, the generated query result file is uploaded to the file server, and the file path of the query result file is updated to the download task table, so that the user can download the query result data according to the download task table.
The following example illustrates the scheme provided by the embodiments of the present invention:
The data request sent by the user is 'data of downloading the order form for half a year', and the current data query condition generated according to the relevant information of the data request is as follows: state (status) is 01-complete, orders placed at A001 and A002 sites (plants), with data volume > 100 million satisfying the conditions
wherein, the business main key PoId, the number of the paging setting is 100, and the page depth warning is 1000.
according to the current data query condition, namely status 01, place in (A001, A002), PoId (parameter null, the condition is not functional, otherwise, the service key > PoId), obtaining the total number allQty of the order list satisfying the current data query condition, setting the number of pages to be 100, calculating the total number of pages allPage 100 to be allQty/100, sequentially carrying out page query in a loop with allPage, and writing the obtained data set (currentData) into a query result file. Meanwhile, the size (currentDataSize) of the data set, that is, the number of the current paging query result pieces, is determined, and if currentDataSize < the number of paging set pieces 100, it indicates that the last page has been processed, and the process is ended. Otherwise, whether the value (pageSize) of the current page number is equal to the page depth early warning 1000 or not is judged, and the PoId of the last record in the result set is taken out when the early warning value is reached. The PoId is used as a parameter to re-iterate the paging query method. Otherwise, the circulation processing is continued.
fig. 3 is a schematic structural diagram illustrating a large-batch data processing apparatus according to an exemplary embodiment, and referring to fig. 3, the apparatus includes:
the first generation module is used for acquiring a data request and generating a current data query condition according to the data request;
The data query module is used for performing paging query in a service table according to the current data query condition and the paging set number;
The data writing module is used for writing the query result data into the query result file;
the first verification module is used for acquiring the number of the pages which are inquired currently and verifying whether the number of the pages which are inquired currently reaches an early warning value or not;
and the second generation module is used for generating a new data query condition according to the related information of the last piece of data in the query result data, and taking the new data query condition as the current data query condition.
As a preferred implementation manner, in an embodiment of the present invention, the second generating module includes:
a primary key obtaining unit, configured to obtain a primary key value of a primary service table of the last piece of data in the query result data, and use the primary key value as an initial value of a primary key during query;
and the condition generating unit is used for adding the initial value of the primary key to the current data query condition, generating a new data query condition and taking the new data query condition as the current data query condition.
As a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes:
and the second verification module is used for acquiring the number of the query results of the current page and verifying whether the number of the query results is less than the set number of the pages.
as a preferred implementation manner, in an embodiment of the present invention, the data query module is further configured to:
And if the paging number which is currently inquired does not reach the early warning value, continuing to carry out paging inquiry in a service table according to the current data inquiry condition and the paging set number.
As a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes:
The data uploading module is used for uploading the query result file to a file server;
And the path updating module is used for updating the file path to a downloading task table so as to download the query result data according to the downloading task table.
In summary, the technical solution provided by the embodiment of the present invention has the following beneficial effects:
1. according to the method and the device for processing the mass data, provided by the embodiment of the invention, the query depth is controlled by timely adjusting the query condition, the problems of reduced query performance of the database and even unavailable overtime caused by overlarge depth of paging query downloading are avoided, and the requirement of mass data downloading of services is met;
2. according to the method and the device for processing the large-batch data, provided by the embodiment of the invention, the data is inquired in batches according to the pages, so that the overflow of the heap memory of the JVM (Java virtual machine) caused by large inquiry amount at a time can be effectively avoided.
It should be noted that: in the processing apparatus for processing large batch of data provided in the foregoing embodiment, when triggering a data processing service, only the division of each functional module is illustrated, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the processing apparatus for large batch data and the processing method for large batch data provided in the above embodiments belong to the same concept, that is, the apparatus is based on the processing method for large batch data, and the specific implementation process thereof is detailed in the method embodiments and will not be described herein again.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disk, or the like.
the above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method for processing large batch of data, the method comprising the steps of:
Acquiring a data request, and generating a current data query condition according to the data request;
executing a paging query, wherein the paging query comprises the following steps:
Performing paging query in a service table according to the current data query condition and the paging set number, and writing query result data into a query result file;
Acquiring the number of pages which are inquired currently, and verifying whether the number of pages which are inquired currently reaches an early warning value or not;
If the number of pages currently queried reaches an early warning value, generating a new data query condition according to the related information of the last piece of data in the query result data;
and taking the new data query condition as the current data query condition, and repeatedly executing the paging query until the query is finished.
2. The method for processing bulk data according to claim 1, wherein the generating a new data query condition according to the related information of the last piece of data in the query result data comprises:
Acquiring a primary key value of a primary service table of the last data in the query result data, and taking the primary key value as an initial value of a primary key during query;
and adding the initial value of the primary key to the current data query condition to generate a new data query condition.
3. the method for processing bulk data according to claim 1 or 2, wherein before obtaining the number of pages currently being queried, the paging query further comprises:
And acquiring the number of the query results of the current query page, verifying whether the number of the query results is less than the set number of the pages, if so, ending the process, and otherwise, acquiring the number of the current query page.
4. the method of processing bulk data according to claim 1 or 2, further comprising:
and if the paging number which is currently inquired does not reach the early warning value, continuing to carry out paging inquiry in a service table according to the current data inquiry condition and the paging set number.
5. The method of processing bulk data according to claim 1 or 2, further comprising:
and uploading the query result file to a file server, and updating a file path to a download task table so as to download the query result data according to the download task table.
6. An apparatus for processing bulk data, the apparatus comprising:
the first generation module is used for acquiring a data request and generating a current data query condition according to the data request;
the data query module is used for performing paging query in a service table according to the current data query condition and the paging set number;
the data writing module is used for writing the query result data into the query result file;
The first verification module is used for acquiring the number of the pages which are inquired currently and verifying whether the number of the pages which are inquired currently reaches an early warning value or not;
And the second generation module is used for generating a new data query condition according to the related information of the last piece of data in the query result data, and taking the new data query condition as the current data query condition.
7. The apparatus for processing bulk data according to claim 6, wherein the second generating module comprises:
A primary key obtaining unit, configured to obtain a primary key value of a primary service table of the last piece of data in the query result data, and use the primary key value as an initial value of a primary key during query;
and the condition generating unit is used for adding the initial value of the primary key to the current data query condition, generating a new data query condition and taking the new data query condition as the current data query condition.
8. the apparatus for processing bulk data according to claim 6 or 7, further comprising:
And the second verification module is used for acquiring the number of the query results of the current page and verifying whether the number of the query results is less than the set number of the pages.
9. The apparatus for processing bulk data according to claim 6 or 7, wherein the data query module is further configured to:
and if the paging number which is currently inquired does not reach the early warning value, continuing to carry out paging inquiry in a service table according to the current data inquiry condition and the paging set number.
10. the apparatus for processing bulk data according to claim 6 or 7, further comprising:
the data uploading module is used for uploading the query result file to a file server;
and the path updating module is used for updating the file path to a downloading task table so as to download the query result data according to the downloading task table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910686474.9A CN110569259A (en) | 2019-07-26 | 2019-07-26 | Method and device for processing mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910686474.9A CN110569259A (en) | 2019-07-26 | 2019-07-26 | Method and device for processing mass data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110569259A true CN110569259A (en) | 2019-12-13 |
Family
ID=68773423
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910686474.9A Pending CN110569259A (en) | 2019-07-26 | 2019-07-26 | Method and device for processing mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110569259A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111368006A (en) * | 2020-03-31 | 2020-07-03 | 中国工商银行股份有限公司 | Mass data strip conditional centralized extraction system and method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103886022A (en) * | 2014-02-24 | 2014-06-25 | 上海上讯信息技术股份有限公司 | Paging-query querying device and method based on primary key fields |
| CN106227894A (en) * | 2016-08-24 | 2016-12-14 | 中国农业银行股份有限公司 | A kind of data page querying method and device |
| CN107016045A (en) * | 2017-02-17 | 2017-08-04 | 阿里巴巴集团控股有限公司 | A kind of method and device of paged data inquiry |
| CN107783978A (en) * | 2016-08-24 | 2018-03-09 | 中安消物联传感（深圳）有限公司 | A kind of big data inquiry return method and device |
| CN109344197A (en) * | 2018-09-13 | 2019-02-15 | 广州帷策智能科技有限公司 | Paging method for down loading and device based on big data |
-
2019
- 2019-07-26 CN CN201910686474.9A patent/CN110569259A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103886022A (en) * | 2014-02-24 | 2014-06-25 | 上海上讯信息技术股份有限公司 | Paging-query querying device and method based on primary key fields |
| CN106227894A (en) * | 2016-08-24 | 2016-12-14 | 中国农业银行股份有限公司 | A kind of data page querying method and device |
| CN107783978A (en) * | 2016-08-24 | 2018-03-09 | 中安消物联传感（深圳）有限公司 | A kind of big data inquiry return method and device |
| CN107016045A (en) * | 2017-02-17 | 2017-08-04 | 阿里巴巴集团控股有限公司 | A kind of method and device of paged data inquiry |
| CN109344197A (en) * | 2018-09-13 | 2019-02-15 | 广州帷策智能科技有限公司 | Paging method for down loading and device based on big data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111368006A (en) * | 2020-03-31 | 2020-07-03 | 中国工商银行股份有限公司 | Mass data strip conditional centralized extraction system and method |
| CN111368006B (en) * | 2020-03-31 | 2023-03-17 | 中国工商银行股份有限公司 | Mass data strip conditional centralized extraction system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111262726B (en) | Configuration information updating method and device and computer readable storage medium | |
| CN113296929A (en) | Resource matching method, device and system based on cloud computing | |
| CN109714249B (en) | Method and related device for pushing applet messages | |
| CN114090179A (en) | Migration method and device of stateful service and server | |
| CN106407395A (en) | A processing method and device for data query | |
| CN110569259A (en) | Method and device for processing mass data | |
| CN113051102A (en) | File backup method, device, system, storage medium and computer equipment | |
| CN107451202B (en) | Data access method and equipment | |
| CN110765125B (en) | Method and device for storing data | |
| CN112202909A (en) | Online upgrading method and system for computer storage system | |
| CN110851433B (en) | Key optimization method for key value storage system, storage medium, electronic device and system | |
| CN111475468A (en) | Log access method, device, equipment and storage medium of newly added system | |
| CN109976896B (en) | Service re-ranking processing method and device | |
| CN114513469A (en) | Traffic shaping method and device for distributed system and storage medium | |
| CN116095074A (en) | Resource allocation method, device, related equipment and storage medium | |
| CN114020446A (en) | Cross-multi-engine routing processing method, device, equipment and storage medium | |
| CN112148496A (en) | Energy efficiency management method and device for computing storage resources of super-fusion virtual machine and electronic equipment | |
| CN116502235B (en) | TMS remote management method and system | |
| CN108009190B (en) | Data export method, data export system and computer device | |
| CN109492422A (en) | A kind of data processing method and system based on user behavior information | |
| CN110837536B (en) | Information processing method, device and storage medium | |
| CN109919808B (en) | Trademark intelligent declaration method, medium, computer equipment and system | |
| CN118051494A (en) | Method and device for determining SPARK target parameters and electronic equipment | |
| CN109885330B (en) | Virtual machine generation method and device | |
| CN112711627B (en) | Data importing method, device and equipment of Greemplum database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191213 |
|
| RJ01 | Rejection of invention patent application after publication |