CN104750806A - Large data query method and system - Google Patents
Large data query method and system Download PDFInfo
- Publication number
- CN104750806A CN104750806A CN201510132308.6A CN201510132308A CN104750806A CN 104750806 A CN104750806 A CN 104750806A CN 201510132308 A CN201510132308 A CN 201510132308A CN 104750806 A CN104750806 A CN 104750806A
- Authority
- CN
- China
- Prior art keywords
- query
- inquiry
- data
- client
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a large data query method and system. The large data query system comprises a query configuration unit, a query scheme construction unit, a query unit, a query storage unit, a data source adaptation unit and a pre-query set unit. The large data query method includes the follow steps that 1, service query rules and a corresponding relation between the service query rules and query conditions are configured; 2, a query history is stored; 3, adaptation is performed on different data source types; 4, a pre-query threshold value and a query tag are set. The large data query method and system have the advantages that compared with the prior art, through configuration of the service query rules and the corresponding relation between the service query rules and the query conditions, a corresponding query rule can be extracted according to the query conditions, an optimized query scheme is constructed to adapt to different query conditions, and the query efficiency is guaranteed.
Description
Technical field
The present invention relates to technical field of information retrieval, specifically a kind of querying method of large data and system.
Background technology
Along with the development of internet, the data volume in each operation system is increasingly huge, particularly telecommunications and internet industry all the more so.In the face of these large data carry out the demand of inquiring about, performance and the efficiency of inquiry are stood in the breach, when the condition such as server hardware and database configuration is fixing, according to traditional inquiry mode, its search efficiency constantly increases along with data volume, can constantly reduce, the response time of user's inquiry can be more and more slower, even there will be owing to inquiring about the situation causing database to use.Although can consider to introduce search engine to realize the query manipulation of mass data, but because search engine effectively can not understand business datum, the express-analysis realized for the business datum of specific industry is inquired about, introduce that search engine difficulty is comparatively large, workload and implementation complexity too high.
Summary of the invention
Technical assignment of the present invention is to provide a kind of querying method and system of large data.
Technical assignment of the present invention realizes in the following manner, and this querying method is:
1) corresponding relation of configuration service rule searching and service inquiry rule and querying condition: according to the analysis result of the inquiry request to client, corresponding service inquiry rule is extracted, according to the service inquiry rule construct query scheme extracted by the querying condition in inquiry request; According to query scheme data query, Query Result is returned to client;
2) storing queries history: record at least one in historical query condition and historical query result corresponding to historical query condition and historical query scheme in query history, if the querying condition in the inquiry request of client is identical with historical query condition, then return corresponding historical query result according to query history directly to client, or according to historical query proposal inquiry data corresponding in query history, Query Result is returned to client;
3) adaptation is carried out to different data source types;
4) pre-inquiry threshold value and inquiry mark are set: inquire about threshold value in advance when this data volume inquired reaches, the data this inquired return to client as this Query Result, and by inquiry mark, this Query Result are marked; And upper once inquire about by same query scheme time according to inquiry mark continue inquiry.
Described configuration service rule searching is the corresponding granularity of query of configuration different business attribute.
The inquiry system of these large data comprises query configuration unit, query scheme tectonic element and query unit;
Query configuration unit: for configuration service rule searching and service inquiry rule and the corresponding relation of querying condition;
Query scheme tectonic element: for according to the analysis result to the inquiry request of client, extract corresponding service inquiry rule, according to the service inquiry rule construct query scheme extracted by the querying condition in inquiry request;
Query unit: for according to query scheme data query, Query Result is returned to client.
The inquiry system of described large data also comprises inquiry storage unit: for storing queries history, the at least one in historical query condition and historical query result corresponding to historical query condition and historical query scheme is recorded in query history, when the querying condition of query unit in the inquiry request of client is identical with historical query condition, corresponding historical query result is returned directly to client according to query history, or according to historical query proposal inquiry data corresponding in query history, Query Result is returned to client.
The inquiry system of described large data also comprises data source adaptation unit, for carrying out adaptation to different data source types.
The inquiry system of described large data also comprises inquires about setting unit in advance, for arranging pre-inquiry threshold value and inquiry mark; Query unit is when the data volume that this inquires reaches pre-inquiry threshold value, and the data this inquired return to client as this Query Result; Pre-inquiry setting unit is marked this Query Result by inquiry mark; Query unit upper once inquire about by same query scheme time according to inquiry mark continue inquiry.
The querying method of a kind of large data of the present invention and system are compared to the prior art, by configuration querying rule and the corresponding relation with querying condition thereof, corresponding rule searching can be extracted according to querying condition, construct the query scheme of optimization, be adapted to different querying conditions, ensure that the efficiency of inquiry.
Accompanying drawing explanation
Accompanying drawing 1 is a kind of configuration diagram of inquiry system of large data;
Accompanying drawing 2 is large data query adaptive arbitrary source system architecture schematic diagram;
Accompanying drawing 3 is the workflow diagram of large data query system;
Accompanying drawing 4 is the business diagnosis process flow diagram of large data query;
Accompanying drawing 5 is large data query processing flow chart;
Accompanying drawing 6 to be dealt with relationship figure for data source adaptation module;
Accompanying drawing 7 is large data query adaptive arbitrary source system file data source processing flow chart.
Embodiment
Embodiment 1:
The inquiry system of these large data is set, comprises query configuration unit, query scheme tectonic element and query unit.
Query configuration unit: for configuration service rule searching and service inquiry rule and the corresponding relation of querying condition.
Query scheme tectonic element: for according to the analysis result to the inquiry request of client, extract corresponding service inquiry rule, according to the service inquiry rule construct query scheme extracted by the querying condition in inquiry request.
Query unit: for according to query scheme data query, Query Result is returned to client.
The inquiry system of described large data also comprises inquiry storage unit: for storing queries history, the at least one in historical query condition and historical query result corresponding to historical query condition and historical query scheme is recorded in query history, when the querying condition of query unit in the inquiry request of client is identical with historical query condition, corresponding historical query result is returned directly to client according to query history, or according to historical query proposal inquiry data corresponding in query history, Query Result is returned to client.
The inquiry system of described large data also comprises data source adaptation unit, for carrying out adaptation to different data source types.
The inquiry system of described large data also comprises inquires about setting unit in advance, for arranging pre-inquiry threshold value and inquiry mark; Query unit is when the data volume that this inquires reaches pre-inquiry threshold value, and the data this inquired return to client as this Query Result; Pre-inquiry setting unit is marked this Query Result by inquiry mark; Query unit upper once inquire about by same query scheme time according to inquiry mark continue inquiry.
This querying method is:
1) corresponding relation of configuration service rule searching and service inquiry rule and querying condition: according to the analysis result of the inquiry request to client, corresponding service inquiry rule is extracted, according to the service inquiry rule construct query scheme extracted by the querying condition in inquiry request; According to query scheme data query, Query Result is returned to client; Configuration service rule searching is the corresponding granularity of query of configuration different business attribute.
2) storing queries history: record at least one in historical query condition and historical query result corresponding to historical query condition and historical query scheme in query history, if the querying condition in the inquiry request of client is identical with historical query condition, then return corresponding historical query result according to query history directly to client, or according to historical query proposal inquiry data corresponding in query history, Query Result is returned to client.
3) adaptation is carried out to different data source types.
4) pre-inquiry threshold value and inquiry mark are set: inquire about threshold value in advance when this data volume inquired reaches, the data this inquired return to client as this Query Result, and by inquiry mark, this Query Result are marked; And upper once inquire about by same query scheme time according to inquiry mark continue inquiry.
Embodiment 2:
This querying method concrete operation step is as follows:
Step 1: control module receives the inquiry request of user, connection data source adaptation module, data source adaptation module, according to system configuration, calls file adaptation processing method.Control module is inquired about from file, checks whether the inquiry that there is same request condition and completed.
Step 2: if had, directly obtains data and returns from the inquiry event memory that inquiry memory module has completed.
Step 3: if do not had, then check whether as inquiring about first.
Step 4: if inquire about first, then enter step 6.
Step 5: if not inquiring about first, enter inquiry memory module, connection data source adaptation module, data source adaptation module is according to system configuration, call file adaptation processing method, by obtaining query scheme and inquiry mark continuation data query, namely from the place inquiry of mark last time, whether inquiry mark covers all data meeting querying condition for the querying flow marking this inquiry, namely all data whether have been inquired about, if one query flow process has not all been inquired about, subsequent query can continue the position continuation inquiry from data query last time, namely inquiry is continued from inquiry mark.
Step 6: enter business diagnosis module, connection data source adaptation module, data source adaptation module, according to system configuration, calls file adaptation processing method.Business diagnosis module can check the query scheme whether existing and ask to mate, if existed, exports query scheme.If there is no, then according to the service inquiry rule of configuration, querying condition is analyzed, by single condition, combination condition and service inquiry rule match, exports the query scheme for this request.
Step 7: inquiry memory module, according to the query scheme that business diagnosis module exports, connection data source adaptation module, data source adaptation module, according to system configuration, calls file adaptation processing method.Inquiry memory module data query from file, according to the mode determining a kind of traversal queries data in query scheme, is inquired about data by which.
Step 8: during data query, according to query scheme traversal queries data, often travel through one to take turns, need to do following judgement: judge whether to cover querying condition completely, all data that condition of namely having inquired about is corresponding, if inquired about all data, then terminate querying flow, exit inquiry memory module, otherwise continue to judge whether to reach pre-data volume of inquiring about, the data volume of pre-inquiry, the i.e. predetermined data volume of once asking correspondence returning to user of system configuration.If reach the data volume of pre-inquiry, then this inquiry is marked, namely inquire which Data Position, can continue next time to inquire about from this Data Position.If do not reach the data volume of inquiry in advance, then continue next round inquiry, until looked into all data or reached pre-data volume of inquiring about, exit querying flow.
By embodiment above, described those skilled in the art can be easy to realize the present invention.But should be appreciated that the present invention is not limited to above-mentioned several embodiments.On the basis of disclosed embodiment, described those skilled in the art can the different technical characteristic of combination in any, thus realizes different technical schemes.
Claims (6)
1. a querying method for large data, is characterized in that, this querying method is:
1) corresponding relation of configuration service rule searching and service inquiry rule and querying condition: according to the analysis result of the inquiry request to client, corresponding service inquiry rule is extracted, according to the service inquiry rule construct query scheme extracted by the querying condition in inquiry request; According to query scheme data query, Query Result is returned to client;
2) storing queries history: record at least one in historical query condition and historical query result corresponding to historical query condition and historical query scheme in query history, if the querying condition in the inquiry request of client is identical with historical query condition, then return corresponding historical query result according to query history directly to client, or according to historical query proposal inquiry data corresponding in query history, Query Result is returned to client;
3) adaptation is carried out to different data source types;
4) pre-inquiry threshold value and inquiry mark are set: inquire about threshold value in advance when this data volume inquired reaches, the data this inquired return to client as this Query Result, and by inquiry mark, this Query Result are marked; And upper once inquire about by same query scheme time according to inquiry mark continue inquiry.
2. the querying method of a kind of large data according to claim 1, is characterized in that, described configuration service rule searching is the corresponding granularity of query of configuration different business attribute.
3. the inquiry system of a kind of large data according to claim 1, is characterized in that, the inquiry system of these large data comprises query configuration unit, query scheme tectonic element and query unit;
Query configuration unit: for configuration service rule searching and service inquiry rule and the corresponding relation of querying condition;
Query scheme tectonic element: for according to the analysis result to the inquiry request of client, extract corresponding service inquiry rule, according to the service inquiry rule construct query scheme extracted by the querying condition in inquiry request;
Query unit: for according to query scheme data query, Query Result is returned to client.
4. the inquiry system of a kind of large data according to claim 3, it is characterized in that, the inquiry system of described large data also comprises inquiry storage unit: for storing queries history, the at least one in historical query condition and historical query result corresponding to historical query condition and historical query scheme is recorded in query history, when the querying condition of query unit in the inquiry request of client is identical with historical query condition, corresponding historical query result is returned directly to client according to query history, or according to historical query proposal inquiry data corresponding in query history, Query Result is returned to client.
5. the inquiry system of a kind of large data according to claim 3, is characterized in that, the inquiry system of described large data also comprises data source adaptation unit, for carrying out adaptation to different data source types.
6. the inquiry system of a kind of large data according to claim 3, is characterized in that, the inquiry system of described large data also comprises inquires about setting unit in advance, for arranging pre-inquiry threshold value and inquiry mark; Query unit is when the data volume that this inquires reaches pre-inquiry threshold value, and the data this inquired return to client as this Query Result; Pre-inquiry setting unit is marked this Query Result by inquiry mark; Query unit upper once inquire about by same query scheme time according to inquiry mark continue inquiry.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510132308.6A CN104750806A (en) | 2015-03-25 | 2015-03-25 | Large data query method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510132308.6A CN104750806A (en) | 2015-03-25 | 2015-03-25 | Large data query method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104750806A true CN104750806A (en) | 2015-07-01 |
Family
ID=53590490
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510132308.6A Pending CN104750806A (en) | 2015-03-25 | 2015-03-25 | Large data query method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104750806A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104750797A (en) * | 2015-03-13 | 2015-07-01 | 深圳市梦域科技有限公司 | Data reading method and terminal |
| CN109710641A (en) * | 2018-12-17 | 2019-05-03 | 浩云科技股份有限公司 | A kind of inquiry processing method and system of mass data |
| CN109949714A (en) * | 2019-04-23 | 2019-06-28 | 跃动创意媒体一人有限公司 | A kind of smart electronicsization mark system |
| CN110071951A (en) * | 2018-01-24 | 2019-07-30 | 江苏迪纳数字科技股份有限公司 | Data query display systems and method under the conditions of a kind of big data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102279849A (en) * | 2010-06-09 | 2011-12-14 | 中兴通讯股份有限公司 | Method and system for big data query |
| WO2012109786A1 (en) * | 2011-02-16 | 2012-08-23 | Empire Technology Development Llc | Performing queries using semantically restricted relations |
-
2015
- 2015-03-25 CN CN201510132308.6A patent/CN104750806A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102279849A (en) * | 2010-06-09 | 2011-12-14 | 中兴通讯股份有限公司 | Method and system for big data query |
| WO2012109786A1 (en) * | 2011-02-16 | 2012-08-23 | Empire Technology Development Llc | Performing queries using semantically restricted relations |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104750797A (en) * | 2015-03-13 | 2015-07-01 | 深圳市梦域科技有限公司 | Data reading method and terminal |
| CN110071951A (en) * | 2018-01-24 | 2019-07-30 | 江苏迪纳数字科技股份有限公司 | Data query display systems and method under the conditions of a kind of big data |
| CN109710641A (en) * | 2018-12-17 | 2019-05-03 | 浩云科技股份有限公司 | A kind of inquiry processing method and system of mass data |
| CN109949714A (en) * | 2019-04-23 | 2019-06-28 | 跃动创意媒体一人有限公司 | A kind of smart electronicsization mark system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107247808B (en) | Distributed NewSQL database system and picture data query method | |
| US20200264923A1 (en) | Information Processing Method and Apparatus | |
| CN106897334B (en) | Question pushing method and equipment | |
| CN105718455B (en) | A kind of data query method and device | |
| US8122008B2 (en) | Joining tables in multiple heterogeneous distributed databases | |
| CN107590123B (en) | Vehicular middle-location context reference resolution method and device | |
| CN104090889A (en) | Method and system for data processing | |
| CN108255958A (en) | Data query method, apparatus and storage medium | |
| CN104516979A (en) | Data query method and data query system based on quadratic search | |
| CN102279849A (en) | Method and system for big data query | |
| CN106293891B (en) | Multidimensional investment index monitoring method | |
| CN104750806A (en) | Large data query method and system | |
| CN111046041B (en) | Data processing method and device, storage medium and processor | |
| US20130159347A1 (en) | Automatic and dynamic design of cache groups | |
| CN107480260B (en) | Big data real-time analysis method and device, computing equipment and computer storage medium | |
| AU2021244852B2 (en) | Offloading statistics collection | |
| CN103152391A (en) | Journal output method and device | |
| CN108256718A (en) | Declaration form service role distribution method, device, computer equipment and storage device | |
| WO2021012861A1 (en) | Method and apparatus for evaluating data query time consumption, and computer device and storage medium | |
| CN108920523A (en) | Data query method, apparatus, equipment, system and medium on block chain | |
| CN114925041A (en) | Data query method and device | |
| US10262033B2 (en) | Method for query execution planning | |
| CN113342876B (en) | Data fuzzy query method and device of multi-tenant CRM system in SaaS environment | |
| US20080082516A1 (en) | System for and method of searching distributed data base, and information management device | |
| CN108153874B (en) | Rapid paging method for large-data high-time-consumption query result set |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150701 |