CN102279849A - Method and system for big data query - Google Patents
Method and system for big data query Download PDFInfo
- Publication number
- CN102279849A CN102279849A CN2010101959794A CN201010195979A CN102279849A CN 102279849 A CN102279849 A CN 102279849A CN 2010101959794 A CN2010101959794 A CN 2010101959794A CN 201010195979 A CN201010195979 A CN 201010195979A CN 102279849 A CN102279849 A CN 102279849A
- Authority
- CN
- China
- Prior art keywords
- query
- inquiry
- data
- scheme
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a system for big data query. The method comprises the steps of configuring business query rules and the corresponding relationship between the business query rules and query conditions; extracting corresponding business query rules from the query conditions of client query requests according to analysis results of the query requests, and constructing a query scheme based on the extracted business query rules; and feeding query results back to the client, in dependence on the query data of the query scheme. In the invention, by configuration of the corresponding relationship between the query rules and the query conditions, corresponding query rules are extracted according to the query conditions, and the optimized query scheme is constructed. The query scheme obtained in this way is suitable for different query conditions and guarantees the query efficiency.
Description
Technical field
The present invention relates to technical field of information retrieval, in particular, relate to a kind of method and system of big data query.
Background technology
Along with Internet development, the data volume in each operation system is huge day by day, and particularly telecommunications and internet industry are all the more so.In the face of the demand that these big data are inquired about, the performance of inquiry and efficient are to stand in the breach.
Under the situation that conditions such as server hardware and database configuration are fixed, if adopt traditional inquiry mode, its search efficiency can constantly reduce along with data volume constantly increases, the response time of user inquiring can be more and more slower, even can occur because the situation that inquiry causes database to use.Although can consider to introduce the query manipulation that search engine is realized mass data, but because search engine can not effectively be understood business datum, realize express-analysis inquiry, introduce that the search engine difficulty is big, workload and implementation complexity be too high at the business datum of specific industry.
Summary of the invention
The main technical problem to be solved in the present invention is, a kind of method and system of big data query are provided, and can export Query Result efficiently, apace.
For solving the problems of the technologies described above, the present invention has adopted following technical scheme:
A kind of querying method of big data comprises:
The corresponding relation of configuration service rule searching and described service inquiry rule and querying condition;
According to analysis result, extract the corresponding business rule searching by the querying condition in the described query requests, according to the service inquiry rule structure query scheme that extracts to the query requests of client;
According to described query scheme data query, Query Result is returned to client.
In a kind of embodiment of the method for the invention, described configuration service rule searching is the corresponding granularity of query of configuration different business attribute.
In a kind of embodiment of the method for the invention, also comprise: storing queries history, record in historical query condition and described historical query condition corresponding historical Query Result and the historical query scheme at least one in the described query history, if the querying condition in the query requests of client is identical with the historical query condition, then directly return the corresponding historical Query Result to client according to query history, perhaps, Query Result is returned to client according to corresponding historical query scheme data query in the query history.
In a kind of embodiment of the method for the invention, also comprise: carry out adaptive different data source types.
In a kind of embodiment of the method for the invention, also comprise: pre-inquiry threshold value and inquiry mark are set; When this data volume that inquires reaches described pre-inquiry threshold value, these data that inquire are returned to client as this Query Result, and this Query Result is carried out mark by the inquiry mark; And when inquiring about by same query scheme, continue inquiry according to the inquiry mark next time.
The present invention also provides a kind of inquiry system of big data, comprising:
The query configuration unit is used for configuration service rule searching and the described service inquiry rule and the corresponding relation of querying condition;
The query scheme tectonic element is used for the analysis result of basis to the query requests of client, extracts the corresponding business rule searching by the querying condition in the described query requests, according to the service inquiry rule structure query scheme that extracts;
Query unit is used for according to described query scheme data query Query Result being returned to client.
In a kind of embodiment of system of the present invention, described configuration service rule searching is the corresponding granularity of query of configuration different business attribute.
In a kind of embodiment of system of the present invention, also comprise the inquiry storage unit, be used for storing queries history, record in historical query condition and described historical query condition corresponding historical Query Result and the historical query scheme at least one in the described query history, when the querying condition of described query unit in the query requests of client is identical with the historical query condition, directly return the corresponding historical Query Result according to query history to client, perhaps, Query Result is returned to client according to corresponding historical query scheme data query in the query history.
In a kind of embodiment of system of the present invention, also comprise the data source adaptation unit, be used for carrying out adaptive to different data source types.
In a kind of embodiment of system of the present invention, comprise that also pre-inquiry is provided with the unit, be used to be provided with pre-inquiry threshold value and inquiry mark; When described query unit reaches described pre-inquiry threshold value in this data volume that inquires, these data that inquire are returned to client as this Query Result; Described pre-inquiry is provided with the unit and by the inquiry mark this Query Result is carried out mark; Described query unit continues inquiry according to the inquiry mark next time when inquiring about by same query scheme.
The present invention by configuration querying rule and with the corresponding relation of querying condition, thereby can extract corresponding rule searching according to querying condition, construct the query scheme of optimization.The query scheme that obtains like this is adapted to different querying conditions, has guaranteed the efficient of inquiry.
Description of drawings
Fig. 1 is the big data query system configuration diagram of the embodiment of the invention;
Fig. 2 is the adaptive arbitrary source system architecture of the big data query synoptic diagram of the embodiment of the invention;
Fig. 3 is the workflow diagram of the big data query system of the embodiment of the invention;
Fig. 4 is the business diagnosis process flow diagram of the big data query of the embodiment of the invention;
Fig. 5 is the big data query processing flow chart of the embodiment of the invention;
Fig. 6 is the data source adaptation module of the embodiment of the invention figure that deals with relationship;
Fig. 7 is the adaptive arbitrary source system file data of the big data query source processing flow chart of the embodiment of the invention.
Embodiment
In conjunction with the accompanying drawings the present invention is described in further detail below by embodiment.
The big data enquire method of the embodiment of the invention, it mainly comprises:
The corresponding relation of configuration service rule searching and described service inquiry rule and querying condition;
According to analysis result, extract the corresponding business rule searching by the querying condition in the described query requests, according to the service inquiry rule structure query scheme that extracts to the query requests of client;
According to described query scheme data query, Query Result is returned to client.
Wherein, the service inquiry rule mainly is the corresponding granularity of query of the different business attribute of configuration service system.Service attribute, for example comprise time, type, rate or the like, for example a kind of service inquiry rule can be: the different time granularity of configuration different business attribute number correspondence, wherein, the minimum particle size of data that can configuration service attribute SP number is minute, and the minimum particle size of the data of configuration service attribute personal number is day.
And in the querying condition common with number and time as querying condition, therefore, corresponding relation by querying condition and service inquiry rule, find the pairing service inquiry rule of enquiry number in the querying condition, if for example the enquiry number of querying condition is the SP number, then the corresponding service rule searching is minute being that granularity is inquired about; Enquiry number is a personal number, and then the corresponding service rule searching is to be that granularity is inquired about with the sky.
If a kind of querying condition is: inquiry SP number data, query time is one day, be 24*60 inquiry then according to granularity division, return the data of a granularity to the user at every turn, and periodic refreshing can for example be set, after promptly returning a granularity data at every turn, timing to or after the data query of a granularity finishes, return the data of next granularity from the trend user.
Similarly, if a kind of querying condition is: querying individual number data, query time are two days, are twice inquiry according to granularity division then, return the data of a granularity to the user at every turn.
And for example, in the service inquiry rule, can dispose rush hour section, such as important festivals or holidays such as three days of 1 to 3 May, seven days of 1 to 7 October, seven days of New Year be rush hour section, if the query time scope in the querying condition comprises rush hour section, be the granularity data query by the hour then, if be non-rush hour during section, by the sky or month carry out data query for granularity.
And for example for the SP numbers of 8860 beginnings, because this type of SP class of business of carrying out is more, and the rate of every kind of message is all different, therefore can the configuration service rule searching for being that granularity is inquired about by the rate type.If the number of querying condition is 8860 beginnings, then finding the corresponding service rule searching according to corresponding relation is to carry out the granularity inquiry by the rate type of data, then only inquires about a kind of data of rate type at every turn.When the user needs follow-up data, inquire about the data of next granularity again.
As shown in Figure 1, a kind of system that realizes above-mentioned big data enquire method comprises: control module, business diagnosis module, enquiry module.
Control module: link to each other with business diagnosis module, enquiry module.Control module receives the query requests of client, controls the flow process relation between each module.Control module judges whether to exist the inquiry of having finished of request correspondence according to the querying condition in the query requests of client, directly enters enquiry module if exist, otherwise will enter the business diagnosis module, makes subsequent treatment.
Business diagnosis module: link to each other with control module.After the request of receiving control module, whether the business diagnosis module has existed the query scheme of request condition correspondence from the business experience library lookup, if not then analyze the querying condition of client-requested, service inquiry rule according to configuration, querying condition and service inquiry rule are carried out The matching analysis, the query scheme that is constructed a kind of optimization by the service inquiry rule of mating outputs to control module, and then the query scheme of the business diagnosis module being exported by control module is delivered to enquiry module.
Enquiry module: link to each other with control module.Enquiry module according to the query scheme data query, returns Query Result to client after receiving the next query scheme of control module transmission.Enquiry module also can attach storage unit and be used for the storing queries result, at this moment, has the enquiry module of storage unit to be called the inquiry memory module with subsidiary.
The inquiry memory module can storing queries history.Query history can be one or more record, and each bar record has write down historical query condition and this historical query condition corresponding historical query scheme and/or historical query result.Control module is after receiving the query requests of client, and the querying condition of analysis and consult request at first extracts query history from the inquiry memory module, and whether the querying condition of comparison query requests is identical with a certain historical query condition.If identical, then can directly return the corresponding historical Query Result according to query history to client.
Directly return historical results, require not only to store the historical query condition, also will store the corresponding historical Query Result to client.Obviously, although this mode can further improve the data query response speed significantly, need bigger storage space to come the storing queries result.Therefore, the scheme of another kind of compromise, storage historical query condition and corresponding historical query scheme.Obviously, the historical query scheme is with respect to the historical query result, and required storage space will significantly reduce, and can save the time that re-constructs query scheme simultaneously, can further improve the data query response speed.
When the inquiry memory module is carried out traversal queries according to query scheme, can adopt two kinds of inquiry modes, a kind of is complete inquiry mode, be that the poll-final condition is: judge whether to finish all data queries, this inquiry mode can once be finished whole inquiries, and shortcoming is that query responding time is longer relatively.Another kind is pre-inquiry mode, and the querying condition of this mode is: judge whether to finish all data queries or judge that the data of current inquiry reach the data volume of pre-inquiry, satisfy one of these two conditions, promptly withdraw from inquiry.Just also be provided with pre-inquiry threshold value, as long as this data volume that inquires reaches pre-inquiry threshold value, the data that this can be inquired return to client as this Query Result.During greater than pre-inquiry threshold value, obviously want to obtain complete Query Result in the data total amount, need repeatedly that inquiry just can obtain, for this reason, in pre-inquiry mode, also be provided with the inquiry mark, this Query Result is carried out mark by the inquiry mark; And when inquiring about by same query scheme, continue inquiry according to the inquiry mark next time.For example, data sort in certain sequence, during inquiry, in this order data are inquired about, and when inquiring data A, reach pre-inquiry threshold value, then finish this inquiry, and with the inquiry mark data A are carried out mark; Begin in the inquiry of same query scheme next time, then find the inquiry mark, data after data A continue inquiry, same, inquiry next time finishes, when for example inquiring data B, reach pre-inquiry threshold value, then with the inquiry mark data B is carried out mark, the rest may be inferred, all inquires about up to all data to finish.
Because data are closely related with the data source type, if the change of data source type for example has database to change into file, each module of system all needs to revise change, for this reason, provides a kind of expanding system as shown in Figure 2.This expanding system comprises control module, business diagnosis module, data source adaptation module, inquiry memory module.The function of control module, business diagnosis module, inquiry memory module is identical with system shown in Figure 1, repeats no more.The key distinction of itself and system shown in Figure 1 is to have increased the data source adaptation module, the data source adaptation module links to each other with control module, business diagnosis module, inquiry memory module, all data are all obtained by this data source adaptation module, by the data source adaptation module different data sources is carried out adaptation processing, the data source that the data source adaptation module is suitable for comprises various data storage methods such as database, file, internal memory.If system need increase or change data source, only need to revise the data source adaptation module or revise the systematic parameter that disposes and to finish other module in the not influence system.By such expansion, can make the present invention be applicable to the miscellaneous service data source, data source as different modes such as database, internal memory, files, the data storage method that makes whole inquiry system and concrete data source is without any relation, can factor do not cause the variation of system architecture according to storage mode changes.
As shown in Figure 3, the workflow of big data query system comprises:
Step S101: control module receives the query requests of user's (client), can inquire about the query requests whether this user had had the same terms with user's request condition and user profile as combination condition.If had the identical request of finishing inquiry, then execution in step S103.If no, then judge whether to be inquiry first.
If inquiry first is delivered to next step with querying condition, execution in step S102.
If not inquiry first, then execution in step S103 makes subsequent treatment.
Step S102: the business diagnosis module is exported a kind of query scheme at this request by the querying condition of request, the business rule of configuration.
Step S103: the inquiry memory module, divide three kinds of different situations to handle respectively:
Finish identical query requests if having, directly obtaining the data that existed from event memory returns to the user;
If not inquiry first, then use the query scheme data query that has existed, and the storing queries result, from event memory, obtain data and return this user;
If inquiry is first then inquired about according to the query scheme of step S102 output, and Query Result is stored, from event memory, obtain data and return this user.
The inquiry memory module can also be provided with pre-inquiry mechanism, i.e. qualified total data has not been inquired about in one query, but just finishes after only inquiring predetermined partial data.As mentioned above, if when there has been the inquiry of finishing in user's query requests, directly from before obtain data the event memory of inquiry and return to the user.And if the inquiry of not finished is then inquired about by the query scheme that the control module transmission comes, under pre-inquiry mechanism, need judge that whether query scheme is for inquiring about first; If not inquiry first, then need to continue the inquiry follow-up data according to the inquiry mark; If for inquiry first, then according to query scheme, the data volume of the pre-inquiry of inquiry.
If the data volume of pre-inquiry does not reach, then continue this pre-inquiry, if reach the data volume of pre-inquiry, then the Data Position of current inquiry is marked, use for subsequent query, withdraw from the inquiry memory module;
If querying condition is not all covered, promptly inquire about qualified all, then inquire about in advance in the manner described above, and if querying condition all covers, then inquiry is designated completely, withdraw from the inquiry memory module.
The described business diagnosis module of step S102, when receiving the querying condition of control module, the main flow process of its business diagnosis comprises as shown in Figure 4:
Configuration service systematic parameter and service inquiry rule, systematic parameter and service inquiry rule can be carried out initial configuration when system start-up, and can after carry out edit-modify.For example, if follow-up have a new rule, then increase corresponding configuration.Wherein, systematic parameter is some primary datas that need dispose when using this system, as the data Source Type, inquire about maximum number of user that maximum amount of data, ephemeral data storage maximum time, ephemeral data maximum storage, system support, inquiry ordering (descending, ascending order) etc. in advance.The service inquiry rule can be understood as the corresponding granularity of query of the different business attribute of operation system, with the number is that querying condition describes, as number with 139,138, beginnings such as 133 think the personal user, 66021 the beginning be SP user, two kinds of user's data amounts have very big-difference, general about 10 of one day size of message of personal user, and the data volume of SP user every day can reach 100,000, perhaps 1,000,000 ranks, therefore according to both different service attributes, just need the rule searching of this two classes number to be done corresponding configuration by different granularity of query, for example the personal user by day or the inquiry of time period of several days, and SP user just needs by the hour or minute rank inquire about.And for example, in the personal user, the transmitting-receiving note amount of age in the crowd every day below 22 years old is bigger, the transmitting-receiving note amount of crowd more than 22 years old is fewer, can be according to crowd's the range of age of querying condition number, and the configuration service rule searching, as the age when the number inquiry note of the crowd below 22 years old, by week is to inquire about in the cycle, and the age the crowd more than 22 years old, then monthly inquire about for the cycle.
The querying condition of analysis request at first, going forward side by side passes through tests the storehouse comparison, promptly from the experience library lookup, whether corresponding query scheme is arranged.Query scheme and the relation between the querying condition, i.e. historical query condition and corresponding historical query scheme that the business diagnosis module has been exported have been comprised in the experience storehouse.
If do not find the query scheme that has existed, then, mate from the service inquiry rule that has been configured by single conditioned disjunction combination condition by the service inquiry rule of configuration, filter the strictly all rules that is fit to this querying condition.The service inquiry rule is a kind of dynamic-configuration to the operation system data query conditions.As different rule searching of customization such as according to number prefix, suffix, length, press the different rule searching of query time scope formulation.
By query analysis, extract the corresponding business rule searching by querying condition, be configured to a kind of query scheme and output of optimization by the service inquiry rule.
This pre-data query upper limit according to system configuration, can further optimize scheme, be that qualified data have a lot, then an inquiry character share the partial data of this request of family, for example system is the WEB system, then for example only inquire about a page data of the WEB page, after the user carries out page turn over operation, continue the inquiry of next page data again.
Business diagnosis outputs to control module with this querying condition and query scheme, and control module passes to the inquiry memory module with query scheme (when storing queries is historical, also needing to transmit querying condition).
(mode 1) as shown in Figure 5 is that example is elaborated to the embodiment of the invention with the database data.
1, control module receives query requests, at first checks the inquiry of having finished that whether has had the same queries condition.
If 2 have, directly from the inquiry event memory that the inquiry memory module has been finished, obtain data and return.Otherwise whether check for inquiring about first.
3 if inquiry first then enters step 5.
4, if not inquiry first, enter the inquiry memory module, by obtaining query scheme and inquiry mark continuation data query, the i.e. place inquiry that identified from last time, whether the querying flow that the inquiry mark is used for this inquiry of mark covers the data that all meet querying condition, promptly whether has inquired about all data, does not finish if the one query flow process is all inquired about, subsequent query can continue promptly to continue to inquire about from the inquiry sign from the position continuation inquiry of data query last time.
5, enter the business diagnosis module, check the query scheme that whether exists and ask coupling.If exist, the query scheme of output coupling.If there is no, then according to the service inquiry rule of configuration, querying condition is analyzed, by single condition, combination condition and service inquiry rule match, output needle is to the query scheme of this request.
6, according to the query scheme of business diagnosis module output, data are inquired about, will determine a kind of mode of traversal queries data in the query scheme, data are inquired about by this mode.
7, during data query, according to query scheme traversal queries data, every traversal one is taken turns, and needs to do to judge:
Judge whether to cover fully querying condition, all data of condition correspondence have promptly been inquired about, if inquired about all data, then finish querying flow, withdraw from the inquiry memory module, otherwise continue to judge whether to reach the data volume of pre-inquiry, the data volume of pre-inquiry, the i.e. corresponding predetermined data volume of once request that returns to the user of system configuration.If reach the data volume of pre-inquiry, then this inquiry is marked, promptly inquire which Data Position, can continue next time from this Data Position inquiry.If do not reach the data volume of pre-inquiry, then continue next round inquiry, up to having looked into all data or having reached the data volume of pre-inquiry, withdraw from querying flow.
During according to the query scheme traversal queries, need the result of each traversal queries be stored.
8, return data responds to the user from event memory.
By the querying condition in the analysis and consult request, export a kind of optimization query scheme at request, can according to query scheme by different level, mode such as granularity inquires about data, by introducing pre-data query quantitative limitation, it is minimum that system loading is dropped to, can farthest reduce the response time, improve system response time.
As shown in Figure 6, the data source adaptation module in the embodiment of the invention is elaborated.
The data source adaptation module provides unified inquiry, storage inlet, and whole inquiry system and concrete data source are had nothing to do, i.e. the realization of big data query system can be irrelevant with concrete data storage method.
The data source adaptation module is made adaptation processing by data source adapter to different data sources, provides unified entrance to other module of system, in the unified processing of data source adaptation module.
The data source adaptation module can be done adaptation processing according to different modes:
One, the data source type by each module of system configuration realizes
Data source type according to system configuration, respectively the disparate modules data source is made adaptation processing, data source type configuration as control module is a file, the data source type configuration of business diagnosis module is a file, the data source type configuration of inquiry memory module is a database, then each module is with after the data source adaptation module links to each other, the data source adaptation module is automatically according to the data source type of each module of system configuration, call different adaptation processing methods, service data, as control module, the data interaction of business diagnosis module is by the file adaptation processing method of data source adaptation module, and the data interaction of inquiry memory module is by the database adaptation processing method of data source adaptation module.
Two, the parameter by interface Data transmission Source Type realizes
After data source adaptation module and control module, business diagnosis module, inquiry memory module link to each other, the data source type that the data source module is transmitted during according to the interface of each module invokes data source adaptation module is made adaptation processing, according to different types, find the adaptation processing method in corresponding data source, as the file adaptation processing method, handle the file data source; The database adaptation processing method, the process database data source; The internal memory adaptation processing method is handled the different corresponding data sources of matched processing method processing such as internal storage data source.
The data source type that claim data source adaptation module is supported in the data source adaptation module, the adaptation processing method of the corresponding a kind of data source of each type, and the data source type is offered other module, as control module, business diagnosis module, inquiry memory module, other module is with after the data source adaptation module is connected, when calling unified inquiry, the memory interface of data source adaptation module, import the data source type into parameter mode, by the data source adaptation module according to data source type specific data source adaptation processing mode.
By shown in Figure 6, after the increase data source adapter module, whole inquiry system and data source are irrelevant.Can use arbitrary source to carry out data query, storage.It is as follows to increase data source adaptation module aftertreatment flow process:
Control module receives query requests, connects the data source adaptation module, and whether inquiry has the query note of the same terms of having finished from file.
The business diagnosis module receives the querying condition of control module, connects the data source adaptation module, obtains service inquiry rule, system configuration parameter, query scheme experience storehouse etc., the output query scheme.
The inquiry memory module according to the query scheme that receives, connects the data source adaptation module, data query, and Query Result stored.
Inquire about memory module return data from the Query Result of having stored at last and give user, process ends.
After increasing the data source adapter module, if system needs to increase, change data source, the adaptation processing method that then only needs change system relevant configuration or realization corresponding data Source Type, do not need other module of system is done any change, strengthened the extendability of system, made it be applicable to arbitrary source.
(mode 2) as shown in Figure 7 is example with the file data source to be described the system of the adaptive arbitrary source of big data query of the embodiment of the invention.
The data source type of system configuration control module, business diagnosis module, inquiry memory module is a file.
The associative operation of data query and storage all inserts the data source adaptation module by unified interface and handles.Flow process need not anyly change, and the detailed process flow process is as follows:
1, control module receives user's query requests, connects the data source adaptation module, and the data source adaptation module is called the file adaptation processing method according to system configuration.Control module is inquired about from file, checks the inquiry that whether has existed the same request condition to finish.
If 2 have, directly from the inquiry event memory that the inquiry memory module has been finished, obtain data and return.
If 3 do not have, then whether check for inquiring about first.
4 if inquiry first then enters step 6.
5, if not inquiry first, enter the inquiry memory module, connect the data source adaptation module, the data source adaptation module is according to system configuration, call the file adaptation processing method, by obtaining query scheme and inquiry mark continuation data query, the i.e. place inquiry that identified from last time, whether the querying flow that the inquiry mark is used for this inquiry of mark covers the data that all meet querying condition, all data promptly whether have been inquired about, if the one query flow process is not all inquired about and finished, subsequent query can continue promptly to inquire about from the continuation of inquiry sign from the position continuation inquiry of data query last time.
6, enter the business diagnosis module, connect the data source adaptation module, the data source adaptation module is called the file adaptation processing method according to system configuration.The business diagnosis module can be checked the query scheme that whether exists and ask coupling, if exist, and the output query scheme.If there is no, then according to the service inquiry rule of configuration, querying condition is analyzed, by single condition, combination condition and service inquiry rule match, output needle is to the query scheme of this request.
7, inquiry memory module according to the query scheme of business diagnosis module output, connects the data source adaptation module, and the data source adaptation module is called the file adaptation processing method according to system configuration.Inquiry memory module data query from file according to the mode of determining a kind of traversal queries data in the query scheme, is inquired about data by this mode.
8, during data query, according to query scheme traversal queries data, every traversal one is taken turns, and needs to do to judge:
Judge whether to cover fully querying condition, all data of condition correspondence have promptly been inquired about, if inquired about all data, then finish querying flow, withdraw from the inquiry memory module, otherwise continue to judge whether to reach the data volume of pre-inquiry, the data volume of pre-inquiry, the i.e. corresponding predetermined data volume of once request that returns to the user of system configuration.If reach the data volume of pre-inquiry, then this inquiry is marked, promptly inquire which Data Position, can continue next time from this Data Position inquiry.If do not reach the data volume of pre-inquiry, then continue next round inquiry, up to having looked into all data or having reached the data volume of pre-inquiry, withdraw from querying flow.
During according to the query scheme traversal queries, need the result of each traversal queries be stored.Connect the data source adaptation module, the data source adaptation module is called the file adaptation processing method according to system configuration, and Query Result is stored as file.
Return data responds to the user from event memory.
After increasing the data source adaptation module, the treatment scheme of total system is not affected, and makes total system can not be subjected to the restriction of data storage method, freely selects suitable data source, can corresponding more flexibly various situations.
The inquiry system of a kind of big data of the embodiment of the invention comprises:
The query configuration unit is used for configuration service rule searching and the described service inquiry rule and the corresponding relation of querying condition;
The query scheme tectonic element is used for the analysis result of basis to the query requests of client, extracts the corresponding business rule searching by the querying condition in the described query requests, according to the service inquiry rule structure query scheme that extracts;
Query unit is used for according to described query scheme data query Query Result being returned to client.
Above-mentioned inquiry system; only be from of the description of functional module division of view to inquiry system, its can software or example, in hardware realize, and; during realization; each functional module can independently realize, also can integratedly realize, and needn't strict form according to the above-mentioned functions module realize; for example; system framework that can be shown in Figure 1 is realized, as long as the system that realizes comprises above-mentioned functions, all belongs within the claimed system scope of the present invention.
The present invention can improve search efficiency when big data query, significantly reduce the stand-by period of user inquiring operation; Secondly, adding data source adaptation module can be implemented the processing to any data source, can be used for the data query of arbitrary data storage mode; Add pre-inquiry mechanism in addition, can alleviate the burden of service end, improve performance.
The present invention has reduced the input of equipment, improved client response speed, strengthened the user experience of client.Data volume for operation system is the inquiry of millions, and this patent is compared characteristics such as possessing input cost is low, easy realization with the introducing search engine in the performance issue that solves traditional inquiry mode.
Above content be in conjunction with concrete embodiment to further describing that the present invention did, can not assert that concrete enforcement of the present invention is confined to these explanations.For the general technical staff of the technical field of the invention, without departing from the inventive concept of the premise, can also make some simple deduction or replace, all should be considered as belonging to protection scope of the present invention.
Claims (10)
1. the querying method of big data is characterized in that, comprising:
The corresponding relation of configuration service rule searching and described service inquiry rule and querying condition;
According to analysis result, extract the corresponding business rule searching by the querying condition in the described query requests, according to the service inquiry rule structure query scheme that extracts to the query requests of client;
According to described query scheme data query, Query Result is returned to client.
2. the method for claim 1 is characterized in that, described configuration service rule searching is the corresponding granularity of query of configuration different business attribute.
3. the method for claim 1, it is characterized in that, also comprise: storing queries history, record in historical query condition and described historical query condition corresponding historical Query Result and the historical query scheme at least one in the described query history, if the querying condition in the query requests of client is identical with the historical query condition, then directly return the corresponding historical Query Result to client according to query history, perhaps, Query Result is returned to client according to corresponding historical query scheme data query in the query history.
4. the method for claim 1 is characterized in that, also comprises: carry out adaptive to different data source types.
5. as the described method of claim 1-4, it is characterized in that, also comprise: pre-inquiry threshold value and inquiry mark are set; When this data volume that inquires reaches described pre-inquiry threshold value, these data that inquire are returned to client as this Query Result, and this Query Result is carried out mark by the inquiry mark; And when inquiring about by same query scheme, continue inquiry according to the inquiry mark next time.
6. the inquiry system of big data is characterized in that, comprising:
The query configuration unit is used for configuration service rule searching and the described service inquiry rule and the corresponding relation of querying condition;
The query scheme tectonic element is used for the analysis result of basis to the query requests of client, extracts the corresponding business rule searching by the querying condition in the described query requests, according to the service inquiry rule structure query scheme that extracts;
Query unit is used for according to described query scheme data query Query Result being returned to client.
7. system as claimed in claim 6 is characterized in that, described configuration service rule searching is the corresponding granularity of query of configuration different business attribute.
8. system as claimed in claim 6, it is characterized in that, also comprise the inquiry storage unit, be used for storing queries history, record in historical query condition and described historical query condition corresponding historical Query Result and the historical query scheme at least one in the described query history, when the querying condition of described query unit in the query requests of client is identical with the historical query condition, directly return the corresponding historical Query Result according to query history to client, perhaps, Query Result is returned to client according to corresponding historical query scheme data query in the query history.
9. system as claimed in claim 6 is characterized in that, also comprises the data source adaptation unit, is used for carrying out adaptive to different data source types.
10. as the described system of claim 6-9, it is characterized in that, comprise that also pre-inquiry is provided with the unit, be used to be provided with pre-inquiry threshold value and inquiry mark; When described query unit reaches described pre-inquiry threshold value in this data volume that inquires, these data that inquire are returned to client as this Query Result; Described pre-inquiry is provided with the unit and by the inquiry mark this Query Result is carried out mark; Described query unit continues inquiry according to the inquiry mark next time when inquiring about by same query scheme.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010101959794A CN102279849A (en) | 2010-06-09 | 2010-06-09 | Method and system for big data query |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010101959794A CN102279849A (en) | 2010-06-09 | 2010-06-09 | Method and system for big data query |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102279849A true CN102279849A (en) | 2011-12-14 |
Family
ID=45105304
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010101959794A Pending CN102279849A (en) | 2010-06-09 | 2010-06-09 | Method and system for big data query |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102279849A (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455560A (en) * | 2013-08-12 | 2013-12-18 | 曙光信息产业股份有限公司 | Data query method and system |
| CN103914562A (en) * | 2014-04-11 | 2014-07-09 | 陈桂芳 | System and method based on large data analysis for achieving IT (information technology) argument collection |
| CN104462267A (en) * | 2014-11-23 | 2015-03-25 | 国云科技股份有限公司 | Fast data query method |
| WO2015074466A1 (en) * | 2013-11-22 | 2015-05-28 | 华为技术有限公司 | Data search method and apparatus |
| CN104750806A (en) * | 2015-03-25 | 2015-07-01 | 浪潮集团有限公司 | Large data query method and system |
| WO2016101557A1 (en) * | 2014-12-23 | 2016-06-30 | 中兴通讯股份有限公司 | Method and device for providing network asset data |
| CN105786932A (en) * | 2014-12-26 | 2016-07-20 | 北大医疗信息技术有限公司 | Query method and query apparatus for clinical business in medical system |
| CN106156307A (en) * | 2016-06-30 | 2016-11-23 | 北京奇虎科技有限公司 | The data handling system of a kind of real-time calculating platform and method |
| CN106294380A (en) * | 2015-05-18 | 2017-01-04 | 中兴通讯股份有限公司 | The querying method of data base and device |
| CN107169103A (en) * | 2017-05-17 | 2017-09-15 | 南京焦点领动云计算技术有限公司 | A kind of universal architecture data store query method and system |
| CN108287833A (en) * | 2017-01-09 | 2018-07-17 | 北京艺鉴通科技有限公司 | It is a kind of for the art work identification to scheme to search drawing method |
| CN109710641A (en) * | 2018-12-17 | 2019-05-03 | 浩云科技股份有限公司 | A kind of inquiry processing method and system of mass data |
| CN110990420A (en) * | 2019-11-27 | 2020-04-10 | 腾讯科技（深圳）有限公司 | Data query method and device |
| CN111444227A (en) * | 2020-04-15 | 2020-07-24 | 中国银行股份有限公司 | Query requirement processing method and system |
| CN111897932A (en) * | 2020-07-21 | 2020-11-06 | 深圳市维度统计咨询股份有限公司 | Query processing method and system for text big data |
| CN112650915A (en) * | 2020-11-30 | 2021-04-13 | 中国科学院信息工程研究所 | Data interaction method and device based on real-time query |
| CN113051332A (en) * | 2021-04-20 | 2021-06-29 | 东莞市盟大塑化科技有限公司 | Multi-source data integration method and system based on big data technology |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101146152A (en) * | 2006-09-14 | 2008-03-19 | 中国电信股份有限公司 | Information collection and search system for telecommunication information station |
| CN101464894A (en) * | 2008-12-30 | 2009-06-24 | 北京中创信测科技股份有限公司 | Data query method and system |
-
2010
- 2010-06-09 CN CN2010101959794A patent/CN102279849A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101146152A (en) * | 2006-09-14 | 2008-03-19 | 中国电信股份有限公司 | Information collection and search system for telecommunication information station |
| CN101464894A (en) * | 2008-12-30 | 2009-06-24 | 北京中创信测科技股份有限公司 | Data query method and system |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455560A (en) * | 2013-08-12 | 2013-12-18 | 曙光信息产业股份有限公司 | Data query method and system |
| WO2015074466A1 (en) * | 2013-11-22 | 2015-05-28 | 华为技术有限公司 | Data search method and apparatus |
| CN103914562A (en) * | 2014-04-11 | 2014-07-09 | 陈桂芳 | System and method based on large data analysis for achieving IT (information technology) argument collection |
| CN104462267A (en) * | 2014-11-23 | 2015-03-25 | 国云科技股份有限公司 | Fast data query method |
| CN105786845A (en) * | 2014-12-23 | 2016-07-20 | 中兴通讯股份有限公司 | Method and device for providing network asset data |
| WO2016101557A1 (en) * | 2014-12-23 | 2016-06-30 | 中兴通讯股份有限公司 | Method and device for providing network asset data |
| CN105786845B (en) * | 2014-12-23 | 2020-03-31 | 中兴通讯股份有限公司 | Method and device for providing network asset data |
| CN105786932A (en) * | 2014-12-26 | 2016-07-20 | 北大医疗信息技术有限公司 | Query method and query apparatus for clinical business in medical system |
| CN105786932B (en) * | 2014-12-26 | 2020-03-27 | 北大医疗信息技术有限公司 | Query method and query device for clinical business in medical system |
| CN104750806A (en) * | 2015-03-25 | 2015-07-01 | 浪潮集团有限公司 | Large data query method and system |
| CN106294380A (en) * | 2015-05-18 | 2017-01-04 | 中兴通讯股份有限公司 | The querying method of data base and device |
| CN106294380B (en) * | 2015-05-18 | 2021-02-12 | 中兴通讯股份有限公司 | Database query method and device |
| CN106156307A (en) * | 2016-06-30 | 2016-11-23 | 北京奇虎科技有限公司 | The data handling system of a kind of real-time calculating platform and method |
| CN108287833A (en) * | 2017-01-09 | 2018-07-17 | 北京艺鉴通科技有限公司 | It is a kind of for the art work identification to scheme to search drawing method |
| CN107169103A (en) * | 2017-05-17 | 2017-09-15 | 南京焦点领动云计算技术有限公司 | A kind of universal architecture data store query method and system |
| CN109710641A (en) * | 2018-12-17 | 2019-05-03 | 浩云科技股份有限公司 | A kind of inquiry processing method and system of mass data |
| CN110990420A (en) * | 2019-11-27 | 2020-04-10 | 腾讯科技（深圳）有限公司 | Data query method and device |
| CN110990420B (en) * | 2019-11-27 | 2024-06-04 | 腾讯科技（深圳）有限公司 | Data query method and device |
| CN111444227A (en) * | 2020-04-15 | 2020-07-24 | 中国银行股份有限公司 | Query requirement processing method and system |
| CN111897932A (en) * | 2020-07-21 | 2020-11-06 | 深圳市维度统计咨询股份有限公司 | Query processing method and system for text big data |
| CN112650915A (en) * | 2020-11-30 | 2021-04-13 | 中国科学院信息工程研究所 | Data interaction method and device based on real-time query |
| CN112650915B (en) * | 2020-11-30 | 2023-03-10 | 中国科学院信息工程研究所 | Data interaction method and device based on real-time query |
| CN113051332A (en) * | 2021-04-20 | 2021-06-29 | 东莞市盟大塑化科技有限公司 | Multi-source data integration method and system based on big data technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102279849A (en) | Method and system for big data query | |
| US20220327125A1 (en) | Query scheduling based on a query-resource allocation and resource availability | |
| US11921672B2 (en) | Query execution at a remote heterogeneous data store of a data fabric service | |
| US11586627B2 (en) | Partitioning and reducing records at ingest of a worker node | |
| US11442935B2 (en) | Determining a record generation estimate of a processing task | |
| US11321321B2 (en) | Record expansion and reduction based on a processing task in a data intake and query system | |
| US11580107B2 (en) | Bucket data distribution for exporting data to worker nodes | |
| US11593377B2 (en) | Assigning processing tasks in a data intake and query system | |
| US7730060B2 (en) | Efficient evaluation of object finder queries | |
| US20190258635A1 (en) | Determining Records Generated by a Processing Task of a Query | |
| US8949222B2 (en) | Changing the compression level of query plans | |
| CN102193917A (en) | Method and device for processing and querying data | |
| US8843436B2 (en) | Systems and methods for performing direct reporting access to transaction databases | |
| US20150026145A1 (en) | Information retrieval system | |
| US20160140205A1 (en) | Queries involving multiple databases and execution engines | |
| US20140046928A1 (en) | Query plans with parameter markers in place of object identifiers | |
| CN104778270A (en) | Storage method for multiple files | |
| CN101937474A (en) | Mass data query method and device | |
| CN102955802B (en) | The method and apparatus of data is obtained from data sheet | |
| CN103353901A (en) | Orderly table data management method and system based on Hadoop distributed file system (HDFS) | |
| Durner et al. | Crystal: a unified cache storage system for analytical databases | |
| CN101901277A (en) | Dynamic ontology modeling method and system based on user situation | |
| CN117312370A (en) | Data query method, system and related equipment | |
| Gu et al. | MANSOR: a module alignment method based on neighbor information for scientific workflow | |
| CN115114354B (en) | Distributed data storage and query system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20111214 |