CN108090154A - A kind of isomerous multi-source data fusion querying method and device - Google Patents
A kind of isomerous multi-source data fusion querying method and device Download PDFInfo
- Publication number
- CN108090154A CN108090154A CN201711315934.4A CN201711315934A CN108090154A CN 108090154 A CN108090154 A CN 108090154A CN 201711315934 A CN201711315934 A CN 201711315934A CN 108090154 A CN108090154 A CN 108090154A
- Authority
- CN
- China
- Prior art keywords
- data source
- data
- query
- inquiry
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of isomerous multi-source data fusion querying method and devices, are related to big data field.Including：Data source query request is received, Connecting quantity corresponding with data source is confirmed in data model according to inquiry form；According to the storage mode in data storage of the type of the type of data source and data source, data query adapter corresponding with data source query request and data source are determined；According to data source query ask in data source interrelational form and screening conditions, the inquiry field in inquiry form is optimized, and the first query result is obtained out of data storage according to the inquiry field after optimization；According to data source query ask in data source associate field and data source relevant parameter, fusion is associated to the first query result, obtains the second query result.
Description
Technical field
The present invention relates to big data fields, more particularly relate to a kind of isomerous multi-source data fusion querying method and device.
Background technology
With the development of Internet of Things and big data, data scale rapid growth, data structure is varied, and data are carrying
For more and more important value, the fusions of isomerous multi-source data and analysis are also more and more paid attention to, relevant method and
Technology is continuous fast-developing.
There is currently a variety of isomerous multi-source data fusion analytical technologies and method, for example, the Pentaho to increase income, various business
IT business system of industry BI softwares and enterprise self-determining customized development etc..The prior art still has for the visual analyzing of big data
Larger limitation, problem are concentrated mainly on several aspects：1), existing BI softwares are not too complete to the support of unstructured data sources
Face, such as major part BI softwares do not support elasticsearch；2), existing BI softwares are in integrated isomerous multi-source data, only
It solves the problems, such as the multisource data fusion of structural data, fails to solve structuring and the multi-source fusion of unstructured data is asked
Topic；3), the IT business system of enterprise self-determining customized development will realize adaptation multiple data sources, and also to accomplish data correlation,
Lack unified, standardization mentality of designing, lead to problems such as overlapping development, development cost high.
The content of the invention
The embodiment of the present invention provides a kind of isomerous multi-source data fusion querying method and device, to solve structuring with it is non-
The multi-source inquiry adaptation of structural data and fusion problem.
The embodiment of the present invention provides a kind of isomerous multi-source data fusion querying method, including：
Data source query request is received, inquiry form is carried in the data source query request, according to the inquiry
Form confirm in data model with the corresponding Connecting quantity of data source query request, according to the Connecting quantity determine with
The data source query asks corresponding data source types；
According to the data source types and the storage mode in data storage of the data source types, determine with
The data source query asks corresponding data query adapter and data source；
According to the data source interrelational form and screening conditions in data source query request, in the inquiry form
Inquiry field optimize, and according to the inquiry field after optimization, by inquiring about the data source, obtain and the number
According to corresponding first query result of source inquiry request；
According to the data source associate field and data source relevant parameter in data source query request, look into described first
It askes result and is associated fusion, obtain the second query result.
Preferably, the data storage includes structural data and unstructured data；
First query result includes at least two corresponding with the type of the data source first as a result, and described the
SQL generator and at least one category of one result respectively comprising at least one dynamic splicing for belonging to and supporting structuring query SQL
In the NoSQL query builders for supporting application programming interface API dynamic calls；
After definite first query result corresponding with data source query request, further include：Support will be belonged to
First result of the SQL generator of the dynamic splicing of structuralized query SQL is stored to the structural data；Branch will be belonged to
First result for holding the NoSQL query builders of application programming interface API dynamic calls is formatted conversion simultaneously
It stores to the structural data.
Preferably, it is described obtain the second query result after, further include：
Second query result is packaged into lattice corresponding with carrying inquiry form in data source query request
Formula encapsulated result, and the form encapsulated result is sent to inquiry request interface.
The embodiment of the present invention also provides a kind of isomerous multi-source data fusion inquiry unit, which is characterized in that including：
For receiving data source query request, inquiry is carried in the data source query request for first determination unit
Form, according to it is described inquiry form confirm in data model with the corresponding Connecting quantity of data source query request, according to
The Connecting quantity determines and the corresponding data source types of data source query request；
Second determination unit, for according to the data source types and the data source types in data storage
Storage mode, determine data query adapter corresponding with data source query request and data source；
First obtains unit, for according to the data source interrelational form in data source query request and screening item
Part optimizes the inquiry field in the inquiry form, and according to the inquiry field after optimization, by described in inquiry
Data source obtains the first query result corresponding with data source query request；
Second obtains unit, for being associated according to the data source associate field in data source query request with data source
Parameter is associated fusion to first query result, obtains the second query result.
Preferably, the data storage includes structural data and unstructured data；
First query result includes at least two corresponding with the type of the data source first as a result, and described the
SQL generator and at least one category of one result respectively comprising at least one dynamic splicing for belonging to and supporting structuring query SQL
In the NoSQL query builders for supporting application programming interface API dynamic calls；
Described first, which obtains unit, is additionally operable to：The SQL generator for the dynamic splicing for supporting structuring query SQL will be belonged to
First result is stored to the structural data；The NoSQL for supporting application programming interface API dynamic calls will be belonged to
First result of query builder, which is formatted, to be converted and stores to the structural data.
Preferably, described second obtain unit and be additionally operable to：
Second query result is packaged into lattice corresponding with carrying inquiry form in data source query request
Formula encapsulated result, and the form encapsulated result is sent to inquiry request interface.
An embodiment of the present invention provides a kind of isomerous multi-source data fusion querying methods, which is characterized in that including：It receives
Data source query is asked, and inquiry form is carried in the data source query request, according to the inquiry form in data model
Interior confirmation and the corresponding Connecting quantity of data source query request, determine and the data source query according to the Connecting quantity
Ask corresponding data source types；According to the depositing in data storage of the data source types and the data source types
Storage mode determines data query adapter corresponding with data source query request and data source；It is looked into according to the data source
The data source interrelational form and screening conditions in request are ask, the inquiry field in the inquiry form is optimized, and root
According to the inquiry field after optimization, by inquiring about the data source, obtain and data source query request corresponding first
Query result；According to the data source associate field and data source relevant parameter in data source query request, to described first
Query result is associated fusion, obtains the second query result.The above method is adapted to isomerous multi-source data query and data are closed
Connection, can inquire about the Various types of data that each IT system of enterprises is stored, data are merged comprehensively in a system,
The Biodiversity Characteristics of big data are embodied, providing data for the various big data applications such as data analysis, data mining supports, excavates
Data value.So as to solve the problems, such as the multi-source inquiry adaptation of structuring and unstructured data and merge.
Description of the drawings
It in order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention, for those of ordinary skill in the art, without creative efforts, can be with
Other attached drawings are obtained according to these attached drawings.
Fig. 1 is a kind of isomerous multi-source data fusion querying method flow diagram provided in an embodiment of the present invention；
Fig. 2 is isomerous multi-source data fusion configuration diagram provided in an embodiment of the present invention；
Fig. 3 is a kind of isomerous multi-source data fusion inquiry unit structure diagram provided in an embodiment of the present invention.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other without making creative work
Embodiment belongs to the scope of protection of the invention.
Fig. 1 illustratively shows that a kind of isomerous multi-source data fusion querying method flow provided in an embodiment of the present invention is shown
It is intended to.As shown in Figure 1, this method mainly includes the following steps that：
Step 101, data source query request is received, inquiry form is carried in the data source query request, according to
It is described inquiry form confirms in data model with the corresponding Connecting quantity of data source query request, according to it is described connect ginseng
Number determines and the corresponding data source types of data source query request；
Step 102, according to the storage side in data storage of the data source types and the data source types
Formula determines data query adapter corresponding with data source query request and data source；
Step 103, according to the data source interrelational form and screening conditions in data source query request, look into described
The inquiry field ask in form optimizes, and according to the inquiry field after optimization, by inquiring about the data source, obtains
The first query result corresponding with data source query request；
Step 104, according to the data source associate field and data source relevant parameter in data source query request, to institute
It states the first query result and is associated fusion, obtain the second query result.
Fig. 2 is isomerous multi-source data fusion configuration diagram provided in an embodiment of the present invention, is provided below in conjunction with Fig. 2
Multiple modules describe to a kind of isomerous multi-source data fusion querying method provided in an embodiment of the present invention in detail.
As shown in Fig. 2, the framework mainly include data store, data model, inquiry request interface, data loading adaptation and
This 5 modules of data operation.
Specifically, by inquiry request interface external module or system can be given to provide isomerous multi-source data query and take
Business.
In embodiments of the present invention, inquiry request interface defines the specification of data source query request, such as interface communication
Agreement, inquires about form and data return to form.
For example, when the inquiry form of data source query request is json, which mainly includes data source name, inquiry
The information such as field, screening conditions, data source associate field, relevant parameter.
It is to inquire about the data source query request that form is json below：
In a step 101, when the data source query of inquiry request interface to data source is asked, and the data source query
Inquiry form is carried in request, for example, the inquiry form is json, you can determined according to the inquiry form in the inquiry form
Including multiple data item, for example, data source name, the field of inquiry, screening conditions, data source associate field and data source are closed
Join the information such as parameter.
After the multiple data item included in inquiry form are confirmed, by the data source name of confirmation, data mould is inquired about
Type can determine Connecting quantity corresponding with the data source name according to data model, further, be determined according to Connecting quantity
The type of data source.
It, can also will be from according to the multiple parameters that include of inquiry form after Connecting quantity corresponding with data source is determined
The parameter got in data model is added in inquiry form.
For example, connecting example, the parameter got is added in inquiry form：
The information such as parameter obtain new json：
Specifically, data model mainly includes following functions：
1. define the Connecting quantity of each data source.
2. for supporting the data of SQL computings, it is necessary to define storehouse, table name and meaning, field name and meaning, field class
Type, field display order etc..
3. for NoSQL storehouses, it is necessary to define data model according to its characteristic.Such as some need to define containing for key-value pair
Data format (such as json structures) built in justice, key assignments；Some need the meaning of index of definition, each field contains in index
Whether justice, field type allow participle etc..
In a step 102, according to the type of definite data source, the storage side of the type of data source in data storage
Formula determines data query adapter corresponding with data source query request, and data model is inquired about by the data query adapter,
It can then determine and the corresponding data source of data source query request.
Example is connected, can be obtained and the corresponding data source of data source query request：
1. data source one：Type：MySQL；Database：Service source；Table：employee.
2. data source two：Type：MongoDB；Database：sns
Specifically, data storage is mainly used for structured data and unstructured data.Wherein, structural data,
The general relevant database being stored in such as MySQL, Oracle and SQL Server etc., such as the distributed file system of HDFS,
These above-mentioned systems provide structured query language, i.e., by hive, the data of hdfs can also be made to carry out SQL query；Non- knot
Structure data, the system for being generally stored in such as Elasticsearch, HBase, Redis, MongoDB non-relational, these are
System has used storages and the inquiry modes such as column storage, document-type storage, it is impossible to increased using the query language of standard, deleted,
Change, look into operation, this part is known as NoSQL data sources, and different NoSQL storehouses provide different API data interfaces, it is necessary to make
With when, be adapted to.
It should be noted that SQL has the grammer of standard, so the querying method of above-mentioned various data-storage systems, substantially
It is similar, but the SQL syntax of peer machine slightly has difference, it is necessary to be adapted to when calling with standard.
In embodiments of the present invention, there are two being included due to data source, and corresponding query result in each data source
Quantity also compares more.In step 103, according in step 101 data source query request in data source interrelational form and
Screening conditions optimize the inquiry field in inquiry form, so as to the inquiry field according to optimization, by inquiring about data source
Obtain the first query result corresponding with data source query request.
It should be noted that in the embodiment of the present invention, query result can be optimized, in order to distinguish explanation, here will
Query result is temporarily defined as the first query result.
For example, connect above-mentioned example：
The interrelational form in json is asked according to data source query (left connection, name field values are equal to author field values)
With querying condition (querying condition of left-handed watch is name Zhang San), query argument can be optimized:
To the querying condition of right table mongodb, increase the inquiry of author, to reduce the return of unnecessary query result
And loading.
1.MySQL (dynamic splicing SQL)：
SELECT name, job FROM business.employee WHERE name=' Zhang San '；
Obtain the first query result：
2.MongoDB：(dynamic call API)：
db.sns.find({author:" Zhang San " }, { content:{$regex:" weather " } }, ' author':1,"
timestamp":1,'title':1,'content':1,'picture':1})
Obtain the first query result：
It should be noted that in practical applications, for supporting the data of SQL computings, it is preferred that emphasis is splicing SQL.
For example, the detailed inquiry of data：SELECT【Grouping field】,【Aminated polyepichlorohydrin field】FROM【Storehouse tables】WHERE【It looks into
Inquiry condition】ORDER BY【Sort field】
For example, aggregate statistics are inquired about：SELECT【Result field】FROM【Storehouse tables】WHERE【Querying condition】GROUP BY
【Grouping field】ORDER BY【Sort field】
Limitation for query result quantity, disparate databases have different grammers.Such as mysql uses limit；sql
Server uses top；Oracle uses ROWNUM.
For NoSQL storehouses, it is necessary to call suitable API.API points are field selection, polymerization, conditional filtering, sequence, result
Several major classes such as quantity limitation.Such as：
For example, the detailed inquiry of the data of MongoDB：db.sns.find({author:{$regex:" Zhang San " },
content:{$regex:" weather " } }, ' title':1,'content':1,'picture':1,'author':0})
For example, the aggregate statistics inquiry of MongoDB：db.mycol.aggregate([{$group:{_id:"$by_
user",num_tutorial:{$sum:1}}}])
It should be noted that in embodiments of the present invention, since data source includes at least two or more, correspondingly,
Two or more first can at least be included in first query result as a result, i.e. the first result can be divided into two classes：Branch
It holds the first result of the SQL generator of the dynamic splicing of structuralized query SQL and application programming interface API dynamics is supported to adjust
First result of NoSQL query builders.In embodiments of the present invention, obtained the first query result storage will can be inquired about
Into the memory of the system, that is, store to data storage, and when storing the first query result to memory, it is necessary to two classes
First result distinguishes, specifically：The first result for belonging to SQL generator is stored to the structural data, and will be belonged to
In the first result of NoSQL query builders need carry out format conversion, and by the first result after format conversion also store to
Structural data.
Further, the first query result can support inquiry or the support two of two or more NOSQL
Or the inquiry of multiple SQL.And the inquiry the inside of NOSQL is supported without the inquiry of SQL, can to support that the inquiry the inside of SQL can
With the inquiry of no NOSQL.
At step 104, according to data source query ask in data source associate field and data source relevant parameter, to
One query result is associated fusion, obtains the second query result.
As shown in Fig. 2, data operation includes isomerous multi-source data correlation and data format.
Specifically, isomerous multi-source data correlation：
It is loaded and is adapted to by previous step data, not homologous data, the memory of the system is loaded into, using suitable
Data structure stored, such as ArrayList.Next, it is exactly each association in being asked according to data source query
Condition travels through relevant 2 parts of data.
Support multiple Correlation Criterias, each condition is only associated 2 parts of data.
If there is 3 parts of inquiry data, such as A, B, C, it is necessary to associate, then Correlation Criteria at least defines 2：One one or two
A is associated with B, is shown that D, the 2nd one or two D are associated with C, is drawn final result E.
Inside single Correlation Criteria, the multiple fields that can define 2 parts of data are associated, by with or relation,
It is calculated.
Such as data A can be with the Correlation Criteria of data B：The a1 fields of data A are equal to the b1 fields and number of data B
The b2 fields of data B are included according to the a2 fields of A
Interrelational form according to the difference of field type, supports different matching ways：
Character string type：Be equal to, include, not including, starting be, ending be
Numeric type：It is equal to, be more than, be less than, be more than or equal to, is less than or equal to
Time type：It is equal to, be more than, be less than, be more than or equal to, is less than or equal to
2. data format
Field information in data model, format data are shown.
Date and time：
The java DateFormat API provided and common time format (such as yyyy-MM-dd HH are provided:mm:ss、
Yyyy/MM/dd, yyyy MM months dd day HH:mm:Ss etc.), first judge the specific time format of field, reuse
DateFormat API are converted to object format；
Numerical value：
Field type is obtained according to data model：Integer or floating type：
Integer：The display mode that purpose is mainly beautification wholesale numerical value as requested is formatted, such as converts " 3000000 "
For " 3,000,000 ", " 3597009 " are converted to " 3,600,000 " etc..It is required therefore, it is necessary to elder generation according to object format, current value is turned
Decimal is changed to, after rounding up and retaining integer, splices target unit.Exemplified by " 3,500,000 " being converted to by " 3597009 "：First will
" 3597009 " are converted to " 359.7009 ", and rounding up and retaining integer obtains " 360 ", and splice template unit " ten thousand " is tied
Fruit " 3,600,000 "；
Floating type：Real-coded GA formats the mainly reservation of decimal place and is converted to percentage.It can directly invoke
Java DecimalFormat API are realized.
Continuous upper example.Since left-handed watch field name and right literary name section author are character strings, and matching operation symbol is eq
(being equal to), so summarizing arrangement obtains result：
Based on same inventive concept, an embodiment of the present invention provides a kind of isomerous multi-source data fusion inquiry unit, due to
The principle that the device solves technical problem is similar to a kind of isomerous multi-source data fusion querying method, therefore the implementation of the device can
With referring to the implementation of method, overlaps will not be repeated.
As shown in figure 3, a kind of isomerous multi-source data fusion inquiry unit structure diagram provided in an embodiment of the present invention, such as
Shown in Fig. 3, which includes the first determination unit 301, and the second determination unit 302, first, which obtains unit 303 and second, obtains list
Member 304.
First determination unit 301 for receiving data source query request, is carried in the data source query request and looked into
Form is ask, is confirmed according to the inquiry form in data model and the corresponding Connecting quantity of data source query request, root
It is determined and the corresponding data source types of data source query request according to the Connecting quantity；
Second determination unit 302, for being stored according to the data source types and the data source types in data
Storage mode in device determines data query adapter corresponding with data source query request and data source；
First obtains unit 303, for according to the data source interrelational form in data source query request and screening
Condition optimizes the inquiry field in the inquiry form, and according to the inquiry field after optimization, by inquiring about
Data source is stated, obtains the first query result corresponding with data source query request；
Second obtains unit 304, for according to the data source associate field and data source in data source query request
Relevant parameter is associated fusion to first query result, obtains the second query result.
Preferably, the data storage includes structural data and unstructured data；
First query result includes at least two corresponding with the type of the data source first as a result, and described the
SQL generator and at least one category of one result respectively comprising at least one dynamic splicing for belonging to and supporting structuring query SQL
In the NoSQL query builders for supporting application programming interface API dynamic calls；
Described first, which obtains unit 303, is additionally operable to：The SQL for belonging to the dynamic splicing for supporting structuring query SQL is generated
First result of device is stored to the structural data；Application programming interface API dynamic calls are supported by belonging to
First result of NoSQL query builders, which is formatted, to be converted and stores to the structural data.
Preferably, described second obtain unit 304 and be additionally operable to：
Second query result is packaged into lattice corresponding with carrying inquiry form in data source query request
Formula encapsulated result, and the form encapsulated result is sent to inquiry request interface.
It should be appreciated that the unit that includes of one of the above isomerous multi-source data fusion inquiry unit only according to the apparatus
The logical partitioning that the function of realization carries out in practical application, can carry out the superposition or fractionation of said units.And the embodiment
A kind of isomerous multi-source that the function that a kind of isomerous multi-source data fusion inquiry unit provided is realized is provided with above-described embodiment
Data fusion querying method corresponds, and for the more detailed process flow that the device is realized, implements in the above method
It has been described in detail in example one, has been not described in detail herein.
It should be understood by those skilled in the art that, the embodiment of the present invention can be provided as method, system or computer program
Product.Therefore, the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware can be used in the present invention
Apply the form of example.Moreover, the computer for wherein including computer usable program code in one or more can be used in the present invention
The computer program production that usable storage medium is implemented on (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
The present invention be with reference to according to the method for the embodiment of the present invention, the flow of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that it can be realized by computer program instructions every first-class in flowchart and/or the block diagram
The combination of flow and/or box in journey and/or box and flowchart and/or the block diagram.These computer programs can be provided
The processor of all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that the instruction performed by computer or the processor of other programmable data processing devices is generated for real
The device for the function of being specified in present one flow of flow chart or one box of multiple flows and/or block diagram or multiple boxes.
These computer program instructions, which may also be stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction generation being stored in the computer-readable memory includes referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one box of block diagram or
The function of being specified in multiple boxes.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted
Series of operation steps is performed on calculation machine or other programmable devices to generate computer implemented processing, so as in computer or
The instruction offer performed on other programmable devices is used to implement in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Although preferred embodiments of the present invention have been described, but those skilled in the art once know basic creation
Property concept, then can make these embodiments other change and modification.So appended claims be intended to be construed to include it is excellent
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
God and scope.In this way, if these modifications and changes of the present invention belongs to the scope of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to comprising including these modification and variations.
Claims (6)
1. a kind of isomerous multi-source data fusion querying method, which is characterized in that including：
Data source query request is received, inquiry form is carried in the data source query request, according to the inquiry form
In data model confirm and the corresponding Connecting quantity of data source query request, according to the Connecting quantity determine with it is described
Data source query asks corresponding data source types；
According to the data source types and the storage mode in data storage of the data source types, determine with it is described
Data source query asks corresponding data query adapter and data source；
According to the data source interrelational form and screening conditions in data source query request, to looking into the inquiry form
It askes field to optimize, and according to the inquiry field after optimization, by inquiring about the data source, obtains and the data source
Corresponding first query result of inquiry request；
According to the data source associate field and data source relevant parameter in data source query request, the described first inquiry is tied
Fruit is associated fusion, obtains the second query result.
2. the method as described in claim 1, which is characterized in that the data storage includes structural data and unstructured
Data；
First query result includes at least two corresponding with the type of the data source first as a result, and first knot
SQL generator of the fruit respectively comprising at least one dynamic splicing for belonging to and supporting structuring query SQL and at least one belong to branch
Hold the NoSQL query builders of application programming interface API dynamic calls；
It is described obtain the first query result corresponding with data source query request after, further include：Support structure will be belonged to
First result for changing the SQL generator of the dynamic splicing of query SQL is stored to the structural data；Support will be belonged to should
Conversion is formatted with first result of the NoSQL query builders of Program Interfaces API dynamic calls and is stored
To the structural data.
3. the method as described in claim 1, which is characterized in that it is described obtain the second query result after, further include：
Second query result is packaged into form corresponding with carrying inquiry form in data source query request to seal
Dress by the form encapsulated result as a result, and be sent to inquiry request interface.
4. a kind of isomerous multi-source data fusion inquiry unit, which is characterized in that including：
For receiving data source query request, inquiry form is carried in the data source query request for first determination unit,
According to it is described inquiry form confirm in data model with the corresponding Connecting quantity of data source query request, according to the company
Parameter is connect to determine and the corresponding data source types of data source query request；
Second determination unit, for the depositing in data storage according to the data source types and the data source types
Storage mode determines data query adapter corresponding with data source query request and data source；
First obtains unit, for according to the data source query request in data source interrelational form and screening conditions, it is right
Inquiry field in the inquiry form optimizes, and according to the inquiry field after optimization, by inquiring about the data
Source obtains the first query result corresponding with data source query request；
Second obtains unit, for associating ginseng with data source according to the data source associate field in data source query request
Number, is associated fusion to first query result, obtains the second query result.
5. device as claimed in claim 4, which is characterized in that the data storage includes structural data and unstructured
Data；
First query result includes at least two corresponding with the type of the data source first as a result, and first knot
SQL generator of the fruit respectively comprising at least one dynamic splicing for belonging to and supporting structuring query SQL and at least one belong to branch
Hold the NoSQL query builders of application programming interface API dynamic calls；
Described first, which obtains unit, is additionally operable to：It will belong to described in the SQL generator for the dynamic splicing for supporting structuring query SQL
First result is stored to the structural data；The NoSQL inquiries for supporting application programming interface API dynamic calls will be belonged to
First result of constructor, which is formatted, to be converted and stores to the structural data.
6. device as claimed in claim 4, which is characterized in that described second, which obtains unit, is additionally operable to：
Second query result is packaged into form corresponding with carrying inquiry form in data source query request to seal
Dress by the form encapsulated result as a result, and be sent to inquiry request interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711315934.4A CN108090154A (en) | 2017-12-08 | 2017-12-08 | A kind of isomerous multi-source data fusion querying method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711315934.4A CN108090154A (en) | 2017-12-08 | 2017-12-08 | A kind of isomerous multi-source data fusion querying method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108090154A true CN108090154A (en) | 2018-05-29 |
Family
ID=62175042
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711315934.4A Pending CN108090154A (en) | 2017-12-08 | 2017-12-08 | A kind of isomerous multi-source data fusion querying method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108090154A (en) |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108932345A (en) * | 2018-07-27 | 2018-12-04 | 北京中关村科金技术有限公司 | One kind realizing across data source distributed Query Processing System and method based on dremio |
| CN109408578A (en) * | 2018-10-30 | 2019-03-01 | 环境保护部华南环境科学研究所 | One kind being directed to isomerous environment monitoring data fusion method |
| CN109445754A (en) * | 2018-09-14 | 2019-03-08 | 中国建设银行股份有限公司 | Data integration access mechanism, method and storage medium |
| CN109558444A (en) * | 2018-11-29 | 2019-04-02 | 苏州思必驰信息科技有限公司 | Data retrieval method and device |
| CN109683952A (en) * | 2018-12-25 | 2019-04-26 | 携程计算机技术（上海）有限公司 | The determination method and system of data set assembling frame |
| CN109710630A (en) * | 2018-12-25 | 2019-05-03 | 中科曙光国际信息产业有限公司 | The querying method and device of heterogeneous data source |
| CN110109766A (en) * | 2019-05-23 | 2019-08-09 | 贵阳块数据城市建设有限公司 | Data request method, device, system, data forwarding device and storage medium |
| CN110232178A (en) * | 2019-05-29 | 2019-09-13 | 北京中亦安图科技股份有限公司 | Report generation method and device |
| CN110781213A (en) * | 2019-09-25 | 2020-02-11 | 中国电子进出口有限公司 | Multi-source mass data correlation searching method and system with personnel as center |
| CN110837585A (en) * | 2019-11-07 | 2020-02-25 | 中盈优创资讯科技有限公司 | Multi-source heterogeneous data association query method and system |
| CN110895544A (en) * | 2018-08-24 | 2020-03-20 | 北京国双科技有限公司 | Interface data processing method, device, system and storage medium |
| CN110909026A (en) * | 2019-10-15 | 2020-03-24 | 北京航空航天大学 | Distributed multi-source robot data management system and method |
| CN111046245A (en) * | 2019-12-11 | 2020-04-21 | 杭州趣链科技有限公司 | Multi-source heterogeneous data source fusion calculation method, system, equipment and storage medium |
| CN111078961A (en) * | 2019-12-24 | 2020-04-28 | 用友网络科技股份有限公司 | Multi-data source query driving system, method, device and storage medium |
| CN111104426A (en) * | 2019-11-22 | 2020-05-05 | 深圳智链物联科技有限公司 | Data query method and system |
| CN111104421A (en) * | 2019-12-31 | 2020-05-05 | 新奥数能科技有限公司 | Data query method and device based on data interface standard configuration |
| CN111339171A (en) * | 2020-02-07 | 2020-06-26 | 拉扎斯网络科技（上海）有限公司 | Data query method, device and equipment |
| CN111475534A (en) * | 2020-05-12 | 2020-07-31 | 北京爱笔科技有限公司 | Data query method and related equipment |
| CN112597185A (en) * | 2020-12-26 | 2021-04-02 | 中国农业银行股份有限公司 | Big data query method and device |
| CN112612857A (en) * | 2020-12-07 | 2021-04-06 | 国网北京市电力公司 | Data processing method and device, computer readable storage medium and processor |
| CN112667662A (en) * | 2020-12-25 | 2021-04-16 | 银盛支付服务股份有限公司 | Common method for analyzing sql and nosql based on json key values |
| CN112749222A (en) * | 2021-01-18 | 2021-05-04 | 北京睿企信息科技有限公司 | Method and device for querying relational database supporting dynamic parameters |
| CN112765183A (en) * | 2021-02-02 | 2021-05-07 | 浙江公共安全技术研究院有限公司 | Multi-source data fusion method and device, storage medium and electronic equipment |
| CN115858504A (en) * | 2023-02-28 | 2023-03-28 | 宏景科技股份有限公司 | Multidimensional data fusion management system and method for Internet of things platform and storage medium |
| CN115905315A (en) * | 2022-11-08 | 2023-04-04 | 北京德塔精要信息技术有限公司 | Multi-type data fusion processing method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100198868A1 (en) * | 2000-06-28 | 2010-08-05 | Microsoft Corporation | Method and apparatus for information transformation and exchange in a relational database environment |
| CN104008135A (en) * | 2014-05-07 | 2014-08-27 | 南京邮电大学 | Multi-source heterogeneous database fusion system and data query method thereof |
| CN106066871A (en) * | 2016-05-30 | 2016-11-02 | 中国人民解放军装备学院 | A kind of multiple and distributing sources access method based on open interface |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
-
2017
- 2017-12-08 CN CN201711315934.4A patent/CN108090154A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100198868A1 (en) * | 2000-06-28 | 2010-08-05 | Microsoft Corporation | Method and apparatus for information transformation and exchange in a relational database environment |
| CN104008135A (en) * | 2014-05-07 | 2014-08-27 | 南京邮电大学 | Multi-source heterogeneous database fusion system and data query method thereof |
| CN106066871A (en) * | 2016-05-30 | 2016-11-02 | 中国人民解放军装备学院 | A kind of multiple and distributing sources access method based on open interface |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
Cited By (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108932345A (en) * | 2018-07-27 | 2018-12-04 | 北京中关村科金技术有限公司 | One kind realizing across data source distributed Query Processing System and method based on dremio |
| CN110895544B (en) * | 2018-08-24 | 2023-02-21 | 北京国双科技有限公司 | Interface data processing method, device, system and storage medium |
| CN110895544A (en) * | 2018-08-24 | 2020-03-20 | 北京国双科技有限公司 | Interface data processing method, device, system and storage medium |
| CN109445754A (en) * | 2018-09-14 | 2019-03-08 | 中国建设银行股份有限公司 | Data integration access mechanism, method and storage medium |
| CN109408578B (en) * | 2018-10-30 | 2020-07-31 | 环境保护部华南环境科学研究所 | Monitoring data fusion method for heterogeneous environment |
| CN109408578A (en) * | 2018-10-30 | 2019-03-01 | 环境保护部华南环境科学研究所 | One kind being directed to isomerous environment monitoring data fusion method |
| CN109558444A (en) * | 2018-11-29 | 2019-04-02 | 苏州思必驰信息科技有限公司 | Data retrieval method and device |
| CN109558444B (en) * | 2018-11-29 | 2020-08-25 | 苏州思必驰信息科技有限公司 | Data retrieval method and device |
| CN109683952B (en) * | 2018-12-25 | 2023-03-21 | 携程计算机技术（上海）有限公司 | Method and system for determining data set assembly framework |
| CN109710630A (en) * | 2018-12-25 | 2019-05-03 | 中科曙光国际信息产业有限公司 | The querying method and device of heterogeneous data source |
| CN109683952A (en) * | 2018-12-25 | 2019-04-26 | 携程计算机技术（上海）有限公司 | The determination method and system of data set assembling frame |
| CN110109766A (en) * | 2019-05-23 | 2019-08-09 | 贵阳块数据城市建设有限公司 | Data request method, device, system, data forwarding device and storage medium |
| CN110109766B (en) * | 2019-05-23 | 2022-02-18 | 贵阳块数据城市建设有限公司 | Data interaction method and device based on cross-department and cross-platform data sharing exchange |
| CN110232178A (en) * | 2019-05-29 | 2019-09-13 | 北京中亦安图科技股份有限公司 | Report generation method and device |
| CN110781213A (en) * | 2019-09-25 | 2020-02-11 | 中国电子进出口有限公司 | Multi-source mass data correlation searching method and system with personnel as center |
| CN110909026B (en) * | 2019-10-15 | 2022-07-29 | 北京航空航天大学 | Distributed multi-source robot data management system and method |
| CN110909026A (en) * | 2019-10-15 | 2020-03-24 | 北京航空航天大学 | Distributed multi-source robot data management system and method |
| CN110837585A (en) * | 2019-11-07 | 2020-02-25 | 中盈优创资讯科技有限公司 | Multi-source heterogeneous data association query method and system |
| CN110837585B (en) * | 2019-11-07 | 2022-12-20 | 中盈优创资讯科技有限公司 | Multi-source heterogeneous data association query method and system |
| CN111104426B8 (en) * | 2019-11-22 | 2024-04-23 | 北京傲速科技有限公司 | Data query method and system |
| CN111104426B (en) * | 2019-11-22 | 2024-04-05 | 深圳智链物联科技有限公司 | Data query method and system |
| CN111104426A (en) * | 2019-11-22 | 2020-05-05 | 深圳智链物联科技有限公司 | Data query method and system |
| CN111046245A (en) * | 2019-12-11 | 2020-04-21 | 杭州趣链科技有限公司 | Multi-source heterogeneous data source fusion calculation method, system, equipment and storage medium |
| CN111078961A (en) * | 2019-12-24 | 2020-04-28 | 用友网络科技股份有限公司 | Multi-data source query driving system, method, device and storage medium |
| CN111078961B (en) * | 2019-12-24 | 2023-09-15 | 用友网络科技股份有限公司 | Multi-data source query driving system, method, device and storage medium |
| CN111104421A (en) * | 2019-12-31 | 2020-05-05 | 新奥数能科技有限公司 | Data query method and device based on data interface standard configuration |
| CN111339171A (en) * | 2020-02-07 | 2020-06-26 | 拉扎斯网络科技（上海）有限公司 | Data query method, device and equipment |
| CN111475534B (en) * | 2020-05-12 | 2023-04-14 | 北京爱笔科技有限公司 | Data query method and related equipment |
| CN111475534A (en) * | 2020-05-12 | 2020-07-31 | 北京爱笔科技有限公司 | Data query method and related equipment |
| CN112612857A (en) * | 2020-12-07 | 2021-04-06 | 国网北京市电力公司 | Data processing method and device, computer readable storage medium and processor |
| CN112667662B (en) * | 2020-12-25 | 2022-11-08 | 银盛支付服务股份有限公司 | Common method for analyzing sql and nosql based on json key values |
| CN112667662A (en) * | 2020-12-25 | 2021-04-16 | 银盛支付服务股份有限公司 | Common method for analyzing sql and nosql based on json key values |
| CN112597185A (en) * | 2020-12-26 | 2021-04-02 | 中国农业银行股份有限公司 | Big data query method and device |
| CN112749222A (en) * | 2021-01-18 | 2021-05-04 | 北京睿企信息科技有限公司 | Method and device for querying relational database supporting dynamic parameters |
| CN112749222B (en) * | 2021-01-18 | 2024-03-19 | 北京睿企信息科技有限公司 | Relational database query method and device supporting dynamic parameters |
| CN112765183A (en) * | 2021-02-02 | 2021-05-07 | 浙江公共安全技术研究院有限公司 | Multi-source data fusion method and device, storage medium and electronic equipment |
| CN112765183B (en) * | 2021-02-02 | 2022-02-11 | 浙江公共安全技术研究院有限公司 | Multi-source data fusion method and device, storage medium and electronic equipment |
| CN115905315A (en) * | 2022-11-08 | 2023-04-04 | 北京德塔精要信息技术有限公司 | Multi-type data fusion processing method and system |
| CN115858504A (en) * | 2023-02-28 | 2023-03-28 | 宏景科技股份有限公司 | Multidimensional data fusion management system and method for Internet of things platform and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108090154A (en) | A kind of isomerous multi-source data fusion querying method and device | |
| US11216302B2 (en) | Modifying task dependencies at worker nodes using precompiled libraries | |
| US20220114172A1 (en) | Query decomposition for scalability of continuous query processing | |
| CN104767813B (en) | Public's row big data service platform based on openstack | |
| US10296630B2 (en) | Graph representation of data extraction for use with a data repository | |
| US7814045B2 (en) | Semantical partitioning of data | |
| CN102819557B (en) | Data query treating apparatus and data query disposal route | |
| EP3832481A1 (en) | Data permissioning through data replication | |
| US20140136472A1 (en) | Methodology supported business intelligence (BI) software and system | |
| CN110019314B (en) | Dynamic data packaging method based on data item analysis, client and server | |
| CN109101575A (en) | Calculation method and device | |
| CN113741883B (en) | RPA lightweight data middling station system | |
| US11893026B2 (en) | Advanced multiprovider optimization | |
| CN104199955B (en) | information management system and method | |
| CN105975489A (en) | Metadata-based online SQL code completion method | |
| CN108536718A (en) | A kind of method and system for the IT application in management realized based on input and output semantization | |
| US20230169116A1 (en) | Ontology-based time series visualization and analysis | |
| CN111290813B (en) | Software interface field data standardization method, device, equipment and medium | |
| CN114116678A (en) | Data migration method, device, equipment, medium and program product | |
| US20230099501A1 (en) | Masking shard operations in distributed database systems | |
| US10235100B2 (en) | Optimizing column based database table compression | |
| US20160371330A1 (en) | Database Calculation Engine Integrating Hierarchy Views | |
| US11442907B2 (en) | Automatic conversion of data models using data model annotations | |
| CN103970900B (en) | Multi-dimensional cross data flexible management method and system based on industrial field | |
| KR20180077830A (en) | Processing method for a relational query in distributed stream processing engine based on shared-nothing architecture, recording medium and device for performing the method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180529 |
|
| RJ01 | Rejection of invention patent application after publication |