CN104750681A - Method and device for processing mass data - Google Patents
Method and device for processing mass data Download PDFInfo
- Publication number
- CN104750681A CN104750681A CN201310726882.5A CN201310726882A CN104750681A CN 104750681 A CN104750681 A CN 104750681A CN 201310726882 A CN201310726882 A CN 201310726882A CN 104750681 A CN104750681 A CN 104750681A
- Authority
- CN
- China
- Prior art keywords
- data
- technical element
- element data
- index
- querying condition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a method and a device for processing mass data, relates to the field of data service, and solves the technical problems of low efficiency when the mass data is queried in the prior art. The method comprises the following steps: receiving a query request described by service metadata, wherein the query request comprises a query condition; according to a mapping relationship between the service metadata and technical metadata, looking up the technical metadata corresponding to a query condition; according to the technical metadata, looking up an index range which conforms to the query condition from a database; and according to the query condition, determining detailed record data in the index range. The method is mainly used for querying mass data.
Description
Technical field
The present invention relates to field of data service, particularly a kind of disposal route of mass data and device.
Background technology
The inquiry, statistical conversion etc. of existing overall analysis system to mass data give relevant technical thought.The query scheme implementation procedure in this overall analysis system as shown in Figure 1.As can be known from Fig. 1, existing overall analysis system adopts traditional Relational DataBase to store many structural datas, and directly externally provides QueryTicket.
It can thus be appreciated that in the inquiry of process mass data and storing process, at least there is following technical matters in existing overall analysis system:
Traditional Relational DataBase storage and query text unstructured data efficiency are extremely low, and text can only adopt big field to store in a database, and no matter be store or retrieval, efficiency is very low.
Inquiry concurrency is low, by rowkey direct detailed forms data of inquiry from relational database, by the constraint of disk I/O and relational database storage mode, can not provide the height of low delay concurrent inquiry.
The technology contents that querying condition needs input more professional, higher to the technical requirement of personnel query, and combination condition query needs to set up very many joint indexs, in order to keep the consistance of data, can warehouse-in efficiency be affected, in order to more conditional combination can be mated, need to set up very many joint indexs, to improve search efficiency, directly cause the maintainability of joint index poor.
Summary of the invention
In order to solve in prior art in the face of the storage of mass data and inquiry time efficiency low, the technical matters such as maintainability is poor, the present invention proposes a kind of disposal route and device of mass data.
A disposal route for mass data, comprising:
The inquiry request that reception describes with service metadata, described inquiry request comprises querying condition;
The technical element data that querying condition is corresponding according to the mapping relationship searching between service metadata with technical element data;
The index range meeting querying condition is searched according in described technical element data to data storehouse;
Detailed forms data is determined according to inquiry request in described index range.
A treating apparatus for mass data, comprising:
Receiver module, for the inquiry request that reception describes with service metadata, described inquiry request comprises querying condition;
Mapping block, for the technical element data that querying condition according to the mapping relationship searching between service metadata with technical element data is corresponding;
Index module, for searching according in described technical element data to data storehouse the index range meeting querying condition;
Determination module, for determining detailed forms data according to querying condition in described index range.
The large data framework of increasing income is combined in scheme provided by the invention, by service metadata automatic mapping being become the means of service metadata, complicated technical pattern is shielded to application, search efficiency owing to needing more professional technology contents to cause when solving personnel query usage data storehouse in prior art is low, the technical matters of poor availability, improve ease for use and the dirigibility of system, and improve the practicality of combination condition query by the range index determined, improve the maintainability of search efficiency and system.
Accompanying drawing explanation
Accompanying drawing is used to provide a further understanding of the present invention, and forms a part for instructions, together with embodiments of the present invention for explaining the present invention, is not construed as limiting the invention.In the accompanying drawings:
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the query scheme implementation procedure schematic diagram in background technology in overall analysis system;
The processing procedure schematic diagram of the mass data that Fig. 2 provides for the embodiment of the present invention 1;
The process flow diagram of the disposal route of the mass data that Fig. 3 provides for the embodiment of the present invention 1;
Relation schematic diagram in the disposal route of the mass data that Fig. 4 provides for the embodiment of the present invention 1 between service metadata and technical element data;
Another relation schematic diagram in the disposal route of the mass data that Fig. 5 provides for the embodiment of the present invention 1 between service metadata and technical element data;
The schematic diagram of multiple index process is realized in the disposal route of the mass data that Fig. 6 provides for the embodiment of the present invention 1;
The structural representation of the treating apparatus of the mass data that Fig. 7 provides for the embodiment of the present invention 2.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Further, following embodiment is possibility of the present invention, embodiment put in order and the numbering of embodiment and its order preferably performed have nothing to do.
Embodiment 1
The present embodiment, provides a kind of disposal route of mass data, and the method is applicable to the process such as the inquiry of mass data in the database (NOSQL) based on distributed file system and non-relational and storage.Concrete, this database can be HBase (Hadoop Database, a kind of distributed, towards the PostgreSQL database arranged).In addition, the executive agent in the present embodiment method is index server, and this index server can be deployed in the appropriate location in HBase Database Systems, and as this index server, can be deployed in relation node server first-class.
If Fig. 2 is the schematic diagram that the method is implemented to improve, wherein by application layer, interface (i.e. inquiry proxy service), metadata, database (i.e. Hbaes), storage (HDFS, document storage system) etc. five standalone module decoupling zeros, be recombined into together, to improve the high concurrency random challenge service of low delay.
As shown in Figure 3, the method mainly comprises:
Step 101, the inquiry request described with service metadata is issued the service metadata escape funtion part of inquiry proxy service by application layer, and this inquiry request comprises querying condition, and described querying condition is at least one business dimension.
Metadata (Metadata) is the data about data.In data warehouse, the data that metadata can help the developer of data warehouse administrator and data warehouse to find them to be easily concerned about; Metadata is the structure of data and the data of method for building up in data of description warehouse, it can be divided into two classes by the difference of purposes: technical element data (Technical Metadata) and service metadata (BusinessMetadata)
1. technical element data store the data about data warehouse ins and outs, for developing the data used with management data warehouse, it mainly comprises following information: the description of (1) Based Data Warehouse System, comprise the definition of store mode, view, dimension, hierarchical structure and derived data, and the position of Data Mart and content; (2) architecture of operation system, data warehouse and Data Mart and pattern; (3) algorithm gathered, comprises tolerance and dimension definition algorithm, data granularity, subject fields, assemble, gather, predefined inquiry and report; (4) by the mapping of operating environment to data warehouse environment, source data and their content, Data Segmentation, data extraction, cleaning, transformation rule and Refresh Data rule, safety (subscriber authorisation and access control) is comprised.
2. service metadata describes the data data warehouse from operational angle, it provides the semantic layer between user and real system, makes the business personnel being ignorant of computer technology also " can understand " data in data warehouse.Service metadata mainly comprises following information: the data model expressed by the business terms of user, object name and attribute-name; The principle of visit data and the source of data; The information etc. of the analytical approach that system provides and formula and form.Because the definition in the present embodiment of technical element data and service metadata is same as the prior art, also belong to the common practise scope of those skilled in the art, so do not repeat simultaneously.
Step 102, after service metadata escape funtion part receives the inquiry request that application layer describes with service metadata, the technical element data corresponding to this querying condition described with service metadata according to the mapping relationship searching between service metadata and technical element data in the metadata.
Metadata is more common in IT system, but the metadata information of most description technique aspect, and the metadata of business information is all use document form to carry out record usually, and do not have means can ensure the consistance of the definition of document and system; Further, technical element data and service metadata relevance are not directly perceived, usually need business personnel to grasp certain IT technology, just can find corresponding relevance, waste time and energy.Therefore in above-mentioned steps 102, service metadata is mapped to the process of technical element data, it is one of improvement of the present embodiment, so the technical element data corresponding to above-mentioned this querying condition described according to the mapping relationship searching service metadata between service metadata and technical element data, specifically comprise:
The technology dimension described by technical element data corresponding to the business dimension comprised in the search request that analysis service metadata describes;
According to the physics table of the technology dimension determination technical element data found, physics table comprises the storing directory (HDFS as in Fig. 4) of physics table, the row field recorded in physics table, the major key rowkey of the technical element data recorded in physics table, the source (the ETL metadata as in Fig. 4) of the technical element data recorded in multiple index and physics table.
Such as: as shown in Figure 4, service metadata and technical element data are two independently partial blocks originally, by increasing one deck mapping relations to service metadata and technical element data in the present embodiment, by the definition to relation, allow service metadata become one piece with technical element Data Integration, make real the having trace to search of service metadata.
As shown in Figure 5, the present embodiment defines the relation between metadata, a service application comprises multiple index, combined by multiple business dimension, and be mapped on a physics table, physics table can be had a clear understanding of by technical element data and which row be made up of, leave in which HDFS catalogue, and which the data in can also being induced sweat by ETL metadata come from, thus reach the object of trace to the source (blood lineage).
Such as: on the application layer for user provides the interface of input inquiry condition, user can input operational indicator as querying condition and business dimension, but at least need the business dimension described at this interface input service metadata, as name, be the physics table comprising named field name according to the technical element data that this name business dimension is mapped to.Meanwhile, also provide rowkey, the storing directory of name in this physics table and can know and know that the data of this table come wherefrom.
Between establishing techniques metadata and service metadata mapping relations process in, mainly contain following critical path:
1. dimension, index Naming conventions; First need dimension, the index Naming conventions set up in service metadata, the name of the dimension in total system, index is unique and business implication is clear.
2. lexicon; From existing index specification, obtain all dimensions and the vocabulary that communicates in index name, form the lexicon that Chinese is corresponding with English.
3. service metadata; The mainly title of operational indicator, business dimension and other relevant information
4. Chinese matching; Operational indicator is carried out Chinese matching with business dimension title with the vocabulary that communicates in lexicon, corresponding English field after coupling, can be obtained
5. dimension coupling; The dimensional attribute of operational indicator and dictionary table are mated, corresponding English table name after coupling, can be obtained
6. associate out technical element data: final system Automatic Combined can go out technical element data corresponding to service metadata.
This metadata that can be mapped to technical element data according to business datum that the present embodiment provides has three obvious benefits:
Business personnel, without the need to grasping IT technology, just can understand business datum source and process, can analyze the accuracy of data, find out data Problems existing fast.
Technician, when modifying to data, clearly can know which business is the data of amendment can have influence on, to reduce the problem that data maintenance process produces.
System adopts the participle pattern of natural language automatically to set up the mapping relations between service metadata and technical element data from backstage.
Step 103, returns the technical element data that find to technical element Meaning transfer funtion part.
Step 104, technical element Meaning transfer funtion part directly searches the index range meeting querying condition from HBaseNOSQL database according to technical element data.
Above-mentionedly from HBase NOSQL database, search the index range meeting querying condition according to technical element data, specifically comprise:
The scope of the rowkey of this business dimension is searched in the dimensional extent index that the every querying condition (i.e. business dimension) with inquiry request matches separately; The rowkey scope of the every business dimension found is got and occurs simultaneously as index range.
The process of searching index range in the present embodiment is a kind of secondary or multiple index process, different from the process that the querying condition inputted according to user in prior art directly searches rowkey.As shown in Figure 6, first from the business dimension corresponding to respective querying condition, locate the scope of the rowkey that each querying condition covers, then undertaken getting process of occuring simultaneously by locating each rowkey scope of obtaining, we just can obtain a less rowkey scope like this.Last tissue filter device again, carries out full table scan in this rowkey scope, obtains Query Result.
Such as: querying condition: the range of age [20,27]; Sex=man; The level of consumption [200,500]
Business dimension location/index: each business dimension has a dimensional extent concordance list, if the tables of data that above-mentioned three conditions will be inquired about is three business dimensions, namely there are three dimensions, then have three dimensional extent concordance lists, i.e. the range of age concordance list, sex range index table and level of consumption range index table, the range of age concordance list sample data is as following table one:
Table one
Table one is the business dimension range index table at an age, and first row is the major key rowkey of concordance list, by the different phase at the age value as rowkey.As above, shown in table one, 20-25 and 25-30 two index lines of the data span in age 20 to 27 scope that inquire about, namely we can draw a circle to approve 20 to 27 years old data rowkey scopes at routine table is [30000,60000].
The concordance list of other dimension tables also roughly the same, then the rowkey scope returning the sensing routine data table come with the range of age concordance list, sex range index table and level of consumption range index table does a common factor, namely represent in routine table, the packet of this rowkey scope, containing condition data, so just reduces the scope wanting data query.
Finally, line by line scan within the scope of this, whether the value of coupling age field is between 20 to 27, and whether the value of sex field is man, and whether the value of level of consumption field is between 200 to 500, returns the result set mated completely.In actual items, in order to the result of fast return coupling first 100 can directly can be returned, and the rowkey pointed by 100, conveniently should be used as page turning and again inquire about, just can directly have matched from the rowkey start range of 100 when page turning.
In the method provided in the present embodiment, the way of advanced line index range-based searching has the benefit of two highly significants:
Do not have network bottleneck, be all two values from business dimension search index data out, take the network bandwidth little.
Conveniently increase dimension, increase a dimension and just mean increase dimensional extent index, and the data volume of this index is also very little.
In above-mentioned index range, technical element Meaning transfer funtion part can determine detailed forms data according to the inquiry combination condition (being now mapped to the form described by technical element data) in inquiry request, specifically comprises step 105-108 below.
Step 105, memstore(MemStore to database is Sorted Memory Buffer, first the data of user's write can put into MemStore) index range in buffer memory searches corresponding detailed forms data according to inquiry request, if find, then perform step 106, if do not find, then perform step 107.
Step 106, returns to application layer by the detailed forms data found, and process ends.
Step 107, searches to bottom document storage system, namely searches in HDFS;
Step 108, returns to application layer by the detailed forms data found, and process ends.
The large data framework of increasing income is utilized in the method that the present embodiment provides, complicated technical pattern is shielded to application layer, the technical requirement to personnel query is alleviated by the mode mapped, improve ease for use and the dirigibility of system, and improve the practicality of combination condition query by business dimension range index, improve the maintainability of search efficiency and system.
Embodiment 2
Realize for the ease of method in embodiment 1, the present embodiment provides a kind for the treatment of apparatus of mass data, and preferably, this device is applicable to HBase, be equivalent to the index server in embodiment 1, as shown in Figure 7, comprise: receiver module 21, mapping block 22, index module 23, determination module 24.
Receiver module 21, for the inquiry request that reception describes with service metadata, described inquiry request comprises querying condition; Mapping block 22, for the technical element data corresponding according to the mapping relationship searching querying condition between service metadata with technical element data; Index module 23, for searching according in technical element data to data storehouse the index range meeting querying condition; Determination module 24, for determining detailed forms data according to querying condition in index range.
Preferably, above-mentioned querying condition is specially at least one business dimension.Wherein, mapping block 22 can be equivalent to the service metadata escape funtion part mentioned in embodiment 1, specifically comprises:
Analytic unit, the technology dimension described by technical element data corresponding to the business dimension comprised in the search request for the description of analysis service metadata;
Determining unit, for the physics table according to the technology dimension determination technical element data found, physics table comprises the storing directory of physics table, the row field recorded in physics table, the major key rowkey of the technical element data recorded in physics table, the source of the technical element data recorded in multiple index and physics table.
Index module 23 can be equivalent to mention amount technical element Meaning transfer funtion part in embodiment 1, specifically comprises:
Range cells, for searching the scope of the rowkey of this business dimension in the dimensional extent index that matches separately at the every business dimension with inquiry request;
Common factor unit, occurs simultaneously as index range for the rowkey scope of the every business dimension found being got.
Further, determination module 24, specifically for searching corresponding detailed forms data according to inquiry request in the index range in the memstore buffer memory of database, if find, then returns; If do not find, then search in bottom document storage system.
The device that the present embodiment provides has technical element data service metadata being mapped to correspondence, and by multiple index determination index range, and in this index range, determine the function of detailed forms data, so solve when inquiring about in high-volume database in prior art, higher to personnel query technical requirement, and the technical matters low to how concurrent querying condition search efficiency, and then achieve the search efficiency improving mass data, increase the technique effect of inquiry dirigibility and availability.
The products such as the said equipment that the embodiment of the present invention provides or device belong to the flow and method of computer program for foundation, and according to each step corresponding consistent mode completely with method flow in embodiment of the method 1 and/or accompanying drawing, the functional module provided.And because this functional module is the software service realized by the mode of computer program, so the functional module specifically do not mentioned for device embodiment 2, owing to considering that the content recorded according to said method embodiment has enough made those skilled in the art determine directly, expectedly to realize the functional module that described step must set up, so be not repeated herein from each process step of method record.
The part that technical scheme of the present invention contributes to prior art in essence is in other words the function embodied with the form of software product, in other words: even if the function body of the method for each equipment of device of the present invention, equipment or composition system performed by it or realization is hardware, but the part in fact realizing above-mentioned functions of the present invention is but module or the unit of computer software product.And this computer software product can be stored in the storage medium that can read, as the floppy disk of computing machine, hard disk or CD etc., comprise some instructions and perform method described in each embodiment of the present invention in order to make an equipment.
The above, be only the specific embodiment of the present invention, but the present invention can have multiple multi-form embodiment, by reference to the accompanying drawings the present invention is illustrated above, this does not also mean that the embodiment that the present invention applies can only be confined in these specific embodiments, those skilled in the art should understand, embodiment provided above is some examples in multiple preferred implementation, and the embodiment of any embodiment the claims in the present invention all should within the claims in the present invention scope required for protection; Those skilled in the art can modify to technical scheme described in each embodiment above, or carries out equivalent replacement to wherein portion of techniques feature.Within the spirit and principles in the present invention all, any amendment done, equivalent to replace or improvement etc., within the protection domain that all should be included in the claims in the present invention.
Claims (10)
1. a disposal route for mass data, is characterized in that, comprising:
The inquiry request that reception describes with service metadata, described inquiry request comprises querying condition;
The technical element data that querying condition is corresponding according to the mapping relationship searching between service metadata with technical element data;
The index range meeting querying condition is searched according in described technical element data to data storehouse;
Detailed forms data is determined according to inquiry request in described index range.
2. method according to claim 1, is characterized in that, described querying condition comprises at least one business dimension; The described technical element data that querying condition is corresponding according to the mapping relationship searching between service metadata with technical element data, specifically comprise:
The technology dimension described by technical element data corresponding to the business dimension comprised in the inquiry request that analysis service metadata describes;
According to the physics table of the technology dimension determination technical element data found, described physics table comprises the storing directory of described physics table, the row field recorded in physics table, the major key rowkey of the technical element data recorded in physics table, the source of the technical element data recorded in multiple index and physics table.
3. method according to claim 2, is characterized in that, describedly searches according in described technical element data to data storehouse the index range meeting querying condition, specifically comprises:
The scope of the rowkey of this business dimension is searched in the dimensional extent index that the every business dimension with inquiry request matches separately;
The rowkey scope of the every business dimension found is got and occurs simultaneously as index range.
4. method according to claim 1 and 2, is characterized in that, describedly in described index range, determines detailed forms data according to inquiry request, specifically comprises:
In the described index range in the buffer memory of database, search corresponding detailed forms data according to inquiry request, if find, then return; If do not find, then search in bottom document storage system.
5. method according to claim 1 and 2, is characterized in that, described database is distributed data base.
6. a treating apparatus for mass data, is characterized in that, comprising:
Receiver module, for the inquiry request that reception describes with service metadata, described inquiry request comprises querying condition;
Mapping block, for the technical element data that querying condition request according to the mapping relationship searching between service metadata with technical element data is corresponding;
Index module, for searching according in described technical element data to data storehouse the index range meeting querying condition;
Determination module, for determining detailed forms data according to querying condition in described index range.
7. device according to claim 6, is characterized in that, described querying condition comprises at least one business dimension;
Described mapping block, specifically comprises:
Analytic unit, the technology dimension described by technical element data corresponding to the business dimension comprised in the search request for the description of analysis service metadata;
Determining unit, for the physics table according to the technology dimension determination technical element data found, described physics table comprises the storing directory of described physics table, the row field recorded in physics table, the major key rowkey of the technical element data recorded in physics table, the source of the technical element data recorded in multistage level index and physics table.
8. device according to claim 7, is characterized in that, described index module, specifically comprises:
Range cells, for searching the scope of the rowkey of this business dimension in the dimensional extent index that matches separately in the every business with inquiry request;
Common factor unit, occurs simultaneously as index range for the rowkey scope of the every querying condition business dimension found being got.
9. the device according to claim 6 or 7, is characterized in that, described determination module, specifically for searching corresponding detailed forms data according to inquiry request in the described index range in the memstore buffer memory of database, if find, then returns; If do not find, then search in bottom document storage system.
10. the device according to claim 6 or 7, is characterized in that, this device is applicable to distributed data base.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310726882.5A CN104750681B (en) | 2013-12-25 | 2013-12-25 | A kind of processing method and processing device of mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310726882.5A CN104750681B (en) | 2013-12-25 | 2013-12-25 | A kind of processing method and processing device of mass data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104750681A true CN104750681A (en) | 2015-07-01 |
| CN104750681B CN104750681B (en) | 2018-02-23 |
Family
ID=53590392
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310726882.5A Active CN104750681B (en) | 2013-12-25 | 2013-12-25 | A kind of processing method and processing device of mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104750681B (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105205162A (en) * | 2015-09-29 | 2015-12-30 | 烽火通信科技股份有限公司 | HBase secondary-index storage and query system and query method thereof |
| CN105302869A (en) * | 2015-09-29 | 2016-02-03 | 烽火通信科技股份有限公司 | HBase secondary index query and storage system and query method |
| CN106326309A (en) * | 2015-07-03 | 2017-01-11 | 阿里巴巴集团控股有限公司 | Data query method and device |
| CN106547807A (en) * | 2015-09-23 | 2017-03-29 | 财团法人工业技术研究院 | Data analysis method and device |
| CN106776861A (en) * | 2016-11-28 | 2017-05-31 | 北京亚信数据有限公司 | A kind of indicator consilience analysis method and analysis system |
| CN106909641A (en) * | 2017-02-16 | 2017-06-30 | 青岛高校信息产业股份有限公司 | A kind of real-time data memory device |
| CN108255829A (en) * | 2016-12-28 | 2018-07-06 | 腾讯科技（北京）有限公司 | Data search method and device |
| CN108628894A (en) * | 2017-03-21 | 2018-10-09 | 阿里巴巴集团控股有限公司 | Data target querying method in data warehouse and device |
| CN108763527A (en) * | 2018-05-31 | 2018-11-06 | 北京明朝万达科技股份有限公司 | A kind of searching method and device of business datum |
| CN108763240A (en) * | 2018-03-22 | 2018-11-06 | 五八有限公司 | Data query method, apparatus, equipment and storage medium based on OLAP |
| CN109582649A (en) * | 2018-12-03 | 2019-04-05 | 郑州云海信息技术有限公司 | A kind of metadata storing method, device, equipment and readable storage medium storing program for executing |
| CN109977699A (en) * | 2019-03-26 | 2019-07-05 | 贝富（广州）新技术有限公司 | House Property information storage means, system and storage medium based on block chain |
| CN110781228A (en) * | 2018-07-13 | 2020-02-11 | 马上消费金融股份有限公司 | Data query method, system and device based on data warehouse and storage medium |
| CN110851663A (en) * | 2019-11-05 | 2020-02-28 | 北京明略软件系统有限公司 | Method and apparatus for managing metadata |
| CN110928979A (en) * | 2019-11-11 | 2020-03-27 | 北京明略软件系统有限公司 | Method and apparatus for managing technical metadata |
| CN110929120A (en) * | 2019-11-15 | 2020-03-27 | 北京明略软件系统有限公司 | Method and apparatus for managing technical metadata |
| CN110941657A (en) * | 2019-11-08 | 2020-03-31 | 支付宝(杭州)信息技术有限公司 | Service data processing method and device |
| CN110968679A (en) * | 2018-09-29 | 2020-04-07 | 北京国双科技有限公司 | Data query method and device |
| CN111581236A (en) * | 2020-04-02 | 2020-08-25 | 中国邮政储蓄银行股份有限公司 | Data query method and device |
| CN112579634A (en) * | 2019-09-30 | 2021-03-30 | 北京国双科技有限公司 | Data query method, device, storage medium and equipment |
| CN112860696A (en) * | 2021-02-07 | 2021-05-28 | 中国邮政储蓄银行股份有限公司 | Data query method and device and data query model |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5727196A (en) * | 1992-05-21 | 1998-03-10 | Borland International, Inc. | Optimized query interface for database management systems |
| CN101178732A (en) * | 2007-12-12 | 2008-05-14 | 江苏省电力公司 | Method for quick-speed realizing data store house process based on metadata |
| CN102033870A (en) * | 2009-09-24 | 2011-04-27 | 中国移动通信集团公司 | Data searching method and device |
| CN102402530A (en) * | 2010-09-13 | 2012-04-04 | 方正国际软件有限公司 | Metadata distributed query method, device and system |
-
2013
- 2013-12-25 CN CN201310726882.5A patent/CN104750681B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5727196A (en) * | 1992-05-21 | 1998-03-10 | Borland International, Inc. | Optimized query interface for database management systems |
| CN101178732A (en) * | 2007-12-12 | 2008-05-14 | 江苏省电力公司 | Method for quick-speed realizing data store house process based on metadata |
| CN102033870A (en) * | 2009-09-24 | 2011-04-27 | 中国移动通信集团公司 | Data searching method and device |
| CN102402530A (en) * | 2010-09-13 | 2012-04-04 | 方正国际软件有限公司 | Metadata distributed query method, device and system |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106326309A (en) * | 2015-07-03 | 2017-01-11 | 阿里巴巴集团控股有限公司 | Data query method and device |
| CN106326309B (en) * | 2015-07-03 | 2020-02-21 | 阿里巴巴集团控股有限公司 | Data query method and device |
| CN106547807A (en) * | 2015-09-23 | 2017-03-29 | 财团法人工业技术研究院 | Data analysis method and device |
| CN106547807B (en) * | 2015-09-23 | 2021-01-22 | 财团法人工业技术研究院 | Data analysis method and device |
| US11086881B2 (en) | 2015-09-23 | 2021-08-10 | Industrial Technology Research Institute | Method and device for analyzing data |
| CN105302869B (en) * | 2015-09-29 | 2020-01-17 | 烽火通信科技股份有限公司 | HBase secondary index query and storage system and query method thereof |
| CN105302869A (en) * | 2015-09-29 | 2016-02-03 | 烽火通信科技股份有限公司 | HBase secondary index query and storage system and query method |
| CN105205162A (en) * | 2015-09-29 | 2015-12-30 | 烽火通信科技股份有限公司 | HBase secondary-index storage and query system and query method thereof |
| CN106776861A (en) * | 2016-11-28 | 2017-05-31 | 北京亚信数据有限公司 | A kind of indicator consilience analysis method and analysis system |
| CN108255829A (en) * | 2016-12-28 | 2018-07-06 | 腾讯科技（北京）有限公司 | Data search method and device |
| CN108255829B (en) * | 2016-12-28 | 2021-10-19 | 腾讯科技（北京）有限公司 | Data searching method and device |
| CN106909641A (en) * | 2017-02-16 | 2017-06-30 | 青岛高校信息产业股份有限公司 | A kind of real-time data memory device |
| CN106909641B (en) * | 2017-02-16 | 2020-09-29 | 青岛高校信息产业股份有限公司 | Real-time data memory |
| CN108628894A (en) * | 2017-03-21 | 2018-10-09 | 阿里巴巴集团控股有限公司 | Data target querying method in data warehouse and device |
| CN108763240A (en) * | 2018-03-22 | 2018-11-06 | 五八有限公司 | Data query method, apparatus, equipment and storage medium based on OLAP |
| CN108763527A (en) * | 2018-05-31 | 2018-11-06 | 北京明朝万达科技股份有限公司 | A kind of searching method and device of business datum |
| CN110781228A (en) * | 2018-07-13 | 2020-02-11 | 马上消费金融股份有限公司 | Data query method, system and device based on data warehouse and storage medium |
| CN110781228B (en) * | 2018-07-13 | 2021-07-13 | 马上消费金融股份有限公司 | Data query method, system and device based on data warehouse and storage medium |
| CN110968679A (en) * | 2018-09-29 | 2020-04-07 | 北京国双科技有限公司 | Data query method and device |
| CN109582649B (en) * | 2018-12-03 | 2022-02-18 | 郑州云海信息技术有限公司 | Metadata storage method, device and equipment and readable storage medium |
| CN109582649A (en) * | 2018-12-03 | 2019-04-05 | 郑州云海信息技术有限公司 | A kind of metadata storing method, device, equipment and readable storage medium storing program for executing |
| CN109977699A (en) * | 2019-03-26 | 2019-07-05 | 贝富（广州）新技术有限公司 | House Property information storage means, system and storage medium based on block chain |
| CN112579634A (en) * | 2019-09-30 | 2021-03-30 | 北京国双科技有限公司 | Data query method, device, storage medium and equipment |
| CN110851663A (en) * | 2019-11-05 | 2020-02-28 | 北京明略软件系统有限公司 | Method and apparatus for managing metadata |
| CN110941657B (en) * | 2019-11-08 | 2023-03-31 | 支付宝(杭州)信息技术有限公司 | Service data processing method and device |
| CN110941657A (en) * | 2019-11-08 | 2020-03-31 | 支付宝(杭州)信息技术有限公司 | Service data processing method and device |
| CN110928979A (en) * | 2019-11-11 | 2020-03-27 | 北京明略软件系统有限公司 | Method and apparatus for managing technical metadata |
| CN110928979B (en) * | 2019-11-11 | 2022-12-20 | 北京明略软件系统有限公司 | Method and apparatus for managing technical metadata |
| CN110929120B (en) * | 2019-11-15 | 2022-09-09 | 北京明略软件系统有限公司 | Method and apparatus for managing technical metadata |
| CN110929120A (en) * | 2019-11-15 | 2020-03-27 | 北京明略软件系统有限公司 | Method and apparatus for managing technical metadata |
| CN111581236A (en) * | 2020-04-02 | 2020-08-25 | 中国邮政储蓄银行股份有限公司 | Data query method and device |
| CN112860696A (en) * | 2021-02-07 | 2021-05-28 | 中国邮政储蓄银行股份有限公司 | Data query method and device and data query model |
| CN112860696B (en) * | 2021-02-07 | 2024-04-12 | 中国邮政储蓄银行股份有限公司 | Data query method and device and data query model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104750681B (en) | 2018-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104750681A (en) | Method and device for processing mass data | |
| CN103177055B (en) | It is stored as row storage and row stores the hybrid database table of the two | |
| CN103177056B (en) | It is stored as row storage and row stores the hybrid database table of the two | |
| CN103177058B (en) | It is stored as row storage and row stores the hybrid database table of the two | |
| US20170193041A1 (en) | Document-partitioned secondary indexes in a sorted, distributed key/value data store | |
| US20150142829A1 (en) | System, apparatus, program and method for data aggregatione | |
| US20120323971A1 (en) | Optimizing data storage and access of an in-memory database | |
| US8825621B2 (en) | Transformation of complex data source result sets to normalized sets for manipulation and presentation | |
| WO2017170459A1 (en) | Method, program, and system for automatic discovery of relationship between fields in environment where different types of data sources coexist | |
| US20100235344A1 (en) | Mechanism for utilizing partitioning pruning techniques for xml indexes | |
| CN106611053A (en) | Data cleaning and indexing method | |
| JP6159908B1 (en) | Method, program, and system for automatic discovery of relationships between fields in a heterogeneous data source mixed environment | |
| US20110289061A1 (en) | Redistribute native xml index key shipping | |
| Weintraub et al. | Needle in a haystack queries in cloud data lakes. | |
| Qian et al. | An evaluation of Lucene for keywords search in large-scale short text storage | |
| Cappellari et al. | A path-oriented rdf index for keyword search query processing | |
| US20210303533A1 (en) | Automated optimization for in-memory data structures of column store databases | |
| Ptiček et al. | Big data and new data warehousing approaches | |
| Li et al. | Answering why-not questions on top-k augmented spatial keyword queries | |
| Desai et al. | An empirical analysis to identify the effect of indexing on influence detection using graph databases | |
| Pokorný et al. | Indexing Patterns in Graph Databases. | |
| Rahman et al. | Analyze Database Optimization Techniques | |
| Pivert | NoSQL data models: trends and challenges | |
| Truică et al. | A scalable document-based architecture for text analysis | |
| Pokorný et al. | Graph pattern index for Neo4j graph databases |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |