CN107016025A - A kind of method for building up and device of non-relational database index - Google Patents
A kind of method for building up and device of non-relational database index Download PDFInfo
- Publication number
- CN107016025A CN107016025A CN201611019641.7A CN201611019641A CN107016025A CN 107016025 A CN107016025 A CN 107016025A CN 201611019641 A CN201611019641 A CN 201611019641A CN 107016025 A CN107016025 A CN 107016025A
- Authority
- CN
- China
- Prior art keywords
- index
- business datum
- relational database
- business
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of method for building up and device of non-relational database index.Methods described includes：Pending business datum, reading attributes field configuration table are obtained, the attribute field allocation list includes setting up the service fields of dimension previously according to the index of the index operation demand setting of business datum；Determine to need to set up in the business datum main business field indexed according to the attribute field allocation list；The field properties data associated according to the main business field and with the main business field set up the non-relational database index using the main business field as index key.Utilize each embodiment of the application, the foundation of wide table realization index that can be based on non-relational database, the business of oneself can be set up according to business scenario to index and ensure data and the uniformity of index, it can flexibly, quickly, easily set up and be indexed with the more preferable non-relational database of business scenario suitability, improve index and set up efficiency.
Description
Technical field
The application belongs to microcomputer data processing field, more particularly to the foundation that a kind of non-relational database is indexed
Method and device.
Background technology
With developing rapidly for computer and Internet technology, the business number dependent on many industries of internet is presented big
Data volume, the trend of rapid growth.The arrival in big data epoch, the storage to business datum proposes new challenge.
At present, conventional business datum storage is also primarily based upon the relational data set up on the basis of relational model
Storehouse, the data come by means of the mathematical concepts such as algebra of sets and method in processing data storehouse.Such as the relationship type of current current main-stream
Database has Oracle, DB2, PostgreSQL, Microsoft SQL Server, Microsoft Access, MySQL.
Under a large number of services data environment, relevant database can not meet the data storage requirement of some business developments, so
The medium of storage is migrated to NoSQL databases (database for referring to non-relational) direction, and the data volume that NoSQL can be stored is non-
Chang great, can be very good to solve the problems, such as point storehouse point table of relevant database.
But because rowkey (row major key) limitations of wide table, at present only in the case where clearly knowing rowkey,
The data of the condition of satisfaction can be inquired, for example generally business personnel does not know that (such as UserId is rowkey typically
The non-rowkey of rowkey, UserMobile), the personnel of data analysis may only know UserMobile information.This is general
Data Analysis Platform (data modeling, case are tried, operation) etc. can not meet data manipulation demand.Have in currently available technology
Some search engines can support of this sort index to set up, e.g., Microsoft JET (Joint Engineering
Technologe) be the embedded data library facility that Access and Visual Basic are provided core element, JET is a kind of complete
Energy relational database engine, can be used to handle most of small and medium databases；ODBC(Open DataBase
Connectivity, Open Database Connection) it is a kind of database access standard defined by Microsoft, it provides a kind of
The data bank access method of standard is to access the database of different platform.Search engine skill based on for example above-mentioned JET or ODBC
Art can realize that the index of the wide table under some business scenarios is set up.
But existing use search engine is set up in the mode of index, there is delay the index construct time, the index constructed
It is same to take very big memory space, for the few business datum index operation demand of some service search dimensions, using searching
Structure time that index holds up tools build index then relative consumption is longer, process is complicated, and causes to build because search dimension is less
All stability it is also poor, overall service efficiency and effect are relatively low.
The content of the invention
The application purpose is to provide a kind of method for building up and device of non-relational database index, can closed based on non-
It is that the wide table of type database realizes the foundation of index, the business of oneself can be set up according to business scenario and indexes and ensures data
With the uniformity of index, flexibly, quickly, easily it can set up and the more preferable non-relational database rope of business scenario suitability
Draw, improve index and set up efficiency.
What the method for building up and device for a kind of non-relational database index that the application is provided were realized in：
A kind of method for building up of non-relational database index, methods described includes：
Pending business datum is obtained, reading attributes field configuration table, the attribute field allocation list includes in advance
The index set according to the index operation demand of business datum sets up the service fields of dimension；
Determine to need to set up in the business datum main business field indexed according to the attribute field allocation list；
The field properties data associated according to the main business field and with the main business field are set up with the main business
Field of being engaged in indexes for the non-relational database of index key.
What a kind of non-relational database was indexed sets up device, and described device includes：
Data read module, the pending business datum for obtaining, reading attributes field configuration table, the attribute field
Allocation list includes setting up the service fields of dimension previously according to the index of the index operation demand setting of business datum；
Index field determining module, for being determined to need to set up in the business datum according to the attribute field allocation list
The main business field of index；
Index construct module, for the field properties number associated according to the main business field and with the main business field
According to the non-relational database index set up using the main business field as index key.
The method for building up and device for a kind of non-relational database index that the application is provided, can be in advance in attribute field
To designing and storing the attribute field for needing to carry out business operation in allocation list (such as scheme tables data), so in business datum
Can be with reading attributes field configuration table during storage, the field properties data associated according to service fields and with the service fields are entered
The foundation of line index field.The embodiment provided using the application, it is possible to achieve realized according to actual business scenario demand
The foundation of the wide table index of non-relational database, so that operating personnel flexibly, quickly, easily sets up and business scenario
The more preferable non-relational database index of suitability, the index for solving the retrieval of the various dimensions under big data environment sets up demand, full
Sufficient user sets up the demand for being adapted to index according to business scenario, improves index and sets up efficiency.
Brief description of the drawings
, below will be to embodiment or existing in order to illustrate more clearly of the embodiment of the present application or technical scheme of the prior art
There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments described in application, for those of ordinary skill in the art, are not paying the premise of creative labor
Under, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is a kind of a kind of method flow signal of embodiment of method for building up of non-relational database index of the application
Figure；
Fig. 2 is the attribute field allocation list schematic diagram designed using scheme in a kind of embodiment scene of the application；
Fig. 3 is a kind of next handling process signal for setting up index of application scenarios for providing index security mechanism of the application
Figure；
Fig. 4 is the time aggregation side of the design Storage table of Hbase index_table in the application one embodiment scene
Formula schematic diagram；
Fig. 5 be the application provide a kind of non-relational database index set up a kind of modular structure of embodiment of device
Implement figure；
Fig. 6 be the application provide a kind of non-relational database index set up a kind of modular structure of embodiment of device
Implement figure.
Embodiment
In order that those skilled in the art more fully understand the technical scheme in the application, it is real below in conjunction with the application
The accompanying drawing in example is applied, the technical scheme in the embodiment of the present application is clearly and completely described, it is clear that described implementation
Example only some embodiments of the present application, rather than whole embodiments.Based on the embodiment in the application, this area is common
The every other embodiment that technical staff is obtained under the premise of creative work is not made, should all belong to the application protection
Scope.
A kind of method for building up of non-relational database index described herein is carried out in detail below in conjunction with the accompanying drawings
Explanation.Fig. 1 is a kind of a kind of method flow diagram of embodiment of method for building up of herein described non-relational database index.Though
So this application provides the method operating procedure as described in embodiment or flow chart, but based on labor conventional or without creativeness
More or operating procedure partly less after merging or modular unit can be included in methods described or device by moving.In logic
Property on the step of necessary causality is not present or in structure, the execution sequence of these steps or the modular structure of device are not limited to
The embodiment of the present application or execution sequence shown in the drawings or modular structure.Described method or the dress in practice of modular structure
Put or when end product is applied, can according to embodiment or method shown in the drawings or modular structure carry out order execution or
Parallel perform (environment of such as parallel processor or multiple threads, even include the implementation environment of distributed treatment).
The method for building up of non-relational database index described herein is set up there is provided a set of combination business to be indexed
Method, business personnel is available to be only needed to provide a set of attribute field allocation list data being pre-designed for business scenario, such as
Scheme is illustrated, it is necessary to which data dimension to set up index at, then can be automatically in data storage using the application method
When landing generation index data.In other embodiments, operating personnel can use database language known to oneself (such as
SQL) operated, the Scheme that SQL aggregate users are designed can be mapped as to the grammer that wide table is performed.Specifically such as Fig. 1 institutes
Show, in a kind of a kind of embodiment of the method for building up for non-relational database index that the application is provided, methods described can be wrapped
Include：
S1：Pending business datum is obtained, reading attributes field configuration table, the attribute field allocation list includes in advance
The index first set according to the index operation demand of business datum sets up the service fields of dimension.
What the pending business datum described in the present embodiment can index to need foundation in non-relational database
Data in business datum, such as NoSQL databases.In other implement scenes, described pending business datum can also
For the business datum in other relevant databases, such as the data in relevant database are transferred in non-relational database
Data in the relevant database of Shi Jianli index process.In other embodiments, described pending business datum
Can be to store business datum in non-relational database the need for obtaining or newly-generated, such as directly from front-end server
The order business datum of acquisition.
Obtain after pending business datum, the present embodiment can arrive non-relation in pending business datum landing storage
Before type database, the attribute field allocation list pre-set is read.The attribute field allocation list mainly includes advance root
Which dimension to set up the service fields of index the need for being set according to business datum in, can be used for reading the category in data storage
Property field configuration table, with the determination and foundation of the index field for realizing non-relational database.Described attribute field allocation list
The customized design construction of business scenario can be combined, can be included using the computer being adapted with database or index data
Language editor is set.In a kind of embodiment that the application is provided, the attribute field allocation list pre-set can include：
S101：Need to build generation using scheme language previously according to the business operation of the business datum.
Described scheme (set of database object) is a succinct, efficient computer language, uses its volume
Cheng personnel can break away from the complexity of language in itself, concentrate in prior problem.It is multiduty as one
Programming language, with metalanguage characteristic, can be used as script, can also be made as the extension language of application software
With.Therefore, the embodiment of the present application can design a set of scheme for the usage scenario of business datum in advance, can describe definition
The main attribute field (service fields) of progress business operation is needed, programming personnel can be stored in storage clothes after designing
It is engaged in end, the foundation that scheme is indexed field being read when business datum storage is to database.Fig. 2 is that the application is a kind of
The attribute field allocation list schematic diagram designed in embodiment scene using scheme, the embodiment of the present application is set using scheme language
Count the attribute field allocation list constructed more efficiently convenient, facilitate programming personnel's operation, non-relational number can be effectively improved
Efficiency is set up according to what storehouse was indexed.
The embodiment of the present application can obtain pending business datum, then can read the attribute field pre-set and match somebody with somebody
Put table.The attribute field allocation list includes the service fields of the foundation index dimension set according to business datum, Ke Yi
The field information set up when business datum is stored according to needed for being indexed the attribute field allocation list shortcoming.
S2：Determine to need to set up in the business datum main business field indexed according to the attribute field allocation list.
Obtain after business datum and attribute field allocation list, it may be determined that need to set up the master indexed in the business datum
Service fields.Usual single pending business datum includes multiple attribute fields, such as certain pending business datum
The attribute field that DATA_001 includes has order number, event identifier, user's name, exchange hour, name on account etc..And institute
The service fields of index dimension are set up the need for being set in the attribute field allocation list stated includes event identifier, ID, electronics
Mail, device identification etc..It can then determine that it is event to need to set up the main business field indexed in business datum DATA_001
Identify (EventID), user's name (UserId).
As described above, if pending business datum includes multiple service fields in the attribute field allocation list,
It can represent that needs set up the service fields of respective dimensions in the index of non-relational database of the business datum is stored
Index.Certainly, if there is no the service fields in the attribute field allocation list in the pending business datum obtained, now
The business datum can be labeled as to abnormal data storage, then can be carried out according to advance abnormal data storage processing mode
Processing, such as isolated storage or deletion.
S3：The field properties data associated according to the main business field and with the main business field are set up with the master
Service fields index for the non-relational database of index key.
Further, it can be set up according to pending business datum and the attribute field allocation list read and store the business
The index of the non-relational database of data., can be with foregoing according to the attribute field during specific index is set up
Allocation list determine pending business datum main business field as index index key, and with the main business word
The Duan Guanlian field properties data generation index data for being used to describe the service fields index information, constructs non-relational number
According to the index in storehouse.As in above-mentioned embodiment scene, needed to set up the main business indexed in the business datum DATA_001 determined
Field be event identifier (EventID), user's name (UserId), then can be according to main business field EventID, UserId
And corresponding field properties data set up the index of business datum DATA_001 two dimensions.So, can be by former business
Different secondary attribute dimensions are associated with the underlying attribute dimension of business datum in data, and non-relational number is constructed jointly
According to the index in storehouse, the index for the non-relational database that various dimensions inquiry is set up according to business scenario requirement definition is realized, is improved
The applicability and index of index set up efficiency.
Certainly, after the index for constructing the non-relational database, place can also be normalized to the index
Reason, to improve the efficiency data query of index.
The method for building up for the non-relational database index that the embodiment of the present application is provided, can be configured in attribute field in advance
To designing and storing the attribute field for needing to carry out business operation in table (such as scheme tables data), so stored in business datum
When can be with reading attributes field configuration table, the field properties data that are associated according to service fields and with the service fields carry out rope
Draw the foundation of field.The embodiment provided using the application, it is possible to achieve non-pass is realized according to actual business scenario demand
Be the foundation of the wide table index of type database, so that operating personnel flexibly, it is quick, easily set up and be adapted to business scenario
Property more preferable non-relational database index, the index for solving the retrieval of the various dimensions under big data environment sets up demand, meets and use
The demand for being adapted to index is set up at family according to business scenario, is improved index and is set up efficiency.
In another embodiment of the application, the foundation that can also be indexed for non-relational database sets security mechanism,
Index storage can be ensured using the task list for defining the generation of task marriage relation type database manipulation language, effectively,
The reliable uniformity for ensureing data and index.Specifically, a kind of foundation for non-relational database index that the application is provided
In another embodiment of method, methods described can also include：
S4：The task list set up in advance for pending business datum is checked according to the timed task of setting, according to detection
As a result judge the storage state of the business datum is with the storage state of the corresponding index data in non-relational database
It is no to be consistent；Whether the index task creation that the task list is configured at least include identifying the business datum is expired
The storage state information of temporal information, the business datum and corresponding index data whether normal storage；
If consistent, job status of the pending business datum in the task list is changed accordingly.
The storage of monitoring business datum is combined using timed task and the task list set up is indexed, and index storage is ensured,
Corresponding all data can be generated when making business datum landing storage into database, can be with effective guarantee data and index
Uniformity.
At the data that task list described herein can be designed using the database language of operating personnel's General Properties
The information table of reason state.In a kind of embodiment that the application is provided, the task list can be generated using MQSQL language designs,
It is easy to operating personnel to operate, improves index process efficiency.Specifically, a kind of non-relational database index that the application is provided
In another embodiment of method for building up,
S401：When the task list is included according to the business datum using the setting of MYSQL language including task creation
Between, task execution time, task status mark, task expired time, task completion time data message monitoring table.
Set up in actual index in processing procedure, the business in task list can be handled one by one according to MYSQL task lists
Data.When it is determined that business datum index data effectively, then can be by business datum normal storage into database, while business
The index data of data, which is set up, to be completed., can be with after the completion of the index task creation of a business datum in some embodiments
Corresponding task status mark is updated, or can also directly be deleted directly from task list and be completed index data
Data in MYSQL task lists, as shown in figure 3, Fig. 3 is that a kind of application scenarios for providing index security mechanism of the application are next
Set up the handling process schematic diagram of index.In other embodiments, it can monitor that index task is handled by foregoing task table
Abnormal situation, knows abnormality processing situation, and handled accordingly in time.Therefore, a kind of non-relation that the application is provided
In another embodiment of the method for building up of type database index, methods described can also include：
S402：When monitoring that the business datum stores the index datastore exception of abnormal or described business datum,
Mark and store the abnormal business datum of the processing or index data, and update corresponding task status mark in the task list
Know.
In a kind of another application scenarios of the method for building up of non-relational database index that the application is provided, root
Factually border business scenario demand, user may need to retrieve information in database according to a variety of dimensions.Therefore, it is herein described
In another embodiment of method, index is set up in order to more efficient, the row major key (RowKey) in index can be carried out
Polymerization.Specifically, in another embodiment, methods described can also include：
S5：Identical row major key carries out the polymerization of predetermined dimension in non-relational database to depositing business datum,
Form non-relational database after polymerization.
The embodiment of the present application can be selected from time dimension according to the real data process demand of most business scenarios or
Same business datum source (being such as all buyer or seller or platform or third-party message), traffic data type are polymerize.
Described predetermined dimension can be chosen by designer according to index polymerization demand, and the design of general described polymerization needs to lead to
Often meet the query feature of hot spot data.It can for example be set different according to the different differences with querying condition of business scenario
Rowkey time aggregation.After polymerization by predetermined dimension, after the method provided using the application can be improved for polymerization
The query processing efficiency for the index that non-relational database is set up.
It is identical in the non-relational database to depositing business datum in another embodiment of herein described method
Row major key carry out the polymerization of predetermined dimension, can include：
S501：The polymerization time section that row major keys different in non-relational database are set is set；
S502：In the non-relational database, by data message of the mutually colleague's major key in polymerization time section
The service data information of same row major key is polymerized to, the service data information of the same row major key is configured to include institute
State the service data information of row major key different time version in polymerization time section.
General, the polymerization of time dimension is typically referred in regular hour scope (for example：Three months) in, it is same
Rowkey business datum is placed in same a line, different time version.Described different time version is primarily referred to as being included in institute
State the business datum of the row major key of the different time points of polymerization time section memory storage record.In a specific example, such as together
One row major key deviceID data, all data rowkey during these three months of 1-3 are polymerized to deviceId_0, false
If the business datum for having 5 row major keys to be deviceID the 1-3 months, then can be polymerized to one by 5 row major keys of these three middle of the month
Row major key is deviceId_0 service data information, in the same a line for being placed on Hbase index_table storage tables.With such
Push away, these three month row major key of next 4-6 is polymerized to deviceId_1 for deviceID business datum.In the present embodiment, no
Same row major key can set different polymerization times sections, and such as above-mentioned row major key of correspondence can set for deviceID business datum
It is 3 months to put polymerization time section, while the polymerization time section that can also set the business datum that row major key is UserID is 1 one
Month, or set row major key be Payment business datum polymerization time section be "None", represent without polymerization.Fig. 4 is this
Apply for the time aggregation schematic diagram of Hbase index_table design Storage table in one embodiment scene.
The index polymerization of the present embodiment passage time dimension, can be ranked up the data under same rowkey, soon
The demand for meeting operating personnel's inquiry of speed, improves the search efficiency of non-relational database index.
Data creation method described in the above embodiments of the present application, can be programmed by including but is not limited to computer application
Design language such as JAV is realized, with reference to corresponding non-relational database such as HBase or OTS, and combines necessary hardware (such as
Memory) and the mode of Logical processing unit (such as CPU) realize.
The method for building up of the non-relational database index provided based on the application, the application is also provided and methods described pair
What the non-relational database answered was indexed sets up device.Fig. 5 is a kind of building for non-relational database index that the application is provided
A kind of modular structure of embodiment of vertical device implements figure, as shown in figure 5, described device can include：
Data read module 101, can be used for obtaining pending business datum, reading attributes field configuration table is described
Attribute field allocation list includes the business word that dimension is set up previously according to the index of the index operation demand setting of business datum
Section；
Index field determining module 102, can be used for being determined in the business datum according to the attribute field allocation list
Need to set up the main business field indexed；
Index construct module 103, can be used for the word associated according to the main business field and with the main business field
Section performance data sets up the non-relational database index using the main business field as index key.
As it was previously stated, the attribute field allocation list pre-set can be grasped previously according to the business of the business datum
Need to build generation using scheme language.
What a kind of non-relational database that the application is provided was indexed sets up device, can be in advance in attribute field allocation list
To designing and storing the attribute field for needing to carry out business operation in (such as scheme tables data), so when business datum is stored
Can be with reading attributes field configuration table, the field properties data associated according to service fields and with the service fields are indexed
The foundation of field.The embodiment provided using the application, it is possible to achieve non-relation is realized according to actual business scenario demand
The foundation of the wide table index of type database, so that operating personnel flexibly, quickly, easily sets up and business scenario suitability
More preferable non-relational database index, the index for solving the retrieval of the various dimensions under big data environment sets up demand, meets user
The demand of adaptation index is set up according to business scenario, index is improved and sets up efficiency.
In another embodiment of the application, the foundation that can also be indexed for non-relational database sets security mechanism,
Index storage can be ensured using the task list for defining the generation of task marriage relation type database manipulation language, effectively,
The reliable uniformity for ensureing data and index.Fig. 6 is a kind of foundation dress for non-relational database index that the application is provided
A kind of modular structure implementation figure of embodiment is put, as shown in fig. 6, in another embodiment for the described device that the application is provided,
Described device can also include：
Index ensures processing module 104, can be used for checking to be pending business in advance according to the timed task of setting
The task list that data are set up, judges the storage state of the business datum with corresponding in non-relational database according to testing result
In the storage state of index data whether be consistent；If consistent, the pending business datum is changed accordingly and is existed
Job status in the task list；The index that the task list is configured at least include identifying the business datum is appointed
Business create whether the storage shape of expired temporal information, the business datum and corresponding index data whether normal storage
State information.
At the data that task list described herein can be designed using the database language of operating personnel's General Properties
The information table of reason state.In a kind of embodiment that the application is provided, the task list can be generated using MQSQL language designs,
It is easy to operating personnel to operate, improves index process efficiency.Therefore, in another embodiment, the index is ensured in processing module
Task list can include according to the business datum using MYSQL language set include task creation time, tasks carrying
Time, task status mark, the data message monitoring table of task expired time, task completion time.
Set up in actual index in processing procedure, the business in task list can be handled one by one according to MYSQL task lists
Data.When it is determined that business datum index data effectively, then can be by business datum normal storage into database, while business
The index data of data, which is set up, to be completed.After the completion of the index task creation of a business datum, corresponding task can be updated
Status indicator, or can also directly delete and be completed in the MYSQL task lists of index data directly from task list
Data.In other embodiments, the abnormal situation of index task processing can be monitored by foregoing task table, is known in time different
Normal disposition, and handled accordingly.In another embodiment, index ensures that processing module 104 can be also used for when prison
When measuring abnormal business datum storage or abnormal index datastore of the business datum, mark and store the processing
Abnormal business datum or index data, and update corresponding task status mark in the task list.
Data generating device described in the above embodiments of the present application, can be programmed by including but is not limited to computer application
Design language such as JAV is realized, with reference to corresponding non-relational database such as HBase or OTS, and combines necessary hardware (such as
Memory) and the mode of Logical processing unit (such as CPU) realize.The terminal form of device can include the necessary hardware of combination and hold
Capable non-relational database index formation component or functional module/unit.Described method or apparatus goes for similar
In the server that the wide table index of non-relational database is set up.
Although being mentioned in teachings herein and building attribute field allocation list using scheme, index is deposited with reference to MYQSL tables
Storage is ensured, the polymerization methodses of NoSQL type database time dimensions and business datum or during index datastore exception
The description that the data definition of processing mode etc., information exchange, storage, polymerization processing, information judge, still, the application not office
It is limited to meet standard computer design language or database and the number of the mode mentioned by syntax rule or embodiment
According to the situation of interaction, processing.Foregoing description involved by the embodiment of each in the application is only some embodiments in the application
In application, processing method amended slightly or customized mark on the basis of some standards, method, design language
Standard/processing mode can also carry out the scheme of each embodiment of above-mentioned the application.Certainly, the application the various embodiments described above are being met
Described in process method step other without creative deformations, still can realize identical application, will not be repeated here.
Although this application provides the method operating procedure as described in embodiment or flow chart, based on conventional or noninvasive
The means for the property made can include more or less operating procedures.The step of being enumerated in embodiment order is only numerous steps
A kind of mode in execution sequence, unique execution sequence is not represented., can be with when device in practice or end product execution
Performed according to embodiment or method shown in the drawings order or parallel execution (such as parallel processor or multiple threads
Environment, even distributed data processing environment).Term " comprising ", "comprising" or its any other variant are intended to
Nonexcludability is included, so that process, method, product or equipment including a series of key elements not only will including those
Element, but also other key elements including being not expressly set out, or also include being this process, method, product or equipment
Intrinsic key element.In the absence of more restrictions, be not precluded from the process including the key element, method, product or
Also there are other identical or equivalent elements in person's equipment.
Unit, device or module that above-described embodiment is illustrated etc., can specifically be realized by computer chip or entity, or
Realized by the product with certain function.For convenience of description, describe to be divided into various modules point during apparatus above with function
Do not describe.Certainly, when implementing the application can the function of each module in same or multiple softwares and/or hardware it is real
It is existing, the module for realizing same function can also be realized by the combination of multiple submodule or subelement etc..Dress described above
It is only schematical to put embodiment, for example, the division of the unit, only a kind of division of logic function, when actually realizing
There can be other dividing mode, such as multiple units or component can combine or be desirably integrated into another system, or one
A little features can be ignored, or not perform.It is another, shown or discussed coupling or direct-coupling or communication link each other
Connect can be can be electrical, machinery or other shapes by some interfaces, the INDIRECT COUPLING or communication connection of device or unit
Formula.
Method, module or device described herein can be embedded in controller mode with readable program and realize, the control
Device processed is realized by specific requirement.For example, controller can take such as microprocessor or processor and storage can be by (micro-)
The computer-readable medium of the computer readable program code (such as software or firmware) of computing device, gate, switch, specially
With integrated circuit (Application Specific Integrated Circuit, ASIC), programmable logic controller (PLC) and embedding
Enter the form of microcontroller, the example of controller includes but is not limited to following microcontroller：ARC 625D、Atmel AT91SAM、
Microchip PIC18F26K20 and Silicone Labs C8051F320, Memory Controller is also implemented as depositing
A part for the control logic of reservoir.It is also known in the art that except real in pure computer readable program code mode
Beyond existing controller, controller can be caused with gate, switch, special by the way that method and step is carried out into programming in logic completely
The form of integrated circuit, programmable logic controller (PLC) and embedded microcontroller etc. realizes identical function.Therefore this controller
A kind of hardware component is considered, and the device for realizing various functions included to its inside can also be considered as hardware
Structure in part.Or even, can by for realize the device of various functions be considered as both can be implementation method software
Module can be the structure in hardware component again.
Part of module in herein described device can be in the general of computer executable instructions
Described in context, such as program module.Usually, program module includes performing particular task or realizes specific abstract data class
The routine of type, program, object, component, data structure, class etc..The application can also be put into practice in a distributed computing environment,
In these DCEs, task is performed by the remote processing devices connected by communication network.In distribution
In computing environment, program module can be located at including in the local and remote computer-readable storage medium including storage device.
As seen through the above description of the embodiments, those skilled in the art can be understood that the application can
Realized by the mode of software plus required hardware.Understood based on such, the technical scheme of the application is substantially in other words
The part contributed to prior art can be embodied in the form of software product, can also be by the implementation of Data Migration
During embody.The computer software product can be stored in storage medium, such as ROM/RAM, magnetic disc, CD, bag
Some instructions are included to so that a computer equipment (can be personal computer, end instrument equipment, server, or net
Network equipment etc.) perform method described in some parts of each embodiment of the application or embodiment.
Each embodiment in this specification is described by the way of progressive, same or analogous portion between each embodiment
Divide mutually referring to what each embodiment was stressed is the difference with other embodiment.The whole of the application or
Person part can be used in numerous general or special purpose computing system environments or configuration.For example：Personal computer, handheld device or
Portable set, the system based on microprocessor, programmable electronic equipment include the distribution of any of the above system or equipment
Computing environment etc..
Although depicting the application by embodiment, it will be appreciated by the skilled addressee that the application have it is many deformation and
Change is without departing from spirit herein, it is desirable to which appended claim includes these deformations and changed without departing from the application's
Spirit.
Claims (12)
1. a kind of method for building up of non-relational database index, it is characterised in that methods described includes：
Obtain pending business datum, reading attributes field configuration table, the attribute field allocation list include previously according to
The index that the index operation demand of business datum is set sets up the service fields of dimension；
Determine to need to set up in the business datum main business field indexed according to the attribute field allocation list；
The field properties data associated according to the main business field and with the main business field are set up with the main business word
Section indexes for the non-relational database of index key.
2. the method for building up of a kind of non-relational database index as claimed in claim 1, it is characterised in that methods described is also
Including：
The task list set up in advance for pending business datum is checked according to the timed task of setting, is judged according to testing result
Whether the storage state of the business datum keeps one with the storage state of the corresponding index data in non-relational database
Cause；The task creation whether expired time that indexes that the task list is configured at least include identifying the business datum believes
The storage state information of breath, the business datum and corresponding index data whether normal storage；
If consistent, job status of the pending business datum in the task list is changed accordingly.
3. a kind of method for building up of non-relational database index as claimed in claim 2, it is characterised in that
The task list is including task creation time, tasks carrying according to the business datum using the setting of MYSQL language
Time, task status mark, the data message monitoring table of task expired time, task completion time.
4. the method for building up of a kind of non-relational database index as claimed in claim 3, it is characterised in that methods described is also
Including：
When monitoring that the business datum stores the index datastore exception of abnormal or described business datum, mark and store
The abnormal business datum of the processing or index data, and update corresponding task status mark in the task list.
5. the method for building up of a kind of non-relational database index as claimed in claim 1, it is characterised in that described to set in advance
The attribute field allocation list put includes：
Need to build generation using scheme language previously according to the business operation of the business datum.
6. a kind of method for building up of non-relational database index as described in any one in claim 1 to 5, its feature exists
In methods described also includes：
Identical row major key carries out the polymerization of predetermined dimension in non-relational database to depositing business datum, forms polymerization
Non-relational database afterwards.
7. the method for building up of a kind of non-relational database index as claimed in claim 6, it is characterised in that to depositing industry
The polymerization of the predetermined dimension of identical row major key progress includes in the non-relational database for data of being engaged in：
The polymerization time section that row major keys different in non-relational database are set is set；
In the non-relational database, data message of the mutually colleague's major key in polymerization time section is polymerized to same
The service data information of individual row major key, the service data information of the same row major key is configured to exist comprising the row major key
The service data information of different time version in the polymerization time section.
What 8. a kind of non-relational database was indexed sets up device, it is characterised in that described device includes：
Data read module, the pending business datum for obtaining, reading attributes field configuration table, the attribute field configuration
Table includes setting up the service fields of dimension previously according to the index of the index operation demand setting of business datum；
Index field determining module, for determining to need to set up in the business datum according to the attribute field allocation list to index
Main business field；
Index construct module, for being built according to the main business field and with the field properties data that the main business field is associated
The vertical non-relational database index using the main business field as index key.
What 9. a kind of non-relational database as claimed in claim 8 was indexed sets up device, it is characterised in that described device is also
Including：
Index ensures processing module, for being checked in advance as appointing that pending business datum is set up according to the timed task of setting
Business table, the storage state and the corresponding index data in non-relational database of the business datum are judged according to testing result
Storage state whether be consistent；If consistent, the pending business datum is changed accordingly in the task list
Job status；The task list be configured at least to include the index task creation for identifying the business datum whether mistake
The storage state information of the temporal information of phase, the business datum and corresponding index data whether normal storage.
What 10. a kind of non-relational database as claimed in claim 9 was indexed sets up device, it is characterised in that the index
Ensure that the task list in processing module includes：
Task creation time, task execution time, task status are included using what MYSQL language was set according to the business datum
The data message monitoring table of mark, task expired time, task completion time.
What 11. a kind of non-relational database as claimed in claim 10 was indexed sets up device, it is characterised in that the index
Ensure that processing module is additionally operable to：
When monitoring that the business datum stores the index datastore exception of abnormal or described business datum, mark and store
The abnormal business datum of the processing or index data, and update corresponding task status mark in the task list.
What 12. a kind of non-relational database as claimed in claim 8 was indexed sets up device, it is characterised in that described advance
The attribute field allocation list of setting includes needing to build using scheme language previously according to the business operation of the business datum
Generation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611019641.7A CN107016025A (en) | 2016-11-17 | 2016-11-17 | A kind of method for building up and device of non-relational database index |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611019641.7A CN107016025A (en) | 2016-11-17 | 2016-11-17 | A kind of method for building up and device of non-relational database index |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107016025A true CN107016025A (en) | 2017-08-04 |
Family
ID=59438901
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611019641.7A Pending CN107016025A (en) | 2016-11-17 | 2016-11-17 | A kind of method for building up and device of non-relational database index |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107016025A (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108959640A (en) * | 2018-07-26 | 2018-12-07 | 浙江数链科技有限公司 | ES index fast construction method and device |
| CN108959501A (en) * | 2018-06-26 | 2018-12-07 | 新华三大数据技术有限公司 | Delete the method and device of ES index |
| CN109788031A (en) * | 2018-12-17 | 2019-05-21 | 平安普惠企业管理有限公司 | Business datum acquisition methods, device, computer equipment and storage medium |
| CN110019190A (en) * | 2017-09-21 | 2019-07-16 | 阿里巴巴集团控股有限公司 | The method and apparatus for creating index |
| CN110196851A (en) * | 2019-05-09 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN110457310A (en) * | 2019-07-03 | 2019-11-15 | 阿里巴巴集团控股有限公司 | Purpose data classifying method and device |
| CN110489418A (en) * | 2019-07-26 | 2019-11-22 | 阿里巴巴集团控股有限公司 | A kind of data aggregation method and system |
| CN110704686A (en) * | 2019-10-08 | 2020-01-17 | 北京锐安科技有限公司 | Quality detection method and device for semi-structured data, storage medium and equipment |
| CN110895549A (en) * | 2019-09-04 | 2020-03-20 | 成都四方伟业软件股份有限公司 | Quantized data retrieval method and system |
| CN111104369A (en) * | 2019-12-16 | 2020-05-05 | 北京明略软件系统有限公司 | Retrieval database construction method and device |
| CN111723245A (en) * | 2019-03-18 | 2020-09-29 | 阿里巴巴集团控股有限公司 | Method for establishing incidence relation of different types of storage objects in data storage system |
| CN111813772A (en) * | 2019-04-12 | 2020-10-23 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN112181993A (en) * | 2020-10-27 | 2021-01-05 | 广州市网星信息技术有限公司 | Service data query method, device, server and storage medium |
| CN112214571A (en) * | 2020-10-10 | 2021-01-12 | 中国平安人寿保险股份有限公司 | KV storage-based index generation method, device, equipment and medium |
| CN112818017A (en) * | 2021-01-22 | 2021-05-18 | 百果园技术(新加坡)有限公司 | Event data processing method and device |
| CN113468199A (en) * | 2021-07-29 | 2021-10-01 | 上海哔哩哔哩科技有限公司 | Index updating method and system |
| CN113742535A (en) * | 2021-09-03 | 2021-12-03 | 上海微盟企业发展有限公司 | Generalized retrieval method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103440302A (en) * | 2013-08-21 | 2013-12-11 | 广东电网公司电力调度控制中心 | Real-time data exchange method and system |
| CN104731922A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | System and method for rapidly retrieving structural data based on distributed type database HBase |
| CN105426459A (en) * | 2015-11-13 | 2016-03-23 | 福建农林大学 | Unirecord index retrieval mode of distributed data of the Internet of things based on NoSQL database |
| CN105426421A (en) * | 2015-11-03 | 2016-03-23 | 武汉地大信息工程股份有限公司 | Tense monitoring data quick visualization method and system |
| CN105488043A (en) * | 2014-09-15 | 2016-04-13 | 南京理工大学 | Data query method and system based on Key-Value data blocks |
-
2016
- 2016-11-17 CN CN201611019641.7A patent/CN107016025A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103440302A (en) * | 2013-08-21 | 2013-12-11 | 广东电网公司电力调度控制中心 | Real-time data exchange method and system |
| CN105488043A (en) * | 2014-09-15 | 2016-04-13 | 南京理工大学 | Data query method and system based on Key-Value data blocks |
| CN104731922A (en) * | 2015-03-26 | 2015-06-24 | 江苏物联网研究发展中心 | System and method for rapidly retrieving structural data based on distributed type database HBase |
| CN105426421A (en) * | 2015-11-03 | 2016-03-23 | 武汉地大信息工程股份有限公司 | Tense monitoring data quick visualization method and system |
| CN105426459A (en) * | 2015-11-13 | 2016-03-23 | 福建农林大学 | Unirecord index retrieval mode of distributed data of the Internet of things based on NoSQL database |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019190A (en) * | 2017-09-21 | 2019-07-16 | 阿里巴巴集团控股有限公司 | The method and apparatus for creating index |
| CN108959501A (en) * | 2018-06-26 | 2018-12-07 | 新华三大数据技术有限公司 | Delete the method and device of ES index |
| CN108959501B (en) * | 2018-06-26 | 2021-09-17 | 新华三大数据技术有限公司 | Method and device for deleting ES index |
| CN108959640B (en) * | 2018-07-26 | 2021-02-12 | 浙江数链科技有限公司 | ES index rapid construction method and device |
| CN108959640A (en) * | 2018-07-26 | 2018-12-07 | 浙江数链科技有限公司 | ES index fast construction method and device |
| CN109788031A (en) * | 2018-12-17 | 2019-05-21 | 平安普惠企业管理有限公司 | Business datum acquisition methods, device, computer equipment and storage medium |
| CN109788031B (en) * | 2018-12-17 | 2022-10-11 | 平安普惠企业管理有限公司 | Service data acquisition method and device, computer equipment and storage medium |
| CN111723245B (en) * | 2019-03-18 | 2024-04-26 | 阿里巴巴集团控股有限公司 | Method for establishing association relation of different types of storage objects in data storage system |
| CN111723245A (en) * | 2019-03-18 | 2020-09-29 | 阿里巴巴集团控股有限公司 | Method for establishing incidence relation of different types of storage objects in data storage system |
| CN111813772A (en) * | 2019-04-12 | 2020-10-23 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN111813772B (en) * | 2019-04-12 | 2024-05-07 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN110196851B (en) * | 2019-05-09 | 2024-05-10 | 腾讯科技（深圳）有限公司 | Data storage method, device, equipment and storage medium |
| CN110196851A (en) * | 2019-05-09 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of date storage method, device, equipment and storage medium |
| CN110457310A (en) * | 2019-07-03 | 2019-11-15 | 阿里巴巴集团控股有限公司 | Purpose data classifying method and device |
| CN110457310B (en) * | 2019-07-03 | 2023-05-23 | 创新先进技术有限公司 | Data collection method and device |
| CN110489418A (en) * | 2019-07-26 | 2019-11-22 | 阿里巴巴集团控股有限公司 | A kind of data aggregation method and system |
| CN110489418B (en) * | 2019-07-26 | 2023-09-22 | 创新先进技术有限公司 | Data aggregation method and system |
| CN110895549A (en) * | 2019-09-04 | 2020-03-20 | 成都四方伟业软件股份有限公司 | Quantized data retrieval method and system |
| CN110895549B (en) * | 2019-09-04 | 2022-12-06 | 成都四方伟业软件股份有限公司 | Quantized data retrieval method and system |
| CN110704686A (en) * | 2019-10-08 | 2020-01-17 | 北京锐安科技有限公司 | Quality detection method and device for semi-structured data, storage medium and equipment |
| CN111104369A (en) * | 2019-12-16 | 2020-05-05 | 北京明略软件系统有限公司 | Retrieval database construction method and device |
| CN112214571A (en) * | 2020-10-10 | 2021-01-12 | 中国平安人寿保险股份有限公司 | KV storage-based index generation method, device, equipment and medium |
| CN112214571B (en) * | 2020-10-10 | 2024-02-20 | 中国平安人寿保险股份有限公司 | Index generation method, device, equipment and medium based on KV storage |
| CN112181993A (en) * | 2020-10-27 | 2021-01-05 | 广州市网星信息技术有限公司 | Service data query method, device, server and storage medium |
| CN112818017A (en) * | 2021-01-22 | 2021-05-18 | 百果园技术(新加坡)有限公司 | Event data processing method and device |
| CN113468199A (en) * | 2021-07-29 | 2021-10-01 | 上海哔哩哔哩科技有限公司 | Index updating method and system |
| CN113742535A (en) * | 2021-09-03 | 2021-12-03 | 上海微盟企业发展有限公司 | Generalized retrieval method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107016025A (en) | A kind of method for building up and device of non-relational database index | |
| US11360950B2 (en) | System for analysing data relationships to support data query execution | |
| US6199059B1 (en) | System and method for classifying and retrieving information with virtual object hierarchy | |
| CN104756106B (en) | Data source in characterize data storage system | |
| CN104050223B (en) | Pivot face for text mining and search | |
| EP3513315A1 (en) | System for data management in a large scale data repository | |
| EP3513313A1 (en) | System for importing data into a data repository | |
| CN107810500A (en) | Data quality analysis | |
| US20050060342A1 (en) | Holistic dynamic information management platform for end-users to interact with and share all information categories, including data, functions, and results, in collaborative secure venue | |
| CN109344153A (en) | The processing method and terminal device of business datum | |
| CN103562905B (en) | Improved data visualization configuration system and method | |
| CN108304522A (en) | Comparison method, device and the terminal device of difference between a kind of database | |
| CA2804441C (en) | Database performance analysis | |
| CN108334321A (en) | The presentation device of program code, method and device | |
| CN109491857A (en) | A kind of data monitoring method, system and the terminal device of rule-based engine | |
| US9727663B2 (en) | Data store query prediction | |
| CN108664509A (en) | A kind of method, apparatus and server of extemporaneous inquiry | |
| US7526473B2 (en) | Traversing a relational model with preferred paths | |
| CN111414410A (en) | Data processing method, device, equipment and storage medium | |
| US7827153B2 (en) | System and method to perform bulk operation database cleanup | |
| Dakrory et al. | Automated ETL testing on the data quality of a data warehouse | |
| CN111984882A (en) | Data processing method, system and equipment | |
| CN107577802A (en) | A kind of data base management method and device | |
| JP6267398B2 (en) | Service design support system and service design support method | |
| US20150363711A1 (en) | Device for rapid operational visibility and analytics automation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200925 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200925 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: Greater Cayman, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170804 |
|
| RJ01 | Rejection of invention patent application after publication |