CN110276002A

CN110276002A - Search for application data processing method, device, computer equipment and storage medium

Info

Publication number: CN110276002A
Application number: CN201910562858.XA
Authority: CN
Inventors: 梁新荣; 沈飞龙; 段曙亮
Original assignee: Zhejiang Dasou Vehicle Software Technology Co Ltd
Current assignee: Zhejiang Dasou Vehicle Software Technology Co Ltd
Priority date: 2019-06-26
Filing date: 2019-06-26
Publication date: 2019-09-24
Anticipated expiration: 2039-06-26
Also published as: CN110276002B

Abstract

This application involves data retrieval technology field, in particular to a kind of search application data processing method, device, computer equipment and storage medium.The described method includes: being monitored to database journal, when monitoring that newly-increased data flow is inserted into database journal, first database mark and the first tables of data mark are read from newly-increased data flow；It searches first database mark and the first tables of data identifies corresponding first metadata structure, and obtain the first application identities of the first metadata structure；Processing is carried out to the data field in newly-increased data flow according to the first metadata structure and generates the first field of application, search application record is generated according to the first field of application；Search is added in search storage engines using record according to the first application identities.It can be improved using this method and improve data query performance.

Description

Search for application data processing method, device, computer equipment and storage medium

Technical field

This application involves field of computer technology, more particularly to a kind of search application data processing method, device, calculating Machine equipment and storage medium.

Background technique

Internet company's product renewing is very fast, and each business side of product is in the business procession of product, per average daily The meeting a large amount of product business datum of output, therefore, most of product requires inquiry database or other data retrieval functions Support.

Existing data search engine is mostly based on relevant database, and various businesses are stored in relevant database High quantity grade data, however the variation of Internet company's business demand is greatly, and different business sides is with different data queries Demand can also establish index on relational database according to demand sometimes, and multiple business sides establish excessively on relational database When index, due to will can not be effectively layered between business datum, the reduction of data query performance, Wu Faman will cause instead The data query requirements of each business side of foot.

Summary of the invention

Based on this, it is necessary to which in view of the above technical problems, providing one kind can be improved data query performance and meet multi-party Search application data processing method, device, computer equipment and the storage medium of data query requirements.

A kind of search application data processing method, which comprises

Database journal is monitored, when monitoring that newly-increased data flow is inserted into the database journal, from newly-increased First database mark and the first tables of data mark are read in data flow；

It searches the first database mark and first tables of data identifies corresponding first metadata structure, and obtain First application identities of first metadata structure；

Processing is carried out to the data field in the newly-increased data flow according to first metadata structure and generates application member Field generates search application record according to the first field of the application；

Described search is added in search storage engines using record according to first application identities.

In one of the embodiments, it is described according to first metadata structure to the data in the newly-increased data flow Field carries out processing and generates the first field of application, comprising:

The corresponding type of service of first metadata structure is obtained, search and enables the corresponding data of the type of service Handle thread；

In the data processing threads, the default processing logic of each data element in first metadata structure is obtained； The corresponding data field of each data element is searched from the newly-increased data flow, according to the default processing logic to the number It is handled and is generated using first field according to field.

After obtaining the corresponding type of service of the first application metadata structure in one of the embodiments, also wrap It includes:

By the newly-increased data flow cache into message-oriented middleware buffer zone corresponding with the type of service；

It is described before searching the corresponding data field of each data element in the newly-increased data flow, comprising:

The newly-increased data flow is read from the corresponding buffer zone of the type of service of the message-oriented middleware.

Method in one of the embodiments, further include:

It receives the tables of data that terminal is sent and chooses instruction, the tables of data, which is chosen, carries the second application identities, the in instruction Two Database Identifications and the second tables of data mark；

Second tables of data is searched from the corresponding database of second Database Identification identifies corresponding tables of data, It obtains the table structure information of the tables of data and returns to the terminal；

The data metamessage that terminal is sent is received, the data metamessage is to carry out logical process to the table structure information What editor was formed, the second metadata structure is generated according to the data metamessage and second application identities；

Second metadata structure and second Database Identification and second tables of data mark are associated Storage.

Method in one of the embodiments, further include:

The field information and index information that each data element is parsed from the data metamessage, according to the field information It generates and applies data template；

In described search storage engines, created according to the application data template corresponding with second application identities Application data sheet；

It is application data sheet creation index according to the index information.

Described search search is added to using record according to first application identities in one of the embodiments, to deposit It stores up in engine, comprising:

The corresponding application data sheet of first application identities is searched from described search storage engines；

The first field of each application in described search application record is added in the application data sheet found In the respective column of new record row.

Method in one of the embodiments, further include:

The data query instruction that terminal is sent is received, target application mark is carried in the data query instruction and retrieval is patrolled The formula of collecting；

The target application is searched from described search storage engines identifies corresponding application data sheet；

According to the retrieval logical formula, corresponding application data sheet progress data retrieval is identified to the target application and is obtained The target data is returned to the terminal by target data.

A kind of search for applies data processing equipment, and described device includes:

Log monitoring module, for being monitored to database journal, when monitoring that newly-increased data flow is inserted into the number When according to library log, first database mark and the first tables of data mark are read from newly-increased data flow；

Meta structure searching module, for searching first database mark and first tables of data mark corresponding the One metadata structure, and obtain the first application identities of first metadata structure；

Data processing module, for according to first metadata structure to the data field in the newly-increased data flow into Row processing generates the first field of application, generates search application record according to the first field of the application；

Engine data adding module, for described search to be added to search using record according to first application identities In storage engines.

Data processing module includes: in one of the embodiments,

Thread searching unit, it is described for obtaining the corresponding type of service of first metadata structure, searching and enabling The corresponding data processing threads of type of service；

Field processing unit, for obtaining each data in first metadata structure in the data processing threads The default processing logic of member；The corresponding data field of each data element is searched from the newly-increased data flow, according to described pre- If processing logic handles the data field and is generated using first field.

Device in one of the embodiments, further include:

Data cache module, for the newly-increased data flow cache is corresponding with the type of service into message-oriented middleware Buffer zone；

Read module is cached, for reading institute from the corresponding buffer zone of the type of service of the message-oriented middleware State newly-increased data flow.

Device in one of the embodiments, further include:

Command reception module is chosen, the tables of data for receiving terminal transmission chooses instruction, and the tables of data chooses instruction The second application identities of middle carrying, the second Database Identification and the second tables of data mark；

Table structure return module, for from the corresponding database of second Database Identification, searching second number Corresponding tables of data is identified according to table, the table structure information of the tables of data is obtained and returns to the terminal；

Result-generation module, the data metamessage sent for receiving the terminal, the data metamessage are to described Table structure information carries out logical process editor formation, generates second according to the data metamessage and second application identities Metadata structure；

Structure memory module, for counting second metadata structure and second Database Identification and described second Storage is associated according to table mark.

Device in one of the embodiments, further include:

Template generation module, for parsing the field information and index letter of each data element from the data metamessage Breath generates according to the field information and applies data template；

Tables of data creation module is used in described search storage engines, according to application data template creation and institute State the corresponding application data sheet of the second application identities；

Index creation module, for being application data sheet creation index according to the index information.

Engine data adding module includes: in one of the embodiments,

Tables of data searching unit, for searching the corresponding application of first application identities from described search storage engines Tables of data；

Field record unit, for being added to the first field of each application in described search application record and finding The application data sheet in new record row respective column in.

Device in one of the embodiments, further include:

Inquiry instruction receiving module is taken in the data query instruction for receiving the data query instruction of terminal transmission Band target application identifies and retrieves logical formula；

Tables of data search module identifies corresponding application for searching the target application from described search storage engines Tables of data；

Data retrieval module, for being identified to the target application corresponding using data according to the retrieval logical formula Table carries out data retrieval and obtains target data, and the target data is returned to the terminal.

A kind of computer equipment, including memory and processor, the memory are stored with computer program, the processing The step of device realizes the above method when executing the computer program.

A kind of computer readable storage medium, is stored thereon with computer program, and the computer program is held by processor The step of above method is realized when row.

Above-mentioned search application data processing method, device, computer equipment and storage medium, supervise database journal Control reads the data table information and correspondence that newly-increased data flow to be inserted into when being tested in newly-increased data stream into data library Search application application identities, and corresponding with data table information metadata structure is searched, by newly-increased data flow according to first number Field processing is carried out according to structure, the search application record for meeting search application demand is generated, the newly-increased search that processing is completed is answered It is added in search storage engines with record foundation application identities, since the search application record basis in search storage engines is answered It is divided with mark, is drawn so that business side when carrying out data search, can search for application locking search storage by selection Corresponding target search application record in holding up, then scanned for based on target search using record, it realizes between business datum Isolation improves data search efficiency.

Detailed description of the invention

Fig. 1 is the application scenario diagram that application data processing method is searched in one embodiment；

Fig. 2 is the flow diagram that application data processing method is searched in one embodiment；

Fig. 3 is the data relationship mapping graph in one embodiment；

Fig. 4 is in one embodiment using the flow diagram of the foundation step of metadata structure；

Fig. 5 is the flow diagram of field processing step in one embodiment；

Fig. 6 is the structural block diagram that data processing equipment is applied in search in one embodiment；

Fig. 7 is the internal structure chart of computer equipment in one embodiment.

Specific embodiment

It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, not For limiting the application.

It is appreciated that term " first " used in this application, " second " etc. can be used to describe various elements herein, But unless stated otherwise, these elements should not be limited by these terms.These terms are only used to by first element and another yuan Part is distinguished.For example, in the case where not departing from scope of the present application, first server can be known as second server, And similarly, second server can be known as first server.

Search application data processing method provided by the present application, can be applied in application environment as shown in Figure 1.Its In, service terminal 102 is communicated by network with server 104.User can carry out newly-increased search on service terminal 102 The visualization of the metadata structure of application constructs operation, and specifically, user can choose existing tables of data, and select tables of data In field, to the data format of field, incidence relation, whether establish index etc. attributes edit, generated after edit validation Newly-increased metadata structure is sent to server 104 by newly-increased metadata structure, service terminal 102, and server 104 is by newly-increased first number Storage is associated according to the tables of data of structure and the tables of data of selection mark and affiliated wide area information server mark.Server The newly-increased data flow being newly inserted into 104 pairs of databases is monitored, and when monitoring newly-increased data flow, is read from newly-increased data flow The corresponding first database mark of the tables of data of write-in and the first tables of data mark are taken, first database mark and the first number are searched The first metadata structure of corresponding search application is identified according to table, and obtains the first of the corresponding search application of the first metadata structure Application identities, server 104 carry out processing to the data field in newly-increased data flow according to the first metadata structure and generate application First field generates search application record according to the first field of application, and is added to search using record according to the first application identities It searches in storage engines.

Wherein service terminal 102 can be, but not limited to be various personal computers, laptop, smart phone, plate Computer and portable wearable device, server 104 can use the service of the either multiple server compositions of independent server Device cluster is realized.When server 104 is independent server, multiple databases, every number can be disposed in server 104 According to the business datum table that can store specific transactions in library；When server 104 is the server cluster of multiple servers composition, It can store the business datum table of specific transactions in the database disposed in each server.

In one embodiment, it as shown in Fig. 2, providing a kind of search application data processing method, applies in this way It is illustrated for server 104 in Fig. 1, in other embodiments, this method also can be applied to terminal, and method includes Following steps:

Step 210, database journal is monitored, when monitoring that newly-increased data flow is inserted into database journal, from First database mark and the first tables of data mark are read in newly-increased data flow.

Whether server is monitored each wide area information server log of deployment, be inserted into the log of monitoring data library New data flow.It, can be by corresponding operation with number when the operation such as having increasing, deleting, change to the data in the tables of data in database It is recorded in database journal according to the form of stream, server monitors the operation behavior in database.As when in tables of data In when increasing data newly, can be instructed by insert and data flow be inserted into database journal, the data flow being inserted into database In include the data table information of operation, database information and specific data content belonging to tables of data.

When monitoring server inserts newly-increased data flow into database journal, server is read from newly-increased data flow First database mark and the first tables of data mark.Wherein, multiple databases may be deployed in server, in each database It can be stored with multiple tables of data again, these data surfaces can be stored in data to multiple business sides, the data of multiple business sides In library, the data stored in each tables of data can be classified with type of service.Tables of data mark for every tables of data into Row unique identification, the database that Database Identification is used to store tables of data carry out unique identification.First in the present embodiment Tables of data is identified as the mark of the tables of data of newly-increased data insertion, and the tables of data that first database is identified as newly-increased data is stored Database mark.

Step 220, it searches first database mark and the first tables of data identifies corresponding first metadata structure, and obtain First application identities of the first metadata structure.

The user of each business side carries out data search in order to facilitate oneself, creation search can apply according to their own needs, The metadata structure that search application needs is set.The data in tables of data involved in search application are set in metadata structure Field, the data format of each data field, field whether add index and with the incidence relation of other fields, field arithmetic logic Etc. information.The identification information of tables of data cited in each data field is associated storage in metadata structure and structure.Search Can be to be multiple using cited tables of data, therefore, tables of data corresponding to metadata structure may be multiple.In addition, Cited in one tables of data can also be applied by the search of multiple business sides.The data applied, between tables of data are searched for by business side Relationship map figure is referring to figure 3..One business side can set multiple search applications, and business side 1 sets search and answers in Fig. 3 B is applied with a and search, the quantity of the cited tables of data of each search application can be one or more, and search is quoted using a Two tables of data, including tables of data a and tables of data b, search refer to a tables of data c using b.

Server searches first database mark and the first tables of data identifies corresponding first metadata structure, and server is looked into The quantity of the first metadata structure found can be multiple.Metadata structure is identified with the application identities for searching for application, Server obtains the application identities of the first metadata structure.

Step 230, processing is carried out to the data field in newly-increased data flow according to the first metadata structure and generates application member Field generates search application record according to the first field of application.

Server obtains the data format of each data field in the first metadata structure, field arithmetic logic, if at this In field establish index etc. field informations, and from read out in newly-increased data flow write-in tables of data in data flow field, according to Field information carries out processing to corresponding data flow field and generates the first field of application, is such as adjusted data format, according to word Field values progress operation is generated new field, establishes index etc. to some data flow fields by section arithmetic logic.

The first field of the application generated after processing arrange according to reference format by server generates search application record.Such as The field name of each data field can be standardized, ultimately generate standardized search application record.

Step 240, search is added in search storage engines using record according to the first application identities.

Search engine is for handling the data search request of user, and search storage engines are for making search engine Search data are stored.The data searched in storage engines carry out classification storage, service according to affiliated search application Device searches data storage areas corresponding with the first application identities in search storage engines, applies record to be added to search corresponding Data storage areas in.For the user of business side when carrying out data retrieval inquiry, can choose needs search application to be used And retrieval type, search engine carries out data retrieval from the corresponding data of search application according to retrieval type, so as to retrieve The range of data effectively reduces, and improves data query recall precision.

In the present embodiment, server is monitored database journal, when being tested with newly-increased data stream into data When in library, the application identities of the data table information that newly-increased data flow to be inserted into and corresponding search application, and lookup and number are read According to the corresponding metadata structure of table information, newly-increased data flow is subjected to field processing according to metadata structure, generation meets search The newly-increased search application record that processing is completed is added to search storage according to application identities by the search application record of application demand In engine, since the search application record in search storage engines is divided according to application identities, so that business side is carrying out When data search, corresponding target search application record in application locking search storage engines, then base can be searched for by selection It is scanned in target search using record, realizes the isolation between business datum, improve data search efficiency.

In one embodiment, processing generation is carried out to the data field in newly-increased data flow according to the first metadata structure It may include: to obtain the corresponding type of service of the first metadata structure using the step of first field, search and enable type of service Corresponding data processing threads；In data processing threads, the default processing for obtaining each data element in the first metadata structure is patrolled Volume；The corresponding data field of each data element is searched from newly-increased data flow, according to default processing logic to data field at It manages and generates using first field.

In the present embodiment, the multiple data processing threads of server settings, multiple data processing threads are according to be processed Type of service belonging to data is divided.Such as type of service may include searching for private business, service for corporate customers in building In application, the type of service of the corresponding inquiry data of search application is also added in metadata structure.

Server extracts service type information from the first metadata structure, and service type information can be type of service Mark, Business Name etc..Server searches data processing threads corresponding with type of service, and notifies the data processing threads pair Data field in newly-increased data flow carries out data processing.

In one embodiment, newly-increased data flow can be directly distributed in data processing threads by server, at data After lineation journey receives newly-increased data flow, the default processing logic of each data element, data element are obtained from the first metadata structure The texture field of corresponding application structure as before the first field processing of application.Data processing threads are searched from newly-increased data flow The corresponding data field of each data element handles data field according to default processing logic and is generated using first field, in advance If handling logic can be the logics such as field format processing, field value operation described in a upper embodiment.

It in the above-described embodiments, can will be different by carrying out multiple threads according to type of service to newly-increased data flow The data flow of type of service is isolated, and realizes the parallel processing of multi-service data, improves data-handling efficiency.

In one embodiment, server the application metadata structure corresponding type of service of acquisition first the step of it Afterwards, can by newly-increased data flow cache into message-oriented middleware buffer zone corresponding with type of service, data processing threads from The step of before searching the corresponding data field of each data element in newly-increased data flow may include: the service class from message-oriented middleware Newly-increased data flow is read in the corresponding buffer zone of type.

In the present embodiment, server is not that directly newly-increased data flow is distributed at corresponding data processing threads Reason, but first all newly-increased data flows are delayed and are deposited into message-oriented middleware, it reinforms corresponding data processing threads and carries out Processing.There are multiple buffer zones in message-oriented middleware, buffer zone is divided according to type of service, and server is among message Buffer zone corresponding with type of service is searched in part, and newly-increased data flow is stored into the buffer zone.Server notice Data processing threads read out newly-increased data from the corresponding buffer zone of type of service when handling newly-increased data flow Stream carries out data processing again.Further, in the message between read flag is carried out to the newly-increased data flow being read in part, and Periodically the newly-increased data flow read in each buffer zone is cleared up.By by newly-increased data flow cache to message-oriented middleware In, data redundancy can be reduced to avoid excessive data table data is backed up in the server.

In one embodiment, as shown in figure 4, search application data processing method can also include applying metadata below The foundation step of structure:

Step 202: receiving the tables of data that terminal is sent and choose instruction, tables of data, which is chosen, carries the second application mark in instruction Know, the second Database Identification and the second tables of data identify.

Business side user can carry out newly-increased using metadata knot when there is new search application demand on service terminal The creation of structure operates.User can set the application identities such as Apply Names of search application, affiliated type of service at the terminal Equal basic informations, then selection search is using the tables of data in the database of required reference, after user-selected number is according to library, terminal exhibition Show the tables of data in the database, user can select tables of data, and after being elected to middle tables of data, terminal is according to the number chosen According to the tables of data mark of table, Database Identification belonging to tables of data and the generation tables of data selection of the application identities of setting is chosen to refer to It enables, and is sent to server.

For example, the title that user sets search application can be for " shop, the affiliated type of service of search application such as borrow private Money etc., user can choose " centralized " database, the tables of data stored in the terminal display database, such as tables of data It may include baseshop, shop_extra, shop_auth etc., user selects tables of data.

After the tables of data selection instruction that server receiving terminal is sent, user's setting is read out from tables of data selection instruction Search application the second application identities, the second data of the tables of data of the second Database Identification and selection of the database of selection Table mark.

Step 204: the second tables of data is searched from the corresponding database of the second Database Identification identifies corresponding tables of data, It obtains the table structure information of tables of data and returns to terminal.

Server finds database corresponding with the second Database Identification, and the search and the from the database found Two tables of data identify corresponding tables of data, obtain the table structure information of the tables of data, may include tables of data in table structure information In each field field name and field informations, the table structure information that server will acquire such as data format be back to terminal.

Step 206: receiving the data metamessage that terminal is sent, the data metamessage is to carry out to the table structure information What logical process editor was formed, the second metadata structure is generated according to data metamessage and the second application identities.

After terminal receives table structure information, table structure information is visualized, it such as can be in the form of a list Field name, field data format are listed, and provides edit operation option for each field.Wherein, edit operation type selecting can be with Including deletion, field type editor, the outer link of addition, setting and the incidence relation of field, setting segmenter, setting in other tables The option of operation such as index.Wherein, segmenter is that preset field values handle logic, and processing logic is carried out code envelope Dress, enables general for each field.For example, field values processing logic can be to be converted to phonetic for Chinese character, by field Number in numerical value such as separates at the logics with Chinese character.Index is set to be this in the tables of data generated later according to metadata structure Field addition index.

User carries out edit operation to the table structure information of displaying, and after editor completes and confirms, it is final that terminal obtains user Editor's information, that is, data metamessage of field in the table structure of selection, user select each field of editor to be known as a data Member.Data metamessage is carried out assembling and generates the second metadata structure by server, and according to the second application identities to binary number It is identified according to structure.

Step 208: the second metadata structure and the second Database Identification and the second tables of data mark are associated storage.

Server when being stored to the second metadata structure, by the tables of data of reference it is corresponding second number libraries mark and Second tables of data mark is associated with the second data structure, such as can recorde the second application identities and the second Database Identification and Mapping relations between second tables of data mark.To find and quote the number when there is new data that tables of data is written According to the search application of table, to be updated to the search using corresponding search storage engines.

In one embodiment, server is by the second metadata structure and the second Database Identification and the second tables of data mark After knowledge the step of being associated storage, can also include: parsed from data metamessage each data element field information and Index information generates according to field information and applies data template；Search storage engines according to application data template creation with The corresponding application data sheet of second application identities；It is application data sheet creation index according to index information.

Server parses the field information and index letter of each data element from the data metamessage of the second metadata structure Breath, whether it is the information such as major key that field information may include field name, field type such as integer, numeric type etc., field, index Whether information is applied to be that the data element establishes index in addition, server is created according to the field information of each data element of acquisition Data template, using data template be data element field information set, such as can in the form of key-value character string into Row storage.Server is according to the field structure using the information setting application data sheet of each field in data template, using data Table can be the file of the formats such as excel, csv, if server uses elasticsearch search engine, application data sheet can Think the tables of data of elasticsearch file format, and is corresponding in application data sheet according to the index information of each data element Field where column be arranged index.Server is identified according to application data sheet of second application identities to generation.

In one embodiment, the step being added to search using record according to the first application identities in search storage engines It suddenly may include: to search the corresponding application data sheet of the first application identities from search storage engines；It will be in search application record Each first field of application, be added in the respective column of record row new in the application data sheet found.

When server monitoring to the corresponding newly-increased data flow of the search application with the first application identities, and to newly-increased data After stream process generates search application record, server searches application number corresponding with the first application identities from search storage engines According to table, new record row is created in application data sheet, will in search application record respectively the title of the first field of application with apply number It is matched according to the field name in table, each field values for applying first field is added to the phase of matching field in new record row Ying Liezhong, to generate new data record in application data sheet.

In one embodiment, user can also at the terminal repair the metadata structure of the search application created The operation such as change, delete.When user selects the search created in application, terminal is by the metadata knot of the application at the terminal Structure carry out visualization display, user can to the search apply in data element carry out edit-modify, including to field type, be The attributes such as no addition index, segmenter change are modified.After the completion of user confirms modification, terminal obtains modified data Metamessage generates modified metadata structure and returns to server.Server is replaced after receiving modified metadata structure Fall search and apply original metadata structure, and to the application data template created and is answered according to modified metadata structure It is modified accordingly with tables of data.

When user selects to delete some search in application, terminal obtains the application mark that user selects the search application deleted Know, and send the search application deletion instruction for carrying the application identities to server, server is extracted from search application instruction The application identities of search application to be deleted out, by the corresponding metadata structure of the application identities, using data template and using number It is deleted according to table.

In one embodiment, above-mentioned search application data processing method can be the following steps are included: receive terminal hair The data query sent instructs, and target application mark and retrieval logical formula are carried in data query instruction；From search storage engines It searches target application and identifies corresponding application data sheet；Corresponding application data sheet is identified to target application according to retrieval logical formula It carries out data retrieval and obtains target data, the processing such as field sequence is carried out according to preset search and output format to target data, By treated, target data returns to terminal.

When user needs to carry out data query, can select that target to be used is needed to search in the data retrieval page of terminal The target application of Suo Yingyong identifies, and inputs retrieval logical formula, and retrieval logical formula can be simple critical field, or Critical field and the combination of search condition etc..When user confirm retrieve when, terminal obtain user selection target application mark and The retrieval logical formula of input, and identify and retrieve logical formula generation data query instruction according to target application and be sent to server.

Server receiving terminal send data query instruction, and from data query instruction in extract target application mark and Logical formula is retrieved, server searches application data sheet corresponding with target application mark from search storage engines, further according to inspection Data in rope logical formula application data table carry out retrieval and inquisition, obtain the target data for meeting retrieval logical formula, will inquire To target data return to terminal.The business datum of magnanimity can be contracted to the number of application data sheet by building search application According to amount, so as to reach the raising of search speed magnitude.

In one embodiment, as shown in figure 5, according to the first metadata structure to the data field in newly-increased data flow into Row processing is generated may include: using the field processing step of first field

Step 232, tables of data hierarchical information is extracted from the first metadata structure, is judged according to tables of data hierarchical information Second tables of data identifies whether corresponding tables of data is main table.

When the tables of data of the search application reference of business side user creation is multiple, needed in editing meta-data structure Set the hierarchical information between each tables of data, i.e., a tables of data need to be set as main table, other tables are subtabulation, subtabulation and main table it Between by major key or other fields carry out data correlation, the field that can establish incidence relation between table and table is associated characters Section.Other fields in subtabulation in addition to associate field can be mutually indepedent with the field in main table, it is also possible to which needs depend on Field in main table carries out operation, and user can select main table, and terminal will select tables of data level letter for the tables of data of main table Breath adds main table mark, addition subtabulation mark in the tables of data hierarchical information of other tables of data, and in data packet hierarchical relationship Record the associate field between subtabulation and main table.It, can be with if user has selected shop_extra and shop_auth two to open tables of data Selected shop_extra tables of data is main table, and shop_auth tables of data is subtabulation.

Server need to first judge the corresponding tables of data of data flow in metadata when carrying out field processing to newly-increased data flow It whether is main table in structure, server extracts tables of data hierarchical information from the first metadata structure, judges tables of data level Information is identified with the presence or absence of main table, and then determining that the second tables of data identifies corresponding tables of data if it exists is main table, if data surface layer There are subtabulation marks in grade information, then determining that the second tables of data identifies corresponding tables of data is subtabulation.

Step 234, it when determining the second tables of data to identify corresponding tables of data is not main table, is identified from the second tables of data Associate field is extracted in corresponding tables of data, is searched from search storage engines corresponding with associate field and raw according to main table At record row.

The corresponding data flow of main table generally can all be written in database prior to subtabulation data.Therefore, when newly-increased data flow pair The tables of data answered be main table when, can directly according to search application metadata structure to the field in newly-increased data flow at Reason, and the main literary name section pair of record row can be increased newly by being first stored in search storage engines according to the field after main table Data Stream Processing In the data column answered, subtabulation field in newly-increased record row is vacated.When the corresponding tables of data of newly-increased data flow is subtabulation, due to Subtabulation field may need to depend on main literary name section and be handled, so when handling subtabulation data flow, it need to be according to pair Major key in table is first from being stored in the search record for obtaining in search storage engines and generating according to main table data flow.Specifically, When it is not main table that server, which determines the second tables of data to identify corresponding tables of data, corresponding data are identified from the second tables of data Associate field is extracted in table, then searches the corresponding storage region of the first application identities from search storage engines, from the storage It is searched and the consistent record row data of major key field in region.Server obtain in tables of data hierarchical information tables of data and main table it Between associate field, extract the corresponding field data of associate field from record row data.

Step 236, according to the field in the first metadata structure and the record row found, to the number in newly-increased data flow Processing, which is carried out, according to field generates the first field of application.

Server is according to the field data in the first metadata structure and the record row extracted in newly-increased data flow Data field is handled, i.e., extracts each processing logic for applying first field from the first metadata structure, patrolled according to processing The data field collected in newly-increased data flow carries out calculation process, when data field is field associated with main table, according to place It manages logic and the first field of calculation process generation application is carried out to associate field and data field.

Search can be wrapped specifically using the step 240 that record is added in search storage engines according to the first application identities It includes: the first field of application is supplemented in the respective column in record row.

Server carries out integration generation search application record according to the first field of application obtained after subtabulation processing, and will search The first field of each application in Suo Yingyong record, is added in the corresponding data column of the record row found.

It should be understood that although each step in the flow chart of Fig. 2-5 is successively shown according to the instruction of arrow, These steps are not that the inevitable sequence according to arrow instruction successively executes.Unless expressly stating otherwise herein, these steps Execution there is no stringent sequences to limit, these steps can execute in other order.Moreover, at least one in Fig. 2-5 Part steps may include that perhaps these sub-steps of multiple stages or stage are not necessarily in synchronization to multiple sub-steps Completion is executed, but can be executed at different times, the execution sequence in these sub-steps or stage is also not necessarily successively It carries out, but can be at least part of the sub-step or stage of other steps or other steps in turn or alternately It executes.

In one embodiment, as shown in fig. 6, providing a kind of search for using data processing equipment, comprising: log monitoring Module 610, meta structure searching module 620, data processing module 630 and engine data adding module 640, in which:

Log monitoring module 610, for being monitored to database journal, when monitoring that newly-increased data flow is inserted into data When the log of library, first database mark and the first tables of data mark are read from newly-increased data flow.

Meta structure searching module 620, it is first yuan corresponding for searching first database mark and the first tables of data mark Data structure, and obtain the first application identities of the first metadata structure.

Data processing module 630, for according to the first metadata structure to the data field in newly-increased data flow at Reason generates the first field of application, generates search application record according to the first field of application.

Engine data adding module 640, for search to be added to search storage using record according to the first application identities In engine.

In one embodiment, data processing module 630 may include:

Thread searching unit searches for obtaining the corresponding type of service of the first metadata structure and enables type of service Corresponding data processing threads.

Field processing unit, in data processing threads, each data element to be default in the first metadata structure of acquisition Handle logic；The corresponding data field of each data element is searched from newly-increased data flow, according to default processing logic to data field It is handled and is generated using first field.

In one embodiment, searching for can also include: using data processing equipment

Data cache module, for by newly-increased data flow cache into message-oriented middleware buffer area corresponding with type of service Domain.

Read module is cached, for reading newly-increased data in the corresponding buffer zone of type of service from message-oriented middleware Stream.

Command reception module is chosen, the tables of data for receiving service terminal transmission chooses instruction, and tables of data chooses instruction The second application identities of middle carrying, the second Database Identification and the second tables of data mark.

Table structure return module, for searching the second tables of data mark pair from the corresponding database of the second Database Identification The tables of data answered obtains the table structure information of tables of data and returns to terminal.

Result-generation module, for receive terminal transmission to table structure information carry out the edited data element of logical process Information generates the second metadata structure according to data metamessage and the second application identities.

Structure memory module, for carrying out the second metadata structure and the second Database Identification and the second tables of data mark Associated storage.

In one embodiment, device can also include:

Template generation module, for parsing the field information and index information of each data element, root from data metamessage It is generated according to field information and applies data template.

Tables of data creation module, for being created and the second application identities in search storage engines according to application data template Corresponding application data sheet.

Index creation module, for being application data sheet creation index according to index information.

In one embodiment, engine data adding module 640 may include:

Tables of data searching unit, for searching the corresponding application data sheet of the first application identities from search storage engines.

Field record unit, for that will search for using each first field of application in record, be added to find using number In respective column according to record row new in table.

In one embodiment, device can also include:

Inquiry instruction receiving module carries mesh in data query instruction for receiving the data query instruction of terminal transmission Mark application identities and retrieval logical formula.

Tables of data search module identifies corresponding application data sheet for searching target application from search storage engines.

Data retrieval module carries out data for identifying corresponding application data sheet to target application according to retrieval logical formula Retrieval obtains target data, and target data is returned to terminal.

In one embodiment, a kind of computer equipment is provided, which can be server, internal junction Composition can be as shown in Figure 7.The computer equipment include by system bus connect processor, memory, network interface and Database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The memory packet of the computer equipment Include non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer program and data Library.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The calculating The database of machine equipment applies data processing related data for searching for.The network interface of the computer equipment is used for and outside Terminal passes through network connection communication.To realize that data processing side is applied in a kind of search when the computer program is executed by processor Method.

It will be understood by those skilled in the art that structure shown in Fig. 7, only part relevant to application scheme is tied The block diagram of structure does not constitute the restriction for the computer equipment being applied thereon to application scheme, specific computer equipment It may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.

In one embodiment, a kind of computer equipment, including memory and processor are provided, which is stored with Computer program, the processor are performed the steps of and are monitored to database journal when executing computer program, when monitoring When newly-increased data flow is inserted into database journal, first database mark and the first tables of data mark are read from newly-increased data flow Know；It searches first database mark and the first tables of data identifies corresponding first metadata structure, and obtain the first metadata knot First application identities of structure；Processing is carried out to the data field in newly-increased data flow according to the first metadata structure and generates application member Field generates search application record according to the first field of application；Search is added to search using record according to the first application identities In storage engines.

In one embodiment, it realizes according to the first metadata structure when processor executes computer program to newly-increased data Data field in stream carries out being also used to obtain the corresponding industry of the first metadata structure when processing generates the step of the first field of application Service type searches and enables the corresponding data processing threads of type of service；In data processing threads, the first metadata knot is obtained The default processing logic of each data element in structure；The corresponding data field of each data element is searched from newly-increased data flow, according to default Processing logic handles data field and is generated using first field.

In one embodiment, it also performs the steps of when processor executes computer program by newly-increased data flow cache The buffer zone corresponding with type of service into message-oriented middleware；From the corresponding buffer zone of type of service of message-oriented middleware Read newly-increased data flow.

In one embodiment, it is also performed the steps of when processor executes computer program and receives the number that terminal is sent It chooses and instructs according to table, tables of data, which is chosen, carries the second application identities, the second Database Identification and the second tables of data mark in instruction； The second tables of data is searched from the corresponding database of the second Database Identification and identifies corresponding tables of data, obtains the table knot of tables of data Structure information simultaneously returns to terminal；The data metamessage that terminal is sent is received, the data metamessage is to carry out to table structure information What logical process editor was formed, the second metadata structure is generated according to data metamessage and the second application identities；By binary number Storage is associated according to structure and the second Database Identification and the second tables of data mark.

In one embodiment, it also performs the steps of when processor executes computer program and is solved from data metamessage The field information and index information of each data element is precipitated, is generated according to field information and applies data template；In search storage engines It is middle that application data sheet corresponding with the second application identities is created according to application data template；It is application data sheet according to index information Creation index.

In one embodiment, it realizes when processor executes computer program and is remembered search application according to the first application identities It is also used to when the step that record is added in search storage engines: searching that the first application identities are corresponding to answer from search storage engines Use tables of data；By the first field of each application in search application record, it is added to record row new in the application data sheet found Respective column in.

In one embodiment, it is also performed the steps of when processor executes computer program and receives the number that terminal is sent According to inquiry instruction, target application mark and retrieval logical formula are carried in data query instruction；Mesh is searched from search storage engines Mark the corresponding application data sheet of application identities；Corresponding application data sheet is identified to target application according to retrieval logical formula to count Target data is obtained according to retrieval, target data is returned into terminal.

In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated Machine program is performed the steps of when being executed by processor and is monitored to database journal, when monitoring newly-increased data flow insertion When to database journal, first database mark and the first tables of data mark are read from newly-increased data flow；Search the first data Library mark and the first tables of data identify corresponding first metadata structure, and obtain the first application mark of the first metadata structure Know；Processing is carried out to the data field in newly-increased data flow according to the first metadata structure and generates the first field of application, according to application First field generates search application record；Search is added in search storage engines using record according to the first application identities.

In one embodiment, it realizes according to the first metadata structure when computer program is executed by processor to newly-increased number It is corresponding to be also used to obtain the first metadata structure when carrying out the step of the first field of processing generation application according to the data field in stream Type of service searches and enables the corresponding data processing threads of type of service；In data processing threads, the first metadata is obtained The default processing logic of each data element in structure；The corresponding data field of each data element is searched from newly-increased data flow, according to pre- If processing logic handles data field and is generated using first field.

In one embodiment, it is also performed the steps of when computer program is executed by processor and delays newly-increased data flow Buffer zone corresponding with type of service is deposited into message-oriented middleware；From the corresponding buffer zone of the type of service of message-oriented middleware It is middle to read newly-increased data flow.

In one embodiment, it is also performed the steps of when computer program is executed by processor and receives service terminal hair The tables of data sent chooses instruction, and tables of data is chosen in instruction and carries the second application identities, the second Database Identification and the second data Table mark；The second tables of data is searched from the corresponding database of the second Database Identification and identifies corresponding tables of data, obtains data The table structure information of table simultaneously returns to terminal；The data metamessage that terminal is sent is received, the data metamessage is to table structure Information carries out logical process editor formation, generates the second metadata structure according to data metamessage and the second application identities；It will Second metadata structure and the second Database Identification and the second tables of data mark are associated storage.

In one embodiment, it is also performed the steps of from data metamessage when computer program is executed by processor The field information and index information for parsing each data element generate according to field information and apply data template；Draw in search storage In holding up, application data sheet corresponding with the second application identities is created according to application data template；It is using number according to index information It creates and indexes according to table.

In one embodiment, it realizes when computer program is executed by processor and is applied search according to the first application identities Be also used to when the step that record is added in search storage engines: it is corresponding to search the first application identities from search storage engines Application data sheet；By the first field of each application in search application record, it is added to record new in the application data sheet found In capable respective column.

In one embodiment, it is also performed the steps of when computer program is executed by processor and receives what terminal was sent Data query instructs, and target application mark and retrieval logical formula are carried in data query instruction；It is searched from search storage engines Target application identifies corresponding application data sheet；According to retrieval logical formula, to target application identify corresponding application data sheet into Row data retrieval obtains target data, and target data is returned to terminal.

Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, To any reference of memory, storage, database or other media used in each embodiment provided herein, Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..

Each technical characteristic of above embodiments can be combined arbitrarily, for simplicity of description, not to above-described embodiment In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance Shield all should be considered as described in this specification.

The several embodiments of the application above described embodiment only expresses, the description thereof is more specific and detailed, but simultaneously It cannot therefore be construed as limiting the scope of the patent.It should be pointed out that coming for those of ordinary skill in the art It says, without departing from the concept of this application, various modifications and improvements can be made, these belong to the protection of the application Range.Therefore, the scope of protection shall be subject to the appended claims for the application patent.

Claims

1. a kind of search application data processing method, which comprises

Database journal is monitored, when monitoring that newly-increased data flow is inserted into the database journal, from newly-increased data First database mark and the first tables of data mark are read in stream；

It searches the first database mark and first tables of data identifies corresponding first metadata structure, and described in acquisition First application identities of the first metadata structure；

Processing is carried out to the data field in the newly-increased data flow according to first metadata structure and generates the first field of application, Search application record is generated according to the first field of the application；

2. the method according to claim 1, wherein described increase newly according to first metadata structure to described Data field in data flow carries out processing and generates the first field of application, comprising:

The corresponding type of service of first metadata structure is obtained, search and enables the corresponding data processing of the type of service Thread；

In the data processing threads, the default processing logic of each data element in first metadata structure is obtained；From institute It states and searches the corresponding data field of each data element in newly-increased data flow, according to the default processing logic to the data word Duan Jinhang is handled and is generated using first field.

3. according to the method described in claim 2, it is characterized in that, the acquisition the first application metadata structure is corresponding After type of service, further includes:

4. the method according to claim 1, wherein the method also includes:

It receives the tables of data that terminal is sent and chooses instruction, the tables of data, which is chosen, carries the second application identities, the second number in instruction According to library mark and the second tables of data mark；

From the corresponding database of second Database Identification, searches second tables of data and identify corresponding tables of data, obtain It takes the table structure information of the tables of data and returns to the terminal；

The data metamessage that the terminal is sent is received, the data metamessage is to carry out logical process to the table structure information What editor was formed, the second metadata structure is generated according to the data metamessage and second application identities；

5. according to the method described in claim 4, it is characterized in that, the method also includes:

The field information and index information that each data element is parsed from the data metamessage are generated according to the field information Using data template；

In described search storage engines, application corresponding with second application identities is created according to the application data template Tables of data；

It is application data sheet creation index according to the index information.

6. according to the method described in claim 5, it is characterized in that, described answer described search according to first application identities It is added in search storage engines with record, comprising:

By the first field of each application in described search application record, it is added in the application data sheet found newly In the respective column for recording row.

7. according to the method described in claim 5, it is characterized in that, the method also includes:

The data query instruction that terminal is sent is received, carries target application mark and retrieval logic in the data query instruction Formula；

According to the retrieval logical formula, corresponding application data sheet progress data retrieval is identified to the target application and obtains target The target data is returned to the terminal by data.

8. data processing equipment is applied in a kind of search, which is characterized in that described device includes:

Log monitoring module, for being monitored to database journal, when monitoring that newly-increased data flow is inserted into the database When log, first database mark and the first tables of data mark are read from newly-increased data flow；

Meta structure searching module, it is first yuan corresponding for searching the first database mark and first tables of data mark Data structure, and obtain the first application identities of first metadata structure；

Data processing module, for according to first metadata structure to the data field in the newly-increased data flow at Reason generates the first field of application, generates search application record according to the first field of the application；

Engine data adding module, for described search to be added to search storage using record according to first application identities In engine.

9. device according to claim 8, the data processing module include:

Thread searching unit searches for obtaining the corresponding type of service of first metadata structure and enables the business The corresponding data processing threads of type；

Field processing unit, in the data processing threads, obtaining each data element in first metadata structure Default processing logic；The corresponding data field of each data element is searched from the newly-increased data flow, according to the default place Reason logic handles the data field and is generated using first field.

10. device according to claim 9, described device further include:

Data cache module, for the newly-increased data flow cache is corresponding with the type of service slow into message-oriented middleware Deposit region；

Read module is cached, it is described new for being read from the corresponding buffer zone of the type of service of the message-oriented middleware Increase data flow.

11. device according to claim 8, described device further include:

Command reception module is chosen, the tables of data for receiving terminal transmission chooses instruction, and the tables of data is chosen in instruction and taken The second application identities of band, the second Database Identification and the second tables of data mark；

Table structure return module, for searching second tables of data from the corresponding database of second Database Identification Corresponding tables of data is identified, the table structure information of the tables of data is obtained and returns to the terminal；

Result-generation module, the data metamessage sent for receiving the terminal, the data metamessage are to the table knot Structure information carries out logical process editor formation, generates binary number according to the data metamessage and second application identities According to structure；

Structure memory module is used for second metadata structure and second Database Identification and second tables of data Mark is associated storage.

12. device according to claim 11, described device further include:

Template generation module, for parsing the field information and index information of each data element, root from the data metamessage It is generated according to the field information and applies data template；

Tables of data creation module, in described search storage engines, according to application data template creation and described the The corresponding application data sheet of two application identities；

13. device according to claim 12, the engine data adding module include:

Tables of data searching unit, it is corresponding using data for searching first application identities from described search storage engines Table；

Field record unit, for being added to the institute found for the first field of each application in described search application record In the respective column for stating record row new in application data sheet.

14. device according to claim 12, described device further include:

Inquiry instruction receiving module carries mesh in the data query instruction for receiving the data query instruction of terminal transmission Mark application identities and retrieval logical formula；

Tables of data search module, it is corresponding using data for searching the target application mark from described search storage engines Table；

Data retrieval module, for according to the retrieval logical formula, to the target application identify corresponding application data sheet into Row data retrieval obtains target data, and the target data is returned to the terminal.

15. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists In the step of processor realizes any one of claims 1 to 7 the method when executing the computer program.

16. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program The step of method described in any one of claims 1 to 7 is realized when being executed by processor.