CN110489417A - A kind of data processing method and relevant device - Google Patents
A kind of data processing method and relevant device Download PDFInfo
- Publication number
- CN110489417A CN110489417A CN201910679327.9A CN201910679327A CN110489417A CN 110489417 A CN110489417 A CN 110489417A CN 201910679327 A CN201910679327 A CN 201910679327A CN 110489417 A CN110489417 A CN 110489417A
- Authority
- CN
- China
- Prior art keywords
- index
- target
- target data
- storage
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/243—Natural language query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2468—Fuzzy queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Automation & Control Theory (AREA)
- Artificial Intelligence (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present application discloses a kind of data processing method and relevant device, the method is applied to technical field of data processing, it include: when receiving the storage request for storing target data from client, field parsing is carried out to target data, if detecting that there is no targets to index in default index storage region based on field parsing result, storage organization type belonging to target data is then determined based on semantic information, and the index request indexed for creating target is sent to the search server of pre-connection, when receiving the target index of search server return, storage target data is indexed according to target.Using the embodiment of the present application, target index can be created, the fractionation to unnecessary field is prevented according to the storage organization type belonging to target data, be conducive to the efficiency for improving creation index.
Description
Technical field
The present invention relates to technical field of data processing more particularly to a kind of data processing method and relevant devices.
Background technique
Currently, many data management platforms can provide the function of carrying out target data inquiry by keyword.If
Need the function of inquiring using target data, it is necessary first to it is corresponding to create the target data in the corresponding storage region of server
Index, after the completion of index creation, can pass through the keyword query target data.Wherein, which is by target
It is obtained after data progress word segmentation processing.
Generally all relevant database, such as MySql, Oracle etc. are arrived in storage for storage of the traditional services to target data.
If server creates the corresponding index of target data by relevant database, thus realize the query function of target data,
It needs in addition to safeguard a set of database service, good word segmentation processing can not be carried out to non-structured target data, to create
It indexes.Therefore, how more efficiently creation index, storage target data become one to realize the inquiry of target data
A urgent problem to be solved.
Summary of the invention
The embodiment of the present application provides a kind of data processing method and relevant device, is conducive to the effect for improving creation index
Rate.
In a first aspect, the embodiment of the present application provides a kind of data processing method, the method is applied to server, the party
Method includes:
The storage request for storing target data from client is received, includes the target in the storage request
Data；
Field parsing is carried out to the target data, to obtain field parsing result, the field parsing result includes institute
State the semantic information of the corresponding field of target data and the field；
With the presence or absence of the corresponding target index of the field in the default index storage region of detection；
If detecting, there is no the targets to index in the default index storage region, true based on institute's semantic information
Storage organization type belonging to the fixed target data；
The index request indexed for creating target is sent to the search server of pre-connection, the index request carries institute
Storage organization type and the target data are stated, is the mesh in order to which described search server is requested according to the index
The target for marking data creation and the storage organization type matching indexes；
The target index that described search server returns is received, and the number of targets is stored according to target index
According to.
In one embodiment, described that storage organization type belonging to the target data is determined based on institute's semantic information
Specific embodiment are as follows:
If detecting that the corresponding field of the target data is searched for complete match based on institute's semantic information, by institute
It states storage organization type belonging to target data and is determined as key word type.
In one embodiment, described that storage organization type belonging to the target data is determined based on institute's semantic information
Specific embodiment are as follows:
If detecting that the corresponding field of the target data is searched for fuzzy matching based on institute's semantic information, by institute
Storage organization type belonging to target data is stated to be determined as segmenting type.
In one embodiment, it is described store the target data in target index after, can be with
The update instruction information for being directed to the target data is sent to the client, the update instruction information is for referring to
Show client according to target data described in default update policy update.
In one embodiment, the default more new strategy includes delay more new strategy or time more new strategy, wherein
The delay more new strategy is used to indicate the client when detecting the trigger action for target data, updates the mesh
Mark data；The time more new strategy is used to indicate the client and updates the target data after preset time.
In one embodiment, described that the index request indexed for creating target is sent to the search server of pre-connection
Later, the method also includes:
If do not receive the target index of described search server return within a preset time, generates one and preset
Index, and the target data is stored according to the default index.
In one embodiment, it after the default index of the generation one, can also create for receiving described search clothes
The asynchronous thread that the target that device returns of being engaged in indexes；Wherein, the target index for receiving described search server and returning, and
If the target index in store the target data specific embodiment be received by the asynchronous thread described in
Search server returns to the target index, then by presetting index described in the target index upgrade, and according to the target
Index stores the target data.
Second aspect, the embodiment of the present application provide a kind of data processing equipment, which includes for holding
The module of the method for the above-mentioned first aspect of row.
The third aspect, the embodiment of the present application provide a kind of server, which includes processor, network interface and deposit
Reservoir, the processor, network interface and memory are connected with each other, wherein control of the network interface by the processor
For messaging, the memory is used to store the computer program for supporting server to execute the above method, the computer
Program includes program instruction, and the processor is configured for calling described program instruction, the method for executing above-mentioned first aspect.
Fourth aspect, the embodiment of the present application provide a kind of computer readable storage medium, the computer-readable storage
Media storage has computer program, and the computer program includes program instruction, and described program instructs when being executed by a processor
The method for making the processor execute above-mentioned first aspect.
In the embodiment of the present application, server can be asked when receiving the storage for storing target data from client
When asking, field parsing is carried out to target data, to obtain the semantic information of the corresponding field of target data and field, and is detected
With the presence or absence of the corresponding target index of field in default index storage region.Further, if detecting default index memory block
There is no targets to index in domain, then determines storage organization type belonging to target data based on semantic information, and to pre-connection
Search server sends the index request for creating target index, is mesh for requesting search server to be requested according to index
The target for marking data creation and storage organization type matching indexes.When receiving the target index of search server return, root
Storage target data is indexed according to target.Using the embodiment of the present application, can be created according to the storage organization type belonging to target data
Target index is built, the fractionation to unnecessary field is prevented, is conducive to the efficiency for improving creation index.
Detailed description of the invention
Technical solution in ord to more clearly illustrate embodiments of the present application, below will be to needed in embodiment description
Attached drawing is briefly described, it should be apparent that, drawings in the following description are some embodiments of the invention, general for this field
For logical technical staff, without creative efforts, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow diagram of data processing method provided by the embodiments of the present application；
Fig. 2 is the flow diagram of another data processing method provided by the embodiments of the present application；
Fig. 3 is a kind of schematic block diagram of data processing equipment provided by the embodiments of the present application；
Fig. 4 is a kind of schematic block diagram of server provided by the embodiments of the present application.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
It is a kind of flow diagram of data processing method provided by the embodiments of the present application referring to Fig. 1, Fig. 1, this method is answered
It for server, can be executed by server, as shown, the data processing method can include:
101: receiving the storage request for storing target data from client, include number of targets in storage request
According to.
102: field parsing being carried out to target data, to obtain field parsing result, which includes described
The semantic information of the corresponding field of target data and field.
Wherein, above-mentioned server can be able to be a server with the corresponding server of data management platform, the server,
Or the server cluster of multiple servers composition, the server can provide the related service of data management.For example, should
Data management platform can be log platform, which can provide the function that log query is carried out by keyword.
Wherein, which can be the corresponding application of log platform or website, or be equipped with log platform application or
Open the terminal device of log platform website.In one embodiment, which can be non-structured data.
103: with the presence or absence of the corresponding target index of the field in the default index storage region of detection, if detecting default
It indexes in storage region there is no target index, then storage organization type belonging to target data is determined based on semantic information.
Wherein, server is in order to realize that the query function of target data, server need in advance in index storage region
The corresponding index of target data is created, after creation indexes, the inquiry that keyword carries out target data can be passed through, wherein should
Index has corresponding relationship with keyword.In one embodiment, presetting includes at least one index in index storage region, often
A index is corresponding with keyword.In response to this, it is used to store target data from client when server is received
When storage request, field parsing can be carried out to target data, obtain at least one corresponding field of target data.Further
Ground can compare at least one above-mentioned field and the keyword of pre-stored each index, if comparison obtains at least
The Keywords matching of either field and any index in one field, it is determined that there are target ropes in default index storage region
Draw.
On the contrary, being not present and the Keywords matching of any index if server detects at least one above-mentioned field
Field, it is determined that there is no targets to index in default index storage region.
Wherein, above-mentioned storage organization type may include key word type and participle type.In one embodiment, work as clothes
Business device detects in default index storage region there is no when target index, can detecte the language that target data corresponds to each field
Adopted information, if detecting that target data corresponds to either field and searches for complete match based on the semantic information, by number of targets
It is determined as key word type according to affiliated storage organization type.
If detecting that the corresponding either field of target data is searched for fuzzy matching based on semantic information, by number of targets
It is determined as segmenting type according to affiliated storage organization type.Wherein, the field searched for complete match is properly termed as the first word
Section, first field have uniqueness, such as the semantic information of first field can characterize address name, certificate number etc., often
One user only corresponds to an address name and certificate number；Wherein, the second word is properly termed as the field of fuzzy search
The semantic information of section, second field does not have uniqueness, such as the semantic information of second field can characterize Business Name
Deng the said firm's title can correspond to multiple users.In this manner, server can be created according to different target datas
Different storage organizations is built, unnecessary character can be split to avoid search server when creating target index, it can
To effectively improve data-handling efficiency.
104: the index request indexed for creating target is sent to the search server of pre-connection, index request carries
Storage organization type and target data are deposited for target data creation with this in order to which search server is requested according to the index
The target of storage structure type matching indexes.
105: receiving the target index that search server returns, and storage target data is indexed according to target.
In one embodiment, it can be pre-configured with the configuration file for search server, include in the configuration file
About the address of connection search server, port, agreement, connection time-out time, the routing number of agreement and maximum number of connections etc.
Relevant configuration.Further, it when server detects the starting of above-mentioned search server, can searched based on above-mentioned configuration file
Rope server registration establishes connection with search server to realize the subsequent data interaction between search server.
Wherein, which can be ElasticSearch, which is one based on Lucene
Search server.It provides the full-text search engine of a distributed multi-user ability, is based on RESTfulweb interface.
Elasticsearch is to be developed with Java, and issue as the open source code under Apache license terms, is currently popular
Enterprise search engine.It is stable designed for real-time search in cloud computing, can be reached, it is reliably, quickly, easy to install and use.
In one embodiment, after server determines storage organization type belonging to target data, can to
The search server for establishing connection sends the index request of creation target index, and index request includes the storage knot of target data
Structure type and the target data.Further, search server can be based on the target data and storage organization type, from
Move for the target data be created that with the target of the storage organization type matching index, and by the target index return server.
After server receives the target index of search server return, it can be indexed in corresponding storage region in target and store the mesh
Data are marked, and distribute keyword for the target data, the keyword is used in order to subsequent, inquires target data.Wherein, should
It can be disk or file that target, which indexes corresponding storage region,.
In one embodiment, above-mentioned data processing method can be applied to the corresponding plug-in unit of a data management platform, should
The object of plug-in unit insertion is search server.Illustratively, which is a log platform, which can be
The file to one's name serviced is created under the service folder of search server, includes being deposited with jar packet form in this document folder
Log cloud plug-in unit, when anterior plug-in operation information and log platform need configuration file.Wherein, the operation information packet
It includes: the description information of plug-in unit, for describing the effect of the plug-in unit；The version information of plug-in unit；Plug-in unit is shown in search server
Title；The entrance of plug-in unit, the java version information that plug-in unit uses；Plug-in unit is published to the corresponding particular version of search server.
In one embodiment, if detecting, there are target indexes in above-mentioned default index storage region, can directly exist
The target data is stored in target index.
In the embodiment of the present application, server can be asked when receiving the storage for storing target data from client
When asking, field parsing is carried out to target data, to obtain the semantic information of the corresponding field of target data and field, and is detected
With the presence or absence of the corresponding target index of field in default index storage region.Further, if detecting default index memory block
There is no targets to index in domain, then determines storage organization type belonging to target data based on semantic information, and to pre-connection
Search server sends the index request for creating target index, is mesh for requesting search server to be requested according to index
The target for marking data creation and storage organization type matching indexes.When receiving the target index of search server return, root
Storage target data is indexed according to target.Using the embodiment of the present application, can be created according to the storage organization type belonging to target data
Target index is built, the fractionation to unnecessary field is prevented, is conducive to the efficiency for improving creation index.
Referring to fig. 2, Fig. 2 is the flow diagram of another data processing method provided by the embodiments of the present application, this method
It can be executed by server, as shown, the data processing method can include:
201: when receiving the storage request for storing target data from client, word being carried out to target data
Section parsing, to obtain field parsing result, which includes the corresponding field of the target data and field
Semantic information.
202: with the presence or absence of the corresponding target index of the field in the default index storage region of detection, if detecting default
It indexes in storage region there is no target index, then storage organization type belonging to target data is determined based on semantic information.
203: the index request indexed for creating target is sent to the search server of pre-connection, index request carries
Storage organization type and target data are deposited for target data creation with this in order to which search server is requested according to the index
The target of storage structure type matching indexes.
204: receiving the target index that search server returns, and storage target data is indexed according to target.
Wherein, the specific embodiment of 201~step 204 of above-mentioned steps, may refer to step 101 in above-described embodiment~
The associated description of step 105, details are not described herein again.
205: Xiang Shangshu client sends the update instruction information for being directed to target data, which is used to indicate
Client is according to default update policy update target data.
In one embodiment, above-mentioned default more new strategy may include postponing more new strategy or time more new strategy,
Wherein, above-mentioned delay more new strategy is used to indicate client when detecting the trigger action for target data, more fresh target
Data；The time more new strategy, is used to indicate client and updates target data after preset time.Wherein, which can
Think the search operation of search target data, or that checks target data checks operation etc., alternatively, other are directed to mesh
Any operation of data is marked, the application is not specifically limited in this embodiment.
In one embodiment, which can be 0s, 1s etc., all can be set in advance by developer's default
It sets, can also be selected by user according to self-demand.Alternatively, user can also root after the preset time has determined
The preset time is adjusted according to self-demand evidence.Wherein, which can be understood as refreshing immediately.
Illustratively, above-mentioned default more new strategy is time more new strategy, and the time more new strategy is used to indicate client
Target data is updated after 0s, in this case, after client receives the update instruction information for target data, Ke Yili
Update the target data.
In one embodiment, when above-mentioned default more new strategy is delay more new strategy, above-mentioned delay more new strategy is used
It is detecting in instruction client for when checking operation of target data, is updating target data.Wherein, this checks operation for example
Can be for for the touch control operation for checking button, or the voice signal etc. for checking target data.Using such side
Formula does not need to update the performance that target data largely consumes hardware immediately, and delay, which refreshes to have target data in next time, touches
Refreshed when hair operation, ensure that the protection of hardware performance and user can inquire the target data of preservation in time.
In one embodiment, it is sent to the search server of pre-connection and requests it for creating the index that target indexes
Afterwards, if do not receive the target index of search server return within a preset time, a default index, and root can be generated
The target data is stored according to default index
In one embodiment, the waiting time for waiting search server to return to target index can be preset (that is, pre-
If the time), when sending the index request for creating target index to the search server of pre-connection, opens timer and carry out
If timing does not receive the mesh of return also detect that the corresponding current duration of timer is more than or equal to the waiting time
Mark index then presets index for available one, and stores the target data in the corresponding storage region of default index.
In one embodiment, it after server generates a default index, can also create for receiving search service
The asynchronous thread for the target index that device returns returns to mesh for continuing waiting for search server that is, opening a new thread
Mark index.Further, if server is indexed by the target that above-mentioned asynchronous thread receives search server return, pass through
The default index of target index upgrade, and storage target data is indexed according to target.
It in one embodiment, can be in advance to event registration one readjustment for receiving search server return target index
Monitor can open the readjustment when server does not receive the target index of search server return within a preset time
Monitor (creates an above-mentioned asynchronous thread), for receiving the target index of search server return.
In one embodiment, the data processing method in the embodiment of the present application is applied to server cluster, the server
Clustered deploy(ment) has N number of node (such as N number of server).In this case, it sends to the search server of pre-connection for creating
Before the index request of target index, the quantity for being also based on server cluster interior joint, which is determined, carries out fragment to target data
In the process, the corresponding main leaf of each node and bluff bit quantity, and the corresponding main leaf of each node and bluff bit quantity are added
It is requested to index, is sent to search server.It further, can be according to every after search server receives index request
The corresponding main leaf of one node and bluff bit quantity carry out fragment processing to target data, and it is corresponding to create each node
Target index after the completion of creation, is back to corresponding node by target index.It in this manner, can basis
The number that node is disposed in cluster goes the quantity of setting main leaf and bluff bit, is effectively reduced excessive bluff bit quantity to hardware resource
Waste.
Illustratively, when there are N number of node, when based on server cluster interior joint quantity determine to target data into
It, can be according to the bluff bit quantity and the of second node when the corresponding main leaf of each node and bluff bit quantity in row Slicing procedure
The main leaf quantity of one node is identical, third node bluff bit quantity is identical as the main leaf quantity of second node, and so on original
Then, the corresponding main leaf of each node and bluff bit quantity are determined.Wherein, the data that main leaf and bluff bit store are identical, can prevent
Hardware problem leads to loss of data.
In the embodiment of the present application, server can be asked when receiving the storage for storing target data from client
When asking, field parsing is carried out to target data, to obtain the semantic information of the corresponding field of target data and field, and is detected
With the presence or absence of the corresponding target index of field in default index storage region.Further, if detecting default index memory block
There is no targets to index in domain, then determines storage organization type belonging to target data based on semantic information, and to pre-connection
Search server sends the index request for creating target index, is mesh for requesting search server to be requested according to index
The target for marking data creation and storage organization type matching indexes.When receiving the target index of search server return, root
Storage target data is indexed according to target, and sends the update instruction information for being directed to target data to above-mentioned client, which refers to
Show that information is used to indicate client according to default update policy update target data.Using the embodiment of the present application, it may not be necessary to
Target data is updated immediately and largely consumes the performance of hardware, is conducive to take into account protection to hardware performance and user query to target
The timeliness of data.
The embodiment of the present application also provides a kind of data processing equipments.The device includes for executing earlier figures 1 or Fig. 2
The module of the method, is configured at server.It specifically, is data processing equipment provided by the embodiments of the present application referring to Fig. 3
Schematic block diagram.The data processing equipment of the present embodiment includes:
Communication module 30, for receiving the storage request for storing target data from client, the storage is asked
It include the target data in asking；
Processing module 31, for carrying out field parsing to the target data, to obtain field parsing result, the field
Parsing result includes the semantic information of the corresponding field of the target data and the field；
The processing module 31 is also used to detect in default index storage region with the presence or absence of the corresponding target of the field
Index, if detecting, there is no the targets to index in the default index storage region, is determined based on institute's semantic information
Storage organization type belonging to the target data；
The communication module 30 is also used to send to the search server of pre-connection and ask for creating the index that target indexes
Ask, and receive the target index of described search server return, the index request carrying storage organization type with
And the target data, in order to which described search server is requested according to the index, for the target data creation with it is described
The target of storage organization type matching indexes；
The processing module 31 is also used to index the storage target data according to the target.
In one embodiment, the processing module 31, if specifically for detecting the mesh based on institute's semantic information
It marks the corresponding field of data to search for complete match, then storage organization type belonging to the target data is determined as key
Word type.
In one embodiment, the processing module 31, if being specifically also used to detect based on institute's semantic information described
The corresponding field of target data is searched for fuzzy matching, then is determined as storage organization type belonging to the target data point
Part of speech type.
In one embodiment, the communication module 30 is also used to store the target data in target index
Later, Xiang Suoshu client sends the update instruction information for being directed to the target data, and the update instruction information is used to indicate
Client is according to target data described in default update policy update.
In one embodiment, the default more new strategy includes delay more new strategy or time more new strategy, wherein
The delay more new strategy is used to indicate the client when detecting the trigger action for target data, updates the mesh
Mark data；The time more new strategy is used to indicate the client and updates the target data after preset time.
In one embodiment, the processing module 31 is also used to send to the search server of pre-connection for creating
After the index request of target index, if not receiving the target index of described search server return within a preset time
When, a default index is generated, and the target data is stored according to the default index.
In one embodiment, the processing module 31 creates after being also used to generate a default index for receiving
The asynchronous thread for the target index that described search server returns；If receiving described search server by the asynchronous thread
The target index is returned, then by presetting index described in the target index upgrade, and storage institute is indexed according to the target
State target data.
It should be noted that the function of each functional module of data processing equipment described in the embodiment of the present application can basis
Method of the Fig. 1 perhaps in embodiment of the method described in Fig. 2 implements its specific implementation process and is referred to Fig. 1's or Fig. 2
The associated description of embodiment of the method, details are not described herein again.
In the embodiment of the present application, asked when communication module 30 receives the storage for storing target data from client
When asking, processing module 31 carries out field parsing to target data, obtains the semantic letter of the corresponding field of target data and field
Breath, and detect with the presence or absence of the corresponding target index of field in default index storage region, if detecting default index memory block
There is no targets to index in domain, then determines storage organization type belonging to target data based on semantic information, and pass through communication mould
Block 30 sends the index request indexed for creating target to the search server of pre-connection, and then receives search server and return
Target index, by processing module 31 according to target index storage target data.It, can be according to mesh using the embodiment of the present application
Storage organization type belonging to data is marked, creation target index prevents the fractionation to unnecessary field, is conducive to improve creation rope
The efficiency drawn.
Fig. 4 is referred to, Fig. 4 is a kind of schematic block diagram of server provided by the embodiments of the present application, as shown in figure 4, should
Server includes processor 401, memory 402 and network interface 403.Above-mentioned processor 401, memory 402 and network interface
403 can be connected by bus or other modes, in Fig. 4 shown in the embodiment of the present application for being connected by bus.Wherein, net
Network interface 403 is used for messaging by the control of the processor, and memory 402 is for storing computer program, the calculating
Machine program includes program instruction, and processor 401 is used to execute the program instruction of the storage of memory 402.Wherein, 401 quilt of processor
It is configured to call described program instruction execution: being used to store target from client when receiving by network interface 403
When the storage request of data, field parsing is carried out to the target data, to obtain field parsing result, the field parsing knot
Fruit includes the semantic information of the corresponding field of the target data and the field；In the default index storage region of detection whether
There are the corresponding target indexes of the field；If detecting, there is no the targets to index in the default index storage region,
Storage organization type belonging to the target data is then determined based on institute's semantic information；By network interface 403 to pre-connection
Search server send the index request for creating target index, the index request carrying storage organization type with
And the target data, in order to which described search server is requested according to the index, for the target data creation with it is described
The target of storage organization type matching indexes；The target rope that described search server returns is received by network interface 403
Draw, and the target data is stored according to target index.
In one embodiment, the processor 401, if specifically for detecting the target based on institute's semantic information
The corresponding field of data is searched for complete match, then storage organization type belonging to the target data is determined as keyword
Type.
In one embodiment, the processor 401, if being specifically also used to detect the mesh based on institute's semantic information
It marks the corresponding field of data to search for fuzzy matching, is then determined as segmenting by storage organization type belonging to the target data
Type.
In one embodiment, the network interface 403 is also used to store the target data in target index
Later, Xiang Suoshu client sends the update instruction information for being directed to the target data, and the update instruction information is used to indicate
Client is according to target data described in default update policy update.
In one embodiment, the default more new strategy includes delay more new strategy or time more new strategy, wherein
The delay more new strategy is used to indicate the client when detecting the trigger action for target data, updates the mesh
Mark data；The time more new strategy is used to indicate the client and updates the target data after preset time.
In one embodiment, the processor 401 is also used to send to the search server of pre-connection for creating mesh
After the index request of mark index, if not receiving the target index of described search server return within a preset time
When, a default index is generated, and the target data is stored according to the default index.
In one embodiment, the processor 401 creates after being also used to generate a default index for receiving
The asynchronous thread for the target index that described search server returns；If receiving described search server by the asynchronous thread
The target index is returned, then by presetting index described in the target index upgrade, and storage institute is indexed according to the target
State target data.
It should be appreciated that in the embodiment of the present application, alleged processor 401 can be central processing unit (Central
Processing Unit, CPU), which can also be other general processors, digital signal processor (Digital
Signal Processor, DSP), specific integrated circuit (Application Specific Integrated Circuit,
ASIC), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic
Device, discrete gate or transistor logic, discrete hardware components etc..General processor can be microprocessor or this at
Reason device is also possible to any conventional processor etc..
The memory 402 may include read-only memory and random access memory, and to processor 401 provide instruction and
Data.The a part of of memory 402 can also include nonvolatile RAM.For example, memory 402 can also be deposited
Store up the information of device type.
In the specific implementation, processor 401, memory 402 described in the embodiment of the present application and network interface 403 can be held
Implementation described in embodiment of the method described in row Fig. 1 or Fig. 2 provided by the embodiments of the present application, also can be performed the application
The implementation of data processing equipment described in embodiment, details are not described herein.
A kind of computer readable storage medium, the computer-readable storage medium are provided in another embodiment of the invention
Matter is stored with computer program, and the computer program includes program instruction, and described program instructs realization when being executed by processor:
When receiving the storage request for storing target data from client, field parsing is carried out to the target data,
To obtain field parsing result, the field parsing result includes the language of the corresponding field of the target data and the field
Adopted information；With the presence or absence of the corresponding target index of the field in the default index storage region of detection；If detecting described default
It indexes in storage region there is no target index, then storage belonging to the target data is determined based on institute's semantic information
Structure type；The index request indexed for creating target is sent to the search server of pre-connection, the index request carries
The storage organization type and the target data are described in order to which described search server is requested according to the index
Target data creation and the target of the storage organization type matching index；Receive the target that described search server returns
Index, and the target data is stored according to target index.
The computer readable storage medium can be the internal storage unit of server described in aforementioned any embodiment,
Such as the hard disk or memory of server.The external storage that the computer readable storage medium is also possible to the server is set
Plug-in type hard disk that is standby, such as being equipped on the server, intelligent memory card (Smart Media Card, SMC), secure digital
(Secure Digital, SD) card, flash card (Flash Card) etc..Further, the computer readable storage medium is also
Can both including the server internal storage unit and also including External memory equipment.The computer readable storage medium is used
Other programs and data needed for storing the computer program and the server.The computer readable storage medium
It can be also used for temporarily storing the data that has exported or will export.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the program can be stored in a computer-readable storage medium
In, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, RAM) etc..
Above disclosed is only section Example of the invention, cannot limit the right of the present invention with this certainly
Range, those skilled in the art can understand all or part of the processes for realizing the above embodiment, and according to right of the present invention
Equivalent variations made by it is required that, still belongs to the scope covered by the invention.
Claims (10)
1. a kind of data processing method, which is characterized in that the described method includes:
The storage request for storing target data from client is received, includes the number of targets in the storage request
According to；
Field parsing is carried out to the target data, to obtain field parsing result, the field parsing result includes the mesh
Mark the semantic information of the corresponding field of data and the field；
With the presence or absence of the corresponding target index of the field in the default index storage region of detection；
If detecting, there is no the targets to index in the default index storage region, determines institute based on institute's semantic information
State storage organization type belonging to target data；
The index request indexed for creating target is sent to the search server of pre-connection, is deposited described in the index request carrying
Storage structure type and the target data are the number of targets in order to which described search server is requested according to the index
It is indexed according to creation and the target of the storage organization type matching；
The target index that described search server returns is received, and the target data is stored according to target index.
2. the method according to claim 1, wherein described determine the target data based on institute's semantic information
Affiliated storage organization type, comprising:
If detecting that the corresponding field of the target data is searched for complete match based on institute's semantic information, by the mesh
Storage organization type belonging to mark data is determined as key word type.
3. the method according to claim 1, wherein described determine the target data based on institute's semantic information
Affiliated storage organization type, comprising:
If detecting that the corresponding field of the target data is searched for fuzzy matching based on institute's semantic information, by the mesh
Storage organization type belonging to mark data is determined as segmenting type.
4. method according to claim 1-3, which is characterized in that it is described the target index in storage described in
After target data, the method also includes:
The update instruction information for being directed to the target data is sent to the client, the update instruction information is used to indicate visitor
Family end is according to target data described in default update policy update.
5. according to the method described in claim 4, it is characterized in that, the default more new strategy include delay more new strategy or
Time more new strategy, wherein the delay more new strategy is used to indicate the client and is detecting the touching for target data
When hair operation, the target data is updated；The time more new strategy is used to indicate the client and updates after preset time
The target data.
6. method according to claim 1-5, which is characterized in that described to be sent to the search server of pre-connection
After index request for creating target index, the method also includes:
If do not receive the target index of described search server return within a preset time, a default rope is generated
Draw, and the target data is stored according to the default index.
7. according to the method described in claim 6, it is characterized in that, the method is also after the default index of the generation one
Include:
Create the asynchronous thread of the target index for receiving the return of described search server；
Wherein, the target index for receiving described search server and returning, and in target index described in storage
Target data, comprising:
It indexes if receiving described search server by the asynchronous thread and returning to the target, is indexed by the target
The default index is updated, and the target data is stored according to target index.
8. a kind of data processing equipment, which is characterized in that described device includes:
Communication module is wrapped in the storage request for receiving the storage request for storing target data from client
Include the target data；
Processing module, for carrying out field parsing to the target data, to obtain field parsing result, the field parsing knot
Fruit includes the semantic information of the corresponding field of the target data and the field；
The processing module is also used to detect in default index storage region with the presence or absence of the corresponding target index of the field,
If detecting, there is no the targets to index in the default index storage region, determines the mesh based on institute's semantic information
Mark storage organization type belonging to data；
The communication module is also used to send the index request indexed for creating target to the search server of pre-connection, and
Receive the target index that described search server returns, the index request carrying storage organization type and described
Target data is tied for target data creation with the storage in order to which described search server is requested according to the index
The target of structure type matching indexes；
The processing module is also used to index the storage target data according to the target.
9. a kind of server, which is characterized in that including processor and memory, the processor is mutually interconnected with the memory
It connects, wherein the memory is for storing computer program, and the computer program includes program instruction, the processor quilt
It is configured to call described program instruction, executes the method according to claim 1 to 7.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer journey
Sequence, the computer program are executed by processor to realize method described in claim 1 to 7 any one.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910679327.9A CN110489417B (en) | 2019-07-25 | 2019-07-25 | Data processing method and related equipment |
| PCT/CN2019/120960 WO2021012553A1 (en) | 2019-07-25 | 2019-11-26 | Data processing method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910679327.9A CN110489417B (en) | 2019-07-25 | 2019-07-25 | Data processing method and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110489417A true CN110489417A (en) | 2019-11-22 |
| CN110489417B CN110489417B (en) | 2023-03-28 |
Family
ID=68548292
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910679327.9A Active CN110489417B (en) | 2019-07-25 | 2019-07-25 | Data processing method and related equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110489417B (en) |
| WO (1) | WO2021012553A1 (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125176A (en) * | 2019-12-20 | 2020-05-08 | 北京百度网讯科技有限公司 | Service data searching method and device, electronic equipment and storage medium |
| CN111274350A (en) * | 2020-02-03 | 2020-06-12 | 广州极尚网络技术有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN111914126A (en) * | 2020-07-22 | 2020-11-10 | 浙江乾冠信息安全研究院有限公司 | Processing method, equipment and storage medium for indexed network security big data |
| CN111949479A (en) * | 2020-07-31 | 2020-11-17 | 中国工商银行股份有限公司 | Interactive system and method and equipment for determining index creation condition |
| CN112100414A (en) * | 2020-09-11 | 2020-12-18 | 深圳力维智联技术有限公司 | Data processing method, device, system and computer readable storage medium |
| WO2021012553A1 (en) * | 2019-07-25 | 2021-01-28 | 深圳壹账通智能科技有限公司 | Data processing method and related device |
| CN112988692A (en) * | 2019-12-13 | 2021-06-18 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN113190623A (en) * | 2021-05-14 | 2021-07-30 | 京东数科海益信息科技有限公司 | Data processing method, device, server and storage medium |
| CN113626443A (en) * | 2021-08-26 | 2021-11-09 | 企查查科技有限公司 | Index data processing method and device, computer equipment and storage medium |
| CN113760931A (en) * | 2021-08-20 | 2021-12-07 | 济南浪潮数据技术有限公司 | Resource information access method, device, equipment and medium |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100152B (en) * | 2020-09-14 | 2024-09-13 | 广州海外扛把子网络科技有限公司 | Service data processing method, system, server and readable storage medium |
| CN112948016A (en) * | 2021-02-25 | 2021-06-11 | 京东数字科技控股股份有限公司 | Configuration information generation method, device and equipment |
| CN113392081B (en) * | 2021-06-10 | 2024-07-09 | 北京猿力未来科技有限公司 | Data processing system and method |
| CN116860428A (en) * | 2022-03-28 | 2023-10-10 | 华为技术有限公司 | Data processing method, coding and decoding accelerator and related equipment |
| CN116737428B (en) * | 2023-08-14 | 2023-11-21 | 中科三清科技有限公司 | Air quality mode operation stability checking method and device and electronic equipment |
| CN117076542B (en) * | 2023-08-29 | 2024-06-07 | 中国中金财富证券有限公司 | Data processing method and related device |
| CN116842223B (en) * | 2023-08-29 | 2023-11-10 | 天津鑫宝龙电梯集团有限公司 | Working condition data management method, device, equipment and medium |
| CN116910260B (en) * | 2023-09-13 | 2023-11-17 | 中国标准化研究院 | Digital asset searching method based on big data |
| CN117896440B (en) * | 2024-03-15 | 2024-05-24 | 江西曼荼罗软件有限公司 | Data caching acquisition method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160154851A1 (en) * | 2013-04-24 | 2016-06-02 | Hitachi Ltd. | Computing device, storage medium, and data search method |
| CN105988996A (en) * | 2015-01-27 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Index file generation method and device |
| CN106326295A (en) * | 2015-07-01 | 2017-01-11 | 中兴通讯股份有限公司 | Method and device for storing semantic data |
| US20170139996A1 (en) * | 2012-05-18 | 2017-05-18 | Splunk Inc. | Collection query driven generation of inverted index for raw machine data |
| CN108874924A (en) * | 2018-05-31 | 2018-11-23 | 康键信息技术（深圳）有限公司 | Creation method, device and the computer readable storage medium of search service |
| CN110019211A (en) * | 2017-11-27 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The methods, devices and systems of association index |
| CN110019646A (en) * | 2017-10-12 | 2019-07-16 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for establishing index |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102016224455A1 (en) * | 2016-12-08 | 2018-06-14 | Bundesdruckerei Gmbh | Database index of several fields |
| CN110489417B (en) * | 2019-07-25 | 2023-03-28 | 深圳壹账通智能科技有限公司 | Data processing method and related equipment |
-
2019
- 2019-07-25 CN CN201910679327.9A patent/CN110489417B/en active Active
- 2019-11-26 WO PCT/CN2019/120960 patent/WO2021012553A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170139996A1 (en) * | 2012-05-18 | 2017-05-18 | Splunk Inc. | Collection query driven generation of inverted index for raw machine data |
| US20160154851A1 (en) * | 2013-04-24 | 2016-06-02 | Hitachi Ltd. | Computing device, storage medium, and data search method |
| CN105988996A (en) * | 2015-01-27 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Index file generation method and device |
| CN106326295A (en) * | 2015-07-01 | 2017-01-11 | 中兴通讯股份有限公司 | Method and device for storing semantic data |
| CN110019646A (en) * | 2017-10-12 | 2019-07-16 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for establishing index |
| CN110019211A (en) * | 2017-11-27 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The methods, devices and systems of association index |
| CN108874924A (en) * | 2018-05-31 | 2018-11-23 | 康键信息技术（深圳）有限公司 | Creation method, device and the computer readable storage medium of search service |
Non-Patent Citations (1)
| Title |
|---|
| 高海宾: "基于SQL Server数据库索引的创建与优化分析", 《九江学院学报(自然科学版)》 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021012553A1 (en) * | 2019-07-25 | 2021-01-28 | 深圳壹账通智能科技有限公司 | Data processing method and related device |
| CN112988692B (en) * | 2019-12-13 | 2024-05-07 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN112988692A (en) * | 2019-12-13 | 2021-06-18 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN111125176B (en) * | 2019-12-20 | 2023-10-03 | 北京百度网讯科技有限公司 | Service data searching method and device, electronic equipment and storage medium |
| CN111125176A (en) * | 2019-12-20 | 2020-05-08 | 北京百度网讯科技有限公司 | Service data searching method and device, electronic equipment and storage medium |
| CN111274350A (en) * | 2020-02-03 | 2020-06-12 | 广州极尚网络技术有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN111914126A (en) * | 2020-07-22 | 2020-11-10 | 浙江乾冠信息安全研究院有限公司 | Processing method, equipment and storage medium for indexed network security big data |
| CN111949479A (en) * | 2020-07-31 | 2020-11-17 | 中国工商银行股份有限公司 | Interactive system and method and equipment for determining index creation condition |
| CN111949479B (en) * | 2020-07-31 | 2023-08-25 | 中国工商银行股份有限公司 | Interactive system and index creation condition determining method and equipment |
| CN112100414A (en) * | 2020-09-11 | 2020-12-18 | 深圳力维智联技术有限公司 | Data processing method, device, system and computer readable storage medium |
| CN112100414B (en) * | 2020-09-11 | 2024-02-23 | 深圳力维智联技术有限公司 | Data processing method, device, system and computer readable storage medium |
| CN113190623A (en) * | 2021-05-14 | 2021-07-30 | 京东数科海益信息科技有限公司 | Data processing method, device, server and storage medium |
| CN113190623B (en) * | 2021-05-14 | 2024-05-17 | 京东科技信息技术有限公司 | Data processing method, device, server and storage medium |
| CN113760931B (en) * | 2021-08-20 | 2023-12-29 | 济南浪潮数据技术有限公司 | Resource information access method, device, equipment and medium |
| CN113760931A (en) * | 2021-08-20 | 2021-12-07 | 济南浪潮数据技术有限公司 | Resource information access method, device, equipment and medium |
| CN113626443B (en) * | 2021-08-26 | 2024-03-15 | 企查查科技股份有限公司 | Index data processing method, device, computer equipment and storage medium |
| CN113626443A (en) * | 2021-08-26 | 2021-11-09 | 企查查科技有限公司 | Index data processing method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021012553A1 (en) | 2021-01-28 |
| CN110489417B (en) | 2023-03-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110489417A (en) | A kind of data processing method and relevant device | |
| US11394805B1 (en) | Automatic discovery of API information | |
| WO2021012568A1 (en) | Data processing method and related device | |
| CN101408899B (en) | Method and apparatus for switching website multiple data sources | |
| CN110896412B (en) | Network request processing method and device and electronic equipment | |
| WO2019192103A1 (en) | Concurrent access control method and apparatus, terminal device, and medium | |
| CN108509523A (en) | Structuring processing method, equipment and the readable storage medium storing program for executing of block chain data | |
| CN110049503B (en) | Method and equipment for acquiring data | |
| CN111258627B (en) | Interface document generation method and device | |
| CN112887121B (en) | Method, equipment and system for managing network service | |
| CN109635550B (en) | Permission verification method, gateway and system for cluster data | |
| CN110247857B (en) | Current limiting method and device | |
| CN114553960A (en) | Data caching method, device, equipment and storage medium | |
| CN108595505A (en) | The method and apparatus of data query | |
| WO2021017907A1 (en) | Method and device for optimized inter-microservice communication | |
| CN105592141B (en) | A kind of connection number control method and device | |
| US20100036892A1 (en) | Determination of an updated data source from disparate data sources | |
| CN108600349B (en) | Connection management method and device in connection pool | |
| US20100218202A1 (en) | Method and Apparatus for Case-Based Service Composition | |
| WO2019006998A1 (en) | Node.js authority control method, storage medium, electronic device, and system | |
| CN105991568A (en) | Proxy realizing device | |
| CN110347659A (en) | Data processing method and device, transaction data processing method and device | |
| KR101888131B1 (en) | Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool | |
| JP2000235497A (en) | Policy macro method and storage medium stored with policy macro program is stored | |
| CN114780267B (en) | Interface calling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |