CN109815261B - Global search function implementation and data real-time synchronization method and device and electronic equipment - Google Patents
Global search function implementation and data real-time synchronization method and device and electronic equipment Download PDFInfo
- Publication number
- CN109815261B CN109815261B CN201811510072.5A CN201811510072A CN109815261B CN 109815261 B CN109815261 B CN 109815261B CN 201811510072 A CN201811510072 A CN 201811510072A CN 109815261 B CN109815261 B CN 109815261B
- Authority
- CN
- China
- Prior art keywords
- configuration information
- database
- data
- index
- search engine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a method and a device for realizing a global search function and synchronizing data in real time and electronic equipment. The method comprises the steps of creating a database table, constructing a mapping relation between the database table and a search engine document set, and generating configuration information; storing the configuration information to a memory; generating an index library and a search interface of the search engine according to the configuration information; and storing the instantly updated data to the index library based on the configuration information and providing the instantly updated data to the search interface for calling. The method, the device and the electronic equipment can realize the global search function and meet the data timeliness.
Description
Technical Field
The present invention relates to a method and an apparatus for implementing a global search function and synchronizing data in real time, and an electronic device.
Background
A Search Engine (Search Engine) is a system that collects information from the internet by using a specific computer program according to a certain policy, organizes and processes the information, provides a Search service for a user, and displays information related to user Search to the user. The global search is used as a quick entrance of an application, so that a user can quickly find out a desired function or data through a keyword, and then related operations can be conveniently completed based on search results. The dependence of users on global search is stronger and stronger, so the global search also becomes an indispensable core function in enterprise applications. How to meet the changing global search requirement, rapidly develop and deliver the increasing global search target range and data timeliness becomes the key points supported by a development platform.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method, an apparatus, and an electronic device for implementing a global search function and synchronizing data in real time, so as to implement the global search function and meet data timeliness.
Based on the above object, an embodiment of the present invention provides a method for implementing a global search function and synchronizing data in real time, including:
creating a database table, constructing a mapping relation between the database table and a search engine document set, and generating configuration information;
storing the configuration information to a memory;
generating an index library and a search interface of the search engine according to the configuration information;
and calling the configuration information, storing the instantly updated data to the index library based on the configuration information, and providing the instantly updated data to the search interface for calling.
Optionally, the constructing a mapping relationship between the database table and the search engine document set includes:
analyzing the structure of the database table to obtain the database name, the table name, the field name, the attribute name and the attribute type contained in the database table;
converting the identified database name, table name, field name, attribute type into a field matching the document set of the search engine.
Optionally, the configuration information is a configuration file, and the mapping relationship includes a mapping relationship between the database table and an ES index object, and a mapping relationship between a table field and an index object of the search engine.
Optionally, the configuration information is a database configuration table, and the mapping relationship includes a mapping relationship between the database table and an ES index structure, and a mapping relationship between a table field and an index structure of the search engine.
Optionally, the storing the configuration information in the memory includes:
acquiring a system starting request, and calling a configuration updating task;
reading the configuration information according to the configuration updating task;
and synchronously updating the configuration information to an in-memory database for storage.
Optionally, the method further includes: and reading the whole or part of the configuration information according to the configuration updating task.
Optionally, the storing the update data to the index repository based on the configuration information includes:
monitoring the log data of the database, and judging whether the database table is updated or not;
and if so, acquiring the updating data in the database table, and synchronizing the updating data into the index database.
Optionally, the obtaining the update data in the database table and synchronizing the update data to the index database includes:
parsing the update data into a table entity object corresponding to the configuration information;
distributing the table entity object to message queues corresponding to different operation modes through a dispatcher of the search engine;
and storing the table entity object into the index library according to the ID information.
The embodiment of the present invention further provides a device for implementing the global search function and synchronizing data in real time, including:
the system comprises a construction module, a search engine document collection module and a configuration module, wherein the construction module is used for creating a database table, constructing a mapping relation between the database table and the search engine document collection and generating configuration information;
the first storage module is used for storing the configuration information to a memory;
the generating module is used for generating an index library and a search interface of the search engine according to the configuration information;
and the second storage module is used for calling the configuration information, storing the immediately updated data to the index library based on the configuration information and providing the immediately updated data for the search interface to be called.
An embodiment of the present invention further provides an electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a global search function implementation and a data real-time synchronization method as described in any one of the above.
As can be seen from the foregoing, according to the method, apparatus and electronic device for implementing a global search function and synchronizing data in real time provided by the embodiments of the present invention, a configuration file is generated by constructing a mapping relationship between a database table and a search engine document set, and is stored in a memory, and when the configuration file is used, the configuration information is directly called from the memory to synchronize update data in real time, so that the global search function and the real-time synchronization of data are quickly implemented, and the timeliness of global search is ensured.
Drawings
FIG. 1 is a flowchart of a method for implementing a global search function and synchronizing data in real time according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an embodiment of storing configuration information in a memory according to the present invention;
FIG. 3 is a schematic diagram of an embodiment of real-time data synchronization according to the present invention;
FIG. 4 is a diagram of a global search function implementation and data real-time synchronization apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
Fig. 1 is a flowchart of a method for implementing a global search function and synchronizing data in real time according to an embodiment of the present invention.
The embodiment of the invention provides a method for realizing a global search function and synchronizing data in real time, which comprises the following steps:
In the embodiment, the configuration file is generated by constructing the mapping relation between the database table and the search engine document set and is stored in the memory, and when the configuration file is used, the configuration information is directly called from the memory to perform real-time synchronization on the updated data, so that the real-time synchronization of the global search function and the data is quickly realized, and the timeliness of global search is ensured.
In this embodiment, the search engine may select elasticsearch (es).
In some embodiments of the present invention, the constructing the mapping relationship between the database table and the search engine document set includes:
step 101, analyzing the structure of the database table, and acquiring the database name, table name, field name, attribute type, and the like contained in the database table.
Step 102, converting the identified database name, table name, field name, attribute name and attribute type into fields matched with the document set of the search engine.
In some embodiments of the invention, the configuration information may be a configuration file or a database configuration table.
If the configuration information is a configuration file, the mapping relationship includes a mapping relationship between the database table and an ES index object and a mapping relationship between a table field and an index object of the search engine. The specific configuration file format is as follows:
and if the configuration information is a database configuration table, the mapping relationship comprises the mapping relationship between the database table and an ES index structure and the mapping relationship between a table field and the index structure of the search engine. The database configuration table is designed as follows:
table name: rzl _ config _ table2es
The following steps are described: mapping relation between table and es index structure
Table name: rzl _ config _ tablefield2es
The following steps are described: mapping relation between table field and es index structure
In other embodiments of the present invention, the storing the configuration information in the memory in step 200 includes:
step 201, acquiring a system start request, and calling a configuration update task.
Step 202, reading the configuration information according to the configuration updating task. Specifically, configuration information is loaded based on the access parameters and the system configuration.
Step 203, synchronously updating the configuration information to the memory database for storage. Specifically, the loaded table and the es configuration mapping information are synchronously updated to the memory database redis for being called by other programs of the system.
Optionally, the method further includes: and reading the whole or part of the configuration information according to the configuration updating task.
In a particular embodiment, the configuration information is stored using redis. Referring to fig. 2, when the system is started or manually triggered, after receiving a system parameter initiation change notification, the configuration loading center reads a configuration file or a database configuration table, constructs a configuration object, and then operates a redis, and creates or updates the redis.
In other embodiments of the present invention, the storing update data to the index repository based on the configuration information of step 400 includes:
step 401, monitoring the log data of the database, and determining whether there is an update operation of the database table. Optionally, the log data is a binlog log.
Step 402, if yes, acquiring the update data in the database table, and synchronizing the update data to the index database.
Optionally, the obtaining the update data in the database table and synchronizing the update data to the index database in step 402 includes:
step 4021, parsing the update data into table entity objects corresponding to the configuration information.
Step 4022, distributing the table entity object to message queues corresponding to different operation modes through the dispatcher of the search engine. The operation modes comprise adding, modifying, deleting and the like. A data, if associated with multiple topics, is sent to multiple corresponding message channels.
Step 4023, storing the table entity object into the index library according to the ID information. Specifically, after a program monitoring the message discovers a new database table operation object, the program updates data (newly added, modified or deleted) in the corresponding index object according to the subject id and the id of the current data, and then provides the updated data to the search interface for calling.
In a specific embodiment, referring to fig. 3, a cancer monitors a database, and when the cancer receives a binlog log updated by the database, a global search engine loads configuration information, converts updated data into an Elasticsearch document object according to a mapping relationship in the configuration information, and then operates an Elasticsearch or newly creates an updated index set based on ID information.
In an alternative embodiment, the global search engine restful api includes:
another aspect of the embodiments of the present invention further provides a device for implementing a global search function and synchronizing data in real time, which is shown in fig. 4 and includes:
the construction module 11 is configured to create a database table, construct a mapping relationship between the database table and a search engine document set, and generate configuration information.
The first storage module 12 is configured to store the configuration information in a memory.
A generating module 13, configured to generate an index library and a search interface of the search engine according to the configuration information;
and the second storage module 14 is used for calling the configuration information, storing the instantly updated data to the index library based on the configuration information and providing the instantly updated data for the search interface to be called.
In a third aspect of the embodiments of the present invention, an embodiment of an apparatus for performing the global search function implementation and the real-time data synchronization method is provided. Fig. 5 is a schematic diagram of a hardware structure of an embodiment of the apparatus for executing the global search function implementation and the data real-time synchronization method according to the present invention.
As shown in fig. 5, the apparatus includes:
one or more processors 901 and a memory 902, with one processor 901 being an example in fig. 5.
The apparatus for performing the data recommendation method may further include: an input device 903 and an output device 904.
The processor 901, the memory 902, the input device 903 and the output device 904 may be connected by a bus or other means, and fig. 5 illustrates the connection by a bus as an example.
The memory 902, which is a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the global search function implementation and the data real-time synchronization method in the embodiment of the present application (for example, the building module 11, the first storage module 12, the generating module 13, and the second storage module 14 shown in fig. 4). The processor 901 executes various functional applications and data processing of the server by running nonvolatile software programs, instructions and modules stored in the memory 1002, that is, implementing the global search function implementation and the data real-time synchronization method of the above method embodiments.
The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the global search function implementation and the use of the data real-time synchronization apparatus, and the like. Further, the memory 902 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 902 may optionally include memory located remotely from processor 901, which may be connected to the member user behavior monitoring device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 903 may receive input numeric or character information and generate key signal inputs related to the global search function implementation and user setting and function control of the data real-time synchronization apparatus. The output device 904 may include a display device such as a display screen.
The one or more modules are stored in the memory 902 and when executed by the one or more processors 901, perform the global search function implementation and data real-time synchronization method in any of the above method embodiments. The technical effect of the embodiment of the device for implementing the global search function and the real-time data synchronization method is the same as or similar to that of any method embodiment.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the idea of the invention, also features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity.
In addition, well known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures for simplicity of illustration and discussion, and so as not to obscure the invention. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the invention, and also in view of the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the present invention is to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the invention, it should be apparent to one skilled in the art that the invention can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
The embodiments of the invention are intended to embrace all such alternatives, modifications and variances that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements and the like that may be made without departing from the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (7)
1. A global search function implementation and data real-time synchronization method is characterized by comprising the following steps:
creating a database table, constructing a mapping relation between the database table and a search engine document set, and generating configuration information;
storing the configuration information to a memory, including:
acquiring a system starting request, and calling a configuration updating task;
reading the configuration information according to the configuration updating task;
synchronously updating the configuration information to a memory database for storage;
generating an index library and a search interface of the search engine according to the configuration information;
calling the configuration information, storing the instantly updated data to the index library based on the configuration information and providing the instantly updated data to a search interface for calling;
wherein the storing of the update data to the index repository based on the configuration information comprises:
monitoring the log data of the database, and judging whether the database table is updated or not;
if yes, acquiring the updating data in the database table, and synchronizing the updating data to the index database, wherein the steps of:
parsing the update data into a table entity object corresponding to the configuration information;
distributing the table entity object to message queues corresponding to different operation modes through a dispatcher of the search engine;
and storing the table entity object into the index library according to the ID information.
2. The method of claim 1, wherein constructing the mapping of the database table to the set of search engine documents comprises:
analyzing the structure of the database table to obtain the database name, the table name, the field name, the attribute name and the attribute type contained in the database table;
converting the identified database name, table name, field name, attribute type into a field matching the document set of the search engine.
3. The method of claim 2, wherein the configuration information is a configuration file, and the mapping relationship comprises a mapping relationship between the database table and an ES index object and a mapping relationship between a table field and an index object of the search engine.
4. The method of claim 2, wherein the configuration information is a database configuration table, and the mapping relationship comprises a mapping relationship between the database table and an ES index structure and a mapping relationship between a table field and an index structure of the search engine.
5. The method of claim 1, further comprising: and reading the whole or part of the configuration information according to the configuration updating task.
6. A global search function implementation and data real-time synchronization device is characterized by comprising:
the system comprises a construction module, a search engine document collection module and a configuration module, wherein the construction module is used for creating a database table, constructing a mapping relation between the database table and the search engine document collection and generating configuration information;
the first storage module is configured to store the configuration information in a memory, and includes:
acquiring a system starting request, and calling a configuration updating task;
reading the configuration information according to the configuration updating task;
synchronously updating the configuration information to a memory database for storage;
the generating module is used for generating an index library and a search interface of the search engine according to the configuration information;
the second storage module is used for calling the configuration information, storing the instantly updated data to the index library based on the configuration information and providing the instantly updated data for a search interface to be called; wherein the storing of the update data to the index repository based on the configuration information comprises:
monitoring the log data of the database, and judging whether the database table is updated or not;
if yes, acquiring the updating data in the database table, and synchronizing the updating data to the index database, wherein the steps of:
parsing the update data into a table entity object corresponding to the configuration information;
distributing the table entity object to message queues corresponding to different operation modes through a dispatcher of the search engine;
and storing the table entity object into the index library according to the ID information.
7. An electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the global search function implementation and the data real-time synchronization method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811510072.5A CN109815261B (en) | 2018-12-11 | 2018-12-11 | Global search function implementation and data real-time synchronization method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811510072.5A CN109815261B (en) | 2018-12-11 | 2018-12-11 | Global search function implementation and data real-time synchronization method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109815261A CN109815261A (en) | 2019-05-28 |
| CN109815261B true CN109815261B (en) | 2021-11-02 |
Family
ID=66602867
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811510072.5A Active CN109815261B (en) | 2018-12-11 | 2018-12-11 | Global search function implementation and data real-time synchronization method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109815261B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111026727A (en) * | 2019-10-24 | 2020-04-17 | 贝壳技术有限公司 | Table dimension retrieval data synchronization method, system and device based on log file |
| CN110941641B (en) * | 2019-11-15 | 2022-06-21 | 南威软件股份有限公司 | Method for searching data across multiple databases |
| CN111142964A (en) * | 2019-12-16 | 2020-05-12 | 苏宁云计算有限公司 | Loading method and system of configuration file information |
| CN111984660A (en) * | 2020-08-14 | 2020-11-24 | 上海好屋网信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN111914066B (en) * | 2020-08-17 | 2024-02-02 | 山东合天智汇信息技术有限公司 | Global searching method and system for multi-source database |
| CN112100275A (en) * | 2020-09-02 | 2020-12-18 | 上海微亿智造科技有限公司 | Data synchronization method, system and electronic equipment |
| CN112507200A (en) * | 2020-12-28 | 2021-03-16 | 浪潮云信息技术股份公司 | Method and apparatus for synchronizing data into search engine |
| CN112817989B (en) * | 2021-01-22 | 2023-07-25 | 网易（杭州）网络有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN114327385B (en) * | 2021-12-03 | 2023-05-02 | 南京行者易智能交通科技有限公司 | Method and device for realizing service data search based on Java reflection and Spring container |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005332521A (en) * | 2004-05-21 | 2005-12-02 | Toshiba Corp | Information recording medium and information reproducing device |
| CN101676899A (en) * | 2008-09-18 | 2010-03-24 | 上海宝信软件股份有限公司 | Profiling and inquiring method for massive database records |
| CN102200974A (en) * | 2010-03-25 | 2011-09-28 | 北京师范大学 | Unified information retrieval intelligent agent system and method for search engine |
| CN102915377B (en) * | 2012-11-14 | 2016-08-03 | 深圳市宏电技术股份有限公司 | Data base's conversion or synchronous method and system |
| CN106611000A (en) * | 2015-10-26 | 2017-05-03 | 北京神州泰岳软件股份有限公司 | Method, device and system for searching resource object |
| CN106095911A (en) * | 2016-06-07 | 2016-11-09 | 腾讯科技（深圳）有限公司 | Search system and method for data synchronization |
| US10423594B2 (en) * | 2016-11-28 | 2019-09-24 | Atlassian Pty Ltd | Systems and methods for indexing source code in a search engine |
| CN108446323A (en) * | 2018-02-11 | 2018-08-24 | 山东省农业信息中心 | A kind of data retrieval method and device based on full-text search engine |
-
2018
- 2018-12-11 CN CN201811510072.5A patent/CN109815261B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109815261A (en) | 2019-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109815261B (en) | Global search function implementation and data real-time synchronization method and device and electronic equipment | |
| EP3382578A1 (en) | Two-dimensional code processing method and terminal | |
| US20170277526A1 (en) | Software categorization method and electronic device | |
| CN103136342A (en) | Searching method, system and searching server of application programs (APP) | |
| CN108958826B (en) | Method and device for dynamically configuring application installation package | |
| CN110941779A (en) | Page loading method and device, storage medium and electronic equipment | |
| CN113760722A (en) | Test system and test method | |
| CN106716418B (en) | Software recommendation method and device, terminal and server | |
| CN110399459B (en) | Online document searching method, device, terminal, server and storage medium | |
| KR102205686B1 (en) | Method and apparatus for ranking candiate character and method and device for inputting character | |
| CN112115103B (en) | File address display method and device, electronic equipment and readable storage medium | |
| CN116743785A (en) | Cloud network data storage method, device, equipment and medium based on fog calculation | |
| CN112788090A (en) | Network resource transmission method, device and system | |
| CN111352993A (en) | Cross-platform data synchronization method and device and electronic equipment | |
| CN111241341A (en) | Video identification information processing method and video searching method, device and server | |
| CN115470754A (en) | Data analysis method and device, computer readable medium and electronic equipment | |
| CN110780898B (en) | Page data upgrading method and device and electronic equipment | |
| CN113360704A (en) | Voice playing method and device and electronic equipment | |
| CN111737218A (en) | File sharing method and device | |
| CN112835671A (en) | Application page scene switching method, device and equipment | |
| CN112416401B (en) | Data updating method, device and equipment | |
| CN113612676B (en) | Social group message synchronization method, device, equipment and storage medium | |
| US11625383B2 (en) | Information processing apparatus, information processing method, and program | |
| CN113568803B (en) | Method, electronic device and computer program product for monitoring a storage system | |
| CN112084440B (en) | Data verification method, device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 1002-1, 10th floor, No.56, Beisihuan West Road, Haidian District, Beijing 100080 Applicant after: Ronglian Technology Group Co., Ltd Address before: 100080, Beijing, Haidian District, No. 56 West Fourth Ring Road, glorious Times Building, 10, 1002-1 Applicant before: UNITED ELECTRONICS Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |