CN114329146A - Index document construction method and system - Google Patents
Index document construction method and system Download PDFInfo
- Publication number
- CN114329146A CN114329146A CN202210016326.8A CN202210016326A CN114329146A CN 114329146 A CN114329146 A CN 114329146A CN 202210016326 A CN202210016326 A CN 202210016326A CN 114329146 A CN114329146 A CN 114329146A
- Authority
- CN
- China
- Prior art keywords
- data
- index document
- module
- index
- constructing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides an index document construction method and system. The system comprises a data acquisition module and an index document construction module, wherein the data acquisition module is provided with indication information for indicating data acquisition, so that the data acquisition module can acquire original data according to the indication information, and the index document construction module can construct an index document according to the original data, thereby solving the problems in the prior art.
Description
Technical Field
The application relates to the technical field of information search, in particular to an index document construction method and system.
Background
With the rapid development of network technology, the amount of information on a network has also increased dramatically. In order to find desired network information among a huge amount of network information, a user may generally search using a search engine. In the search process, the search engine needs to obtain a search result based on a pre-constructed index document. Therefore, how to construct the index document is the key to the search engine to be able to obtain the search results.
Disclosure of Invention
The embodiment of the application aims to provide a method and a system for generating an index document, which are used for generating the index document.
A first aspect of an embodiment of the present application provides an index document building system, including a data acquisition module and an index document building module, where the data acquisition module is provided with instruction information for instructing to perform data acquisition, where:
the data acquisition module is used for acquiring original data according to the set indication information;
and the index document construction module is used for constructing an index document according to the original data.
In an embodiment, the indication information specifically includes:
a data source and a query statement for performing a data query from the data source; or the like, or, alternatively,
a storage path for a data collection scripting tool.
In an embodiment, the index document construction module specifically includes a data processing sub-module and an index document first construction sub-module, wherein:
the data processing submodule is used for processing the data of the original data, and the data processing specifically comprises any one or more of data cleaning, data conversion and data processing;
the first construction submodule of the index document is used for constructing the index document according to the original data processed by the data processing.
In an embodiment, the index document constructing module further includes a judging sub-module and an index document second constructing sub-module, where:
the judgment submodule is used for judging whether the original data conforms to the field format of the corresponding index field in the search engine; the second index document constructing submodule is used for constructing an index document according to the original data under the condition that the original data accords with the field format of the corresponding index field in the search engine; and the number of the first and second groups,
the data processing submodule is used for processing the data of the original data under the condition that the original data does not conform to the field format of the corresponding index field in the search engine; the first construction submodule of the index document is used for constructing the index document according to the original data processed by the data processing.
In one embodiment, the system further comprises: and the sending module is used for sending the index document to the search engine.
In one embodiment, the system further comprises: and the first setting module is used for setting the indication information in the data acquisition module according to the input data of the graphical user interface.
In one embodiment, the system further comprises: the second setting module is used for setting a timing task for constructing the index document; then the process of the first step is carried out,
the data acquisition module is configured to acquire original data according to the set indication information, and specifically includes:
and the data acquisition module is used for acquiring original data according to the set indication information under the condition that the timing task is triggered.
In one embodiment, the system further comprises: and the instance generation module is used for generating an instantiation script for constructing the index document according to the setting of the first setting module and/or the second setting module.
A second aspect of the embodiments of the present application provides a method for constructing an index document, where the method includes:
running an instantiation script used for constructing an index document, wherein the instantiation script comprises indicating information used for indicating data acquisition;
acquiring original data according to the indication information;
and constructing an index document according to the original data.
In one embodiment, the method further comprises: obtaining the instantiation script from an instantiation library; or the like, or, alternatively,
and generating the instantiation script.
By adopting the index document construction system provided by the embodiment of the application, the system comprises the data acquisition module and the index document construction module, and the data acquisition module is provided with the indication information for indicating data acquisition, so that the data acquisition module can acquire original data according to the indication information, and the index document construction module can construct an index document according to the original data, thereby solving the problems in the prior art.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
FIG. 1 is a schematic structural diagram of an index document construction system according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of an index document construction system according to another embodiment of the present application;
FIG. 3 is a flowchart illustrating a method for constructing an index document according to an embodiment of the present application;
fig. 4 is a schematic flowchart of an index document construction method in a specific application scenario according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the present application, the terms "first," "second," and the like are used solely to distinguish one from another and are not intended to indicate or imply relative importance or precedence.
As mentioned above, the search engine needs to obtain the search result based on the index document constructed in advance during the search process. Therefore, how to construct the index document is the key to the search engine to be able to obtain the search results.
Based on this, the embodiment of the application provides an index document construction system, and the index document construction system can be used for constructing the index document so as to be used for searching of a search engine. Referring to fig. 1, a schematic diagram of the system 10 is shown, the system 10 includes: the index document constructing method comprises a data acquisition module 11 and an index document constructing module 12, wherein the index document constructing module 12 is connected with the data acquisition module 11.
It should be noted that, the data acquisition module 11 is provided with instruction information for instructing data acquisition, and data acquisition can be performed by the instruction of the instruction information, so that the data acquisition module 11 can be used to acquire raw data according to the set instruction information.
In practical applications, the indication information may specifically include: a data source and a query statement for performing a data query from the data source; or, a storage path of the data collection scripting tool.
Since the data structure of the relational database is relatively simple compared with the non-relational database, one or more databases (the type of the database is the relational database) can be used as a data source to obtain the original data from the data source, and the indication information can be the data source and the query statement for performing data query from the data source. For example, a database with a database type of a relational database may be set as a data source, and a query statement may be set to enable the data acquisition module 11 to acquire the query result of the query statement for data in the data source as the acquired original data.
Of course, the indication information set in the data acquisition module 11 may also be a storage path of the data acquisition script tool, so that the data acquisition module 11 can acquire and run the data acquisition script tool through the storage path, thereby acquiring the original data. Generally, in order to facilitate a data acquisition script tool to acquire original data, a data source for data acquisition may be set in a code of the data acquisition script tool, and when a database type of the data source is a relational database, an inquiry statement for the data source needs to be set in the code; or, when the database type of the data source is a non-relational database, an acquisition condition for the non-relational database needs to be set in the code, so as to enable data acquisition.
The index document constructing module 12 in the system 10 can be used for constructing an index document according to the raw data acquired by the data acquiring module 11.
It should be noted that, when constructing an index document from raw data, data processing is generally performed on the raw data, and the data processing may specifically include any one or more of data cleaning, data conversion, and data processing. For example, the data processing of the original data is specifically to perform data cleaning on the original data, delete the original data with missing primary keys through the data cleaning, or fill in missing values of non-primary keys, and the like; or, the data processing of the raw data is specifically to perform data conversion on the raw data, so as to convert or merge the raw data to form a description form suitable for subsequent processing, and the data conversion may generally include any one or more of the following processing contents: smoothing, aggregation, data generalization, normalization, attribute construction, and the like; or, the data processing of the raw data is specifically data processing of the raw data, including data extraction, rearrangement, integration processing, and the like.
Naturally, the data processing may be performed on the original data, or the data processing may be performed on the original data after the data cleaning is performed on the original data, and then the data conversion and the data processing are performed respectively; or, the data processing is carried out on the original data, or the data cleaning is carried out on the original data firstly, and then the data conversion is carried out; or, the data processing is carried out on the original data, or the data processing can be carried out after the data cleaning is carried out on the original data; or, the data processing is performed on the original data, or the data processing may be performed after the data conversion is performed on the original data; or the data processing may be performed on the original data, or other manners, which are not described one by one here.
The format of the original data processed by the data processing is more uniform, and the original data lacking the main key and the like are deleted, so that the efficiency of subsequently constructing the index document can be improved. Therefore, as shown in fig. 2, in practical applications, the index document building module 12 may specifically include a data processing sub-module 121 and an index document first building sub-module 122, where: the data processing sub-module 121 can be used to perform data processing on the raw data. The first index document constructing submodule 122 can be configured to construct an index document according to the raw data after the data processing.
For example, the data processing sub-module 121 may further include a data cleaning sub-unit, a data converting sub-unit and/or a data processing sub-unit, which are respectively used for cleaning, converting and processing data.
By adopting the index document construction system provided by the embodiment of the application, the system comprises the data acquisition module and the index document construction module, and the data acquisition module is provided with the indication information for indicating data acquisition, so that the data acquisition module can acquire original data according to the indication information, and the index document construction module can construct an index document according to the original data, thereby solving the problems in the prior art.
In addition, the index document construction system provided by the embodiment of the application configures and componentizes the construction process of the index document through each module, so that the software development cost can be reduced, and the development efficiency can be improved.
Because the constructed index document usually needs to be applied to a search engine, when the index document is constructed according to the original data, the original data collected by the data collection module 11 can be judged by combining the field format of the corresponding index field in the search engine, for example, whether the original data conforms to the field format of the corresponding index field in the search engine can be judged, and in the case of conformity, the original data can be directly applied to construct the index document, or in the case of non-conformity, the original data needs to be processed, so that the original data after data processing can conform to the field format of the corresponding index field in the search engine, and then be used for constructing the index document.
Therefore, the index document constructing module 12 may further include a judging sub-module 123 and an index document second constructing sub-module 124, wherein: the judging submodule 123, which can be used to judge whether the original data conforms to the field format of the corresponding index field in the search engine; the data processing submodule 121 is configured to, when the original data does not conform to the field format of the corresponding index field in the search engine, perform data processing on the original data; the first index document constructing submodule 122 can be used for constructing an index document according to the original data after data processing; the index document second constructing sub-module 124 can be used for constructing an index document according to the original data under the condition that the original data conforms to the field format of the corresponding index field in the search engine.
That is to say, after the data acquisition module 11 acquires the original data, the judgment submodule 123 judges whether the original data conforms to the field format of the corresponding index field in the search engine, and if so, the second construction submodule 124 directly constructs an index document by using the original data, or if not, the data processing submodule 121 performs data processing on the original data, and then the first construction submodule 122 of the index document constructs an index document according to the original data after the data processing.
The corresponding configuration information may be set in the determining submodule 123 according to the field format of the corresponding index field in the search engine, so that the determining submodule 123 can determine whether the original data conforms to the field format of the corresponding index field in the search engine based on the configuration information.
Certainly, in order to further improve the efficiency of constructing the index document, corresponding configuration information may also be set in the data acquisition module 11 according to the field format of the corresponding index field in the search engine, so as to instruct the data acquisition module 11 to acquire the raw data, so that the acquired raw data has a higher possibility of conforming to the field format of the corresponding index field in the search engine, and further, the possibility of performing data processing on the raw data is reduced, and therefore, it is more likely that the index document is constructed by directly using the raw data through the second construction sub-module 124.
The system 10 may further include a sending module 13, the sending module 13 being operable to send the constructed index document to a search engine.
When sending the index document, the sending module 13 may send the index document to a search engine associated with the search engine address according to the set search engine address, where the search engine address may be an address of a certain search engine in a distributed cluster, so as to send the index document to the search engine; the search engine address can also be an address corresponding to a plurality of search engines in the same distributed cluster or an address corresponding to a plurality of search engines in a plurality of distributed clusters, so that the index document is sent to a plurality of different search engines, and the sending efficiency is improved in a packaging sending mode.
When the sending module 13 sends a plurality of index documents, for the sending method of the index documents, the index documents may be sent one by one according to the index type and/or the information of the search engine to which the index documents belong, or a predetermined number of index documents may be sent in a package, for example, a plurality of index documents that need to be sent to a certain search engine may be sent in a package according to the identifier of the index document.
In practical applications, in order to facilitate setting the indication information in the data acquisition module 11, a Graphical User Interface (GUI) may be generally provided, so that a User can input data through the GUI, and accordingly, the system 10 may include a first setting module, where the first setting module may be configured to set the indication information in the data acquisition module according to the input data of the GUI.
Generally, the system 10 may further include a second setting module, which can be used to set a timing task for constructing the index document, for example, a periodic timing task for every month, every week or every day is set by the second setting module, or a timing task for a certain time point is set by the second setting module, so that the data collection module 11 can collect the original data according to the set indication information in case that the timing tasks are triggered.
Of course, a second graphical user interface may also be provided, so that a user may input data in the second graphical user interface, for example, input a trigger time point of the timing task, and correspondingly, the second setting module may set the timing task according to the input data of the second graphical user interface.
In addition, the timing task may further include information such as an index type and/or a search engine to which the timing task belongs, for example, the timing task includes the index type, so as to be used for indicating an index document that is finally generated with the index type, and the index type may include a global index, a primary key index, a local index, or the like; or, the timing task includes information of a search engine to which the timing task belongs, so as to indicate that the generated index document belongs to the search engine, and the sending module 13 may also send the index document to the search engine.
It should be further noted that the system 10 may further include a third setting module, and the third setting module can be used to set data processing on the raw data, such as setting data cleaning or data conversion on the raw data. Of course, a third graphical user interface may also be provided, so that a user may input data in the third graphical user interface, and the third setting module may be configured to perform data processing on the original data according to the input data of the third graphical user interface.
In practical applications, in order to improve the efficiency of constructing the index document, an instantiation script for constructing the index document may be generated in advance, so that the index document may be constructed by running the instantiation script. Accordingly, the system 10 may further include: and the instance generation module is used for generating the instantiation script according to the setting of the first setting module, the second setting module and/or the third setting module.
The generated instantiation script can be stored in an instantiation library, so that when the index document needs to be built, the instantiation script can be obtained from the instantiation library and used for building the index document. For example, a plurality of instantiation scripts can be generated in advance through the instance generation module, and are stored in the instance library, when the index document needs to be constructed, the corresponding instantiation scripts can be screened from the instance library, and the index document can be constructed by running the instantiation scripts.
In addition, in practical applications, the technical effects obtained by combining the index document construction system provided by the embodiment of the present application with hardware devices also belong to the protection scope of the embodiment of the present application. For example, different modules in the index document construction system provided in the embodiment of the present application may be respectively arranged at different nodes in the distributed cluster, so that the functions of the index document construction system are realized through cooperation between the nodes, that is, construction of the index document is performed.
Based on the same inventive concept as the index document construction system provided by the embodiment of the present application, the embodiment of the present application also provides an index document construction method, and for the embodiment of the method, if there is ambiguity, reference may be made to relevant contents of other embodiments. Fig. 3 shows a specific flow diagram of the method, which includes the following steps:
step S21: acquiring original data according to indication information for indicating data acquisition;
step S22: and constructing an index document according to the original data.
Because the index document construction method is based on the same inventive concept as the index document construction system provided by the embodiment of the application, the problems in the prior art can be solved, and the description is omitted here.
In practical application, in order to improve the efficiency of constructing the index document, the index document can be constructed by running an instantiation script for constructing the index document. Therefore, in the method, step S23 may be further included: and running an instantiation script for constructing the index document, wherein the citation document instantiation script comprises indicating information for indicating data collection. Therefore, the method may first execute step S23 to run the instantiation script, and then execute step S21 and step S22.
Of course, for the executed instantiation script, the instantiation script may be obtained from the instance library, such as the instantiation script obtained from the instance library first, and then the step S23 is executed to execute the instantiation script; of course, the executed instantiation script is also an instantiation script generated directly, for example, when the required instantiation script is not stored in the instantiation library, the instantiation script can be generated, and then the index document is constructed by executing the instantiation script.
The specific way of running the instantiation script can be that the instantiation script is run under the trigger of the timing task, or the instantiation script is run according to the execution instruction input by the user.
Of course, after the index document is constructed, the index document may also be sent to the search engine. The sending mode of the index documents can be one by one, or a plurality of index documents can be sent in a packaging way, or the index documents can be sent to a single search engine, or sent to a plurality of different search engines.
The above is a specific description of the index document construction system and method provided in the embodiments of the present application, and for convenience of understanding, the embodiments of the present application may further describe the system and method in combination with a specific application scenario. In the application scene, the system can comprise a data acquisition module, an index document construction module, a sending module, a first setting module, a second setting module and a third setting module; the data acquisition module comprises a data processing submodule, an index document first construction submodule, a judgment submodule and an index document second construction submodule.
Preprocessing can be performed, for example, the setting is performed through the first setting module, the second setting module and the third setting module, for example, the timing task is set through the second setting module according to the input data of the user on the second graphical user interface; the first setting module sets the indication information in the data acquisition module according to the input data of the user on the graphical user interface; and the third setting module is used for processing the data of the original data. As shown in fig. 4, in the application scenario, the index document construction method includes the following steps:
step S31: and under the condition that the timing task is triggered, the data acquisition module acquires original data according to the set indication information.
Step S32: the determining submodule determines whether the original data matches the field format of the corresponding index field in the search engine, if yes, then step S33 is executed, and if not, then step S34 is executed.
Step S33: and the second construction submodule of the index document constructs the index document according to the original data.
Step S34: and the data processing submodule performs data processing on the original data.
Step S35: and the first construction submodule of the index document constructs the index document according to the original data after the data processing.
Step S36: the sending module sends the constructed index document to a search engine.
Based on the same inventive concept as the method embodiment of the present application, an embodiment of the present application further provides an electronic device, as shown in fig. 5, which is a schematic structural diagram of the electronic device 40, and includes a processor 401 and a memory 402, where the memory 402 stores a program or an instruction that can be executed on the processor 401, and when the program or the instruction is executed by the processor 401, the steps of the method embodiment of the present application are implemented, and the same technical effects can be achieved, and in order to avoid repetition, details are not repeated here.
It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices. For example, in one embodiment, the electronic device 40 may be a mobile phone, a notebook computer, a desktop computer, or a large server thereof.
An embodiment of the present invention further provides a non-transitory electronic device readable storage medium, including: a program that, when run on an electronic device, causes the electronic device to perform all or part of the procedures of the methods in the above-described embodiments. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like. The storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. The index document construction system is characterized by comprising a data acquisition module and an index document construction module, wherein the data acquisition module is provided with indication information used for indicating data acquisition, and the index document construction system comprises:
the data acquisition module is used for acquiring original data according to the set indication information;
and the index document construction module is used for constructing an index document according to the original data.
2. The system according to claim 1, wherein the indication information specifically includes:
a data source and a query statement for performing a data query from the data source; or the like, or, alternatively,
a storage path for a data collection scripting tool.
3. The system according to claim 1, wherein the index document construction module specifically includes a data processing sub-module and an index document first construction sub-module, wherein:
the data processing submodule is used for processing the data of the original data, and the data processing specifically comprises any one or more of data cleaning, data conversion and data processing;
the first construction submodule of the index document is used for constructing the index document according to the original data processed by the data processing.
4. The system of claim 3, wherein the index document building module further comprises a judgment sub-module and an index document second building sub-module, wherein:
the judgment submodule is used for judging whether the original data conforms to the field format of the corresponding index field in the search engine; the second index document constructing submodule is used for constructing an index document according to the original data under the condition that the original data accords with the field format of the corresponding index field in the search engine; and the number of the first and second groups,
the data processing submodule is used for processing the data of the original data under the condition that the original data does not conform to the field format of the corresponding index field in the search engine; the first construction submodule of the index document is used for constructing the index document according to the original data processed by the data processing.
5. The system of claim 4, further comprising: and the sending module is used for sending the index document to the search engine.
6. The system of claim 1, further comprising: and the first setting module is used for setting the indication information in the data acquisition module according to the input data of the graphical user interface.
7. The system of claim 6, further comprising: the second setting module is used for setting a timing task for constructing the index document; then the process of the first step is carried out,
the data acquisition module is configured to acquire original data according to the set indication information, and specifically includes:
and the data acquisition module is used for acquiring original data according to the set indication information under the condition that the timing task is triggered.
8. The system of claim 7, further comprising: and the instance generation module is used for generating an instantiation script for constructing the index document according to the setting of the first setting module and/or the second setting module.
9. A method for constructing an index document, the method comprising:
running an instantiation script used for constructing an index document, wherein the instantiation script comprises indicating information used for indicating data acquisition;
acquiring original data according to the indication information;
and constructing an index document according to the original data.
10. The method of claim 9, further comprising: obtaining the instantiation script from an instantiation library; or the like, or, alternatively,
and generating the instantiation script.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210016326.8A CN114329146A (en) | 2022-01-07 | 2022-01-07 | Index document construction method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210016326.8A CN114329146A (en) | 2022-01-07 | 2022-01-07 | Index document construction method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114329146A true CN114329146A (en) | 2022-04-12 |
Family
ID=81024570
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210016326.8A Pending CN114329146A (en) | 2022-01-07 | 2022-01-07 | Index document construction method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114329146A (en) |
-
2022
- 2022-01-07 CN CN202210016326.8A patent/CN114329146A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209652B (en) | Data table migration method, device, computer equipment and storage medium | |
| CN112910945B (en) | Request link tracking method and service request processing method | |
| CN103210386B (en) | Method, system and the equipment of conglomeration search | |
| CN102135985B (en) | Method and system for searching by calling search result of third-party search engine | |
| CN111258978B (en) | Data storage method | |
| KR101355273B1 (en) | A computing system, a method for controlling thereof, and a computer-readable recording medium having a computer program for controlling thereof | |
| CN109389299B (en) | Workflow process deployment method, workflow process deployment device, computer equipment and storage medium | |
| CN104035938A (en) | Performance continuous integration data processing method and device | |
| US20200142674A1 (en) | Extracting web api endpoint data from source code | |
| US10506392B1 (en) | Stream-processing of telecommunication diameter event records | |
| CN111818175A (en) | Enterprise service bus configuration file generation method, device, equipment and storage medium | |
| CN114356971A (en) | Data processing method, device and system | |
| CN110780950B (en) | Interface metadata management method, device, computer equipment and storage medium | |
| CN110825594A (en) | Data reporting and issuing method, client and server | |
| CN114238703A (en) | Event flow arrangement method, device and application | |
| CN112650529B (en) | System and method for configurable generation of mobile terminal APP codes | |
| CN108959294B (en) | Method and device for accessing search engine | |
| CN112507010B (en) | Service data processing method and device, computer equipment and storage medium | |
| CN112035466B (en) | External index development framework for block chain query | |
| CN114281320A (en) | Plug-in arrangement method and system in API gateway | |
| CN114329146A (en) | Index document construction method and system | |
| CN115858322A (en) | Log data processing method and device and computer equipment | |
| CN107977381B (en) | Data configuration method, index management method, related device and computing equipment | |
| CN114371884A (en) | Method, device, equipment and storage medium for processing Flink calculation task | |
| CN112035413B (en) | Metadata information query method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |