KR101744017B1 - Method and apparatus for indexing data for real time search - Google Patents

Abstract

A method and apparatus for data indexing for real-time searching are provided. A method of data indexing, comprising: writing a document in a memory to a log file; selecting a quantity of documents including at least a portion of the information read in the log file; Creating a temporary segment for at least one temporary segment, exposing at least one temporary segment to a search of a search engine, and merging a document contained in at least one temporary segment with at least one temporary segment being exposed, ), The deletion request file storing the identifier of the deletion candidate document is generated.

Description

TECHNICAL FIELD [0001] The present invention relates to a data indexing method and apparatus for real-

The present invention relates to a real-time search technology, and more particularly, to a data indexing method and apparatus for real-time search.

An index is a data structure that speeds up operations on tables in the database field. Indexes are usually created using a single column or multiple columns in a table. By using this index structure well, it is possible to perform high-speed search operation as well as efficient ordering operation related to record access in data retrieval.

The disk space for storing the index is generally smaller than the disk space required for storing the table. That is because the table has other details but the index only has the key field. In relational databases, an index can be a single copy of a table part.

The index is also used to set unique constraints on the data. Since unique indexes prevent duplicate entries from being registered, uniqueness is ensured in the table being indexed.

BACKGROUND ART [0002] Recently, with the development of technology fields such as electricity, electronics, and communication, a large amount of data is being generated in the Internet, wired and wireless communication, and the like. In particular, companies providing social network services or portal services on the Internet can use large-scale fixed or unstructured data sets (hereinafter referred to as "big data") that are beyond the capability of collecting, storing, managing and analyzing data with existing database management tools. , Or to extract values from big data and analyze the results.

Particularly, as the latest data have a high importance, data related to social network service (SNS), news articles, news comments, and search keywords are required to be efficiently and quickly reflected in the database. In addition, data corresponding to temporal issues, such as SNS-related data, news articles, news comments, and search keywords, often occasionally explode, and a technology capable of indexing data accumulated in preparation for such problems sufficiently fast and stably .

Korean Patent No. 10-0835706 (May 30, 2008)

An object of the present invention is to provide a data indexing method and apparatus for real-time retrieval, which can quickly process data that can not be easily processed by a conventional indexing technique.

It is another object of the present invention to provide a data indexing method and apparatus capable of effectively real-time searching data having high importance of latest data like data.

According to an aspect of the present invention, there is provided a data indexing method for real-time search performed in a computing device, the method comprising: recording a document in a memory as a log file; Selecting a predetermined amount of the document including at least a part of the information read from the log file; Creating at least one temporary segment for the document; Exposing the at least one temporary segment to a search of a search engine; And generating a deletion request file storing an identifier of the deletion candidate document if the document included in the at least one temporary segment is merging while the at least one temporary segment is exposed. A method of data indexing is provided.

Here, the file name of the deletion request file may include a number determined by a temporal fixed value.

Here, the number of reference references may be set in the deletion request file. The reference reference count may be subtracted by a certain number when the deletion request file is referred to in the merging.

Here, the data indexing method may further include, after the generating step, deleting the document corresponding to the deletion request file below the lower limit when the reference number of reference is less than the lower limit by reference at the merging .

Here, the data indexing method may further include, after the exposing step, checking the number of live documents in the exposing step. The range of the number of live documents can be classified into log-realistic ranges.

Here, the data indexing method may further include, after the checking step, concurrently processing the merging indexes for each of the plurality of merging target segments classified into the logarithmic ranges.

Here, the logarithmic ranges may include classification ranges of 0, 100, 1K, 10K, 100K and 1M in the number of documents included in each segment.

Here, the recording step may be such that the document logger attaches a document to a file at each flush period and generates the log file by replacing the file at every rolling cycle.

Here, the document logger may be connected to two memories or memory areas that are replaced each time a flush of reading the log file is performed. The log file is inserted into a file queue, the file queue includes a log file status object, and the log file status object may include a file object and whether it is closed.

Here, the data indexing method may further include obtaining a signal or information for an index request job in the document logger before the writing step.

Here, the selecting may extract a predetermined number or a number of log files according to the indexing capacity or performance of the computing device.

According to another aspect of the present invention, there is provided a data indexing apparatus comprising: a processor; a memory coupled to the processor; and a program stored in the memory, The functions of the scheduler can be performed. The document logger records a document of the memory in a log file; Wherein the dynamic index scheduler selects a quantity of documents containing at least a portion of the information read in the log file, generates at least one temporary segment for the document, exposes the at least one temporary segment to a search of a search engine And generating a deletion request file storing an identifier of the deletion candidate document when the deletion candidate document included in the at least one temporary segment is being merged.

Here, the processor further performs the function of a merging index scheduler by the program, and the merging index scheduler can merge a plurality of index segments dynamically indexed by the dynamic index scheduler. The processor may generate a deletion request file including a temporary ID for the index segment to be dynamically indexed during the merging index operation of the merging index scheduler and including the temporary ID in the file name.

Here, when the processor exposes the second index segment generated by the merging index scheduler to the search, the processor may delete the document included in the first index segment that has already been exposed to the search using the deletion request file.

According to another aspect of the present invention, there is provided a data processing method performed by an indexer including an index scheduler, the index scheduler reading a log file recorded in a memory, Selecting a quantity of documents based on information read from the LOG file; Creating a temporary segment for the document, exposing the temporary segment to a search, and storing the identifier of the document in an erase request file if merge for the temporary segment is in progress or if a merge document exists A method of operating an index structure is provided.

According to another aspect of the present invention, there is provided a data processing method performed by an indexer including a merging scheduler, comprising: a merging scheduler periodically checking a segment list in the index data; Selecting segments to merge among the lists, grouping the segments to be merged by document level, merging the segments by document level, creating a temporary segment by document level for the merged segments, Reading a segment list of the deletion request file at a merging step, and deleting the segment from the temporary segment if the segment belonging to the segment list exists.

According to another aspect of the present invention, there is provided an information processing apparatus including an index request processing unit for writing a document of a storage requested by a log file, an index scheduler for reading the LOG file to perform dynamic indexing, And a merging scheduler for merging the segments selected in the segment list in the data according to the merge policy, wherein the merged segments are generated as a temporary segment and are exposed to the search.

Here, the index structure may correspond to an indexer, an index node, an index server, or an index service device that corresponds to an indexer or performs an index processing method or an indexing operation method.

When the index structure and the operation method for real-time search according to the embodiment of the present invention as described above are used, it is possible to complete the indexing work within a few seconds, for example, within a few seconds as compared with the prior art for a certain amount of big data.

In addition, the new index structure enables quick retrieval of documents such as data, and efficient management of the index structure allows new data to be indexed and merged into existing data indexes. Can contribute to service implementation.

In addition, the latest data such as social network service (SNS) -related data, news articles, news comments, search keywords, and the like can be efficiently and quickly reflected in the database.

In addition, even when data corresponding to temporal issues such as SNS-related data, news articles, news comments, search keywords, etc. are exploding, data can be indexed sufficiently fast and stably.

1 is a schematic flowchart of a data indexing method (hereinafter, simply referred to as 'indexing method') according to an embodiment of the present invention.
2 is a flowchart of an indexing method according to another embodiment of the present invention.
3 is a flowchart of an index request procedure that can be employed in the indexing method according to the present embodiment.
4 is a flowchart of an indexing procedure that can be employed in the indexing method according to the present embodiment.
5 is a flow chart of a merging procedure that can be employed in the indexing method according to the present embodiment.
6 is a view for explaining the operation principle of real-time indexing in the indexing method according to the present embodiment.
7 is a diagram for explaining the operation principle of the document logger of Fig.
FIG. 8 is a diagram for explaining the operation principle of the index scheduler of FIG. 6; FIG.
FIG. 9 is a diagram for explaining the operation principle of the dynamic index module of FIG. 6; FIG.
FIG. 10A is a diagram for explaining the life cycle of the index segment of FIG. 6. FIG.
FIG. 10B is a view for explaining another embodiment of the life cycle of the index segment of FIG. 6. FIG.
FIG. 11 is a diagram illustrating a timeline of the life cycle of the index segment of FIG. 6. FIG.
12 is a block diagram of a data indexing apparatus according to another embodiment of the present invention.
13 is a block diagram of a data indexing apparatus according to another embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms related to "comprising "," having ", and the like are intended to specify the presence of stated features, integers, steps, operations, elements, parts, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Also, in the present specification, when a subscript of a character has different subscripts, other subscripts of the subscript can be displayed in the same size or form as the subscript for convenience of display.

Unless otherwise defined herein, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted in a manner consistent with the contextual meaning of the related art, and are not to be construed as ideal or overly formal, unless explicitly defined herein.

Some of the terms used in this embodiment are defined as follows.

Regarding the term "real-time search ", this embodiment provides a method for searching an indexed document within a few seconds.

The term "node" may refer to an engine daemon process running in the server. A "node" can also refer to the server itself if only one engine is running in the server. For example, the search node may be a search server, and the index node may be an index server.

The term "segment" is a collection of documents of the basic units that constitute the index data and can refer to an actual file directory. Each time the index is executed, the segment can be incremented by one. In this embodiment, at the time of one search, all the segments that are exposed to the current search environment can be searched and all the searched segments can be combined into one end result (another segment).

The term " merging "refers to the process of merging several segments into one. Without merging, too many segments can slow down the search, and disk space may be scarce. Therefore, in the present embodiment, the number of files can be reduced through periodic merging. Such merging can basically have a working principle similar to that of a computer device, but its specific operating principle is different as described in the following detailed description.

The term "dynamic index" refers to an indexing scheme in which some newly updated documents are indexed and pasted into an existing index. With dynamic indexing, it is possible to retrieve documents quickly and accurately by restricting the number of indexed segments from increasing while indexing the latest data periodically.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic flowchart of a data indexing method (hereinafter, simply referred to as 'indexing method') according to an embodiment of the present invention.

Referring to FIG. 1, an indexing method according to an embodiment of the present invention is for processing an index of data for real-time searching. The indexing method first accumulates index-requested documents in a file, reads documents from the file, To complete the operation. Such an indexing method can be implemented by an indexing device including a computing device or a computing device capable of retrieving an indexed document and managing the index file appropriately.

That is, in the indexing method according to the present embodiment, the index request is received for predetermined data, and the index request operation can be executed in the indexing device according to the index request (S11). Particularly, when executing the index request operation, a predetermined capacity or a predetermined number of log files among the log files preset for the entire document to be input in the extraction of the index target document is extracted, thereby effectively performing the subsequent indexing operation or merging operation can do.

Next, the indexed data or document can be dynamically indexed (S12). Dynamic indexing is an indexing method that indexes some newly updated documents and attaches them to existing indexes. By performing such a dynamic indexing operation, it is possible to effectively remove overlapping segments and efficiently control the number of segments exposed to real-time search.

Next, a plurality of segments may be merged during or after the indexing operation or the dynamic indexing operation (S13). By executing such merging operation or merging indexing operation, overlapping segments can be more effectively removed.

In the present embodiment, the indexing method is described as being performed sequentially in the order described in the index requesting operation, indexing operation (or dynamic indexing operation), and merging operation (or merging indexing operation), but the present invention is not limited thereto, Indexing may be omitted.

2 is a flowchart of an indexing method according to another embodiment of the present invention.

Referring to FIG. 2, the indexing method according to the present embodiment can process a plurality of dynamic indexes in the same manner through a plurality of dynamic indexing processes for index-requested data. That is, the first dynamic index S12a and the second dynamic index S12b may be performed asynchronously or independently, and may be concurrently processed according to the implementation.

According to the implementation, the indexing method according to the present embodiment can perform the dynamic indexes S12a and / or S12b and the merging index S13 asynchronously, independently or in parallel.

For example, in the context of a segment, each segment may be performed only an index request (see S11 in FIG. 1), only a dynamic index (S12a and / or S12b), or only a merging index S13. Further, depending on the implementation, a combination of at least two of the index request, dynamic index (S12a or S12b) and merging index (S13) may be performed.

When only an index request is performed, the indexed data may be the result primarily processed by an existing indexing method. Such indexed data may be stored in a storage system such as a memory and then processed by a dynamic index scheduler or a merging index scheduler according to a separate procedure and output as indexed data. The dynamic index scheduler is simply referred to as an index scheduler, and the merging index scheduler may be referred to as a merging scheduler.

3 is a flowchart of an index request procedure that can be employed in the indexing method according to the present embodiment.

Referring to FIG. 3, the indexing method according to the present embodiment may receive a document indexing request through an application programming interface (API) listener of the index server, for example, at step S111. The initial signal for the document index request may be passed from the user terminal or the administrator terminal to the index server.

An index server corresponding to an index server or an index server can write or maintain an index document in memory as an index request job is created within a job pool of the index server, that is, according to a document index request from an API listener (S112). An index server or indexer may include a dynamic index module.

Maintaining an index document in memory may be implemented to specify that it is an indexing target or to write an index document to a storage location at a particular location in memory that stores the indexing destination. The memory may be implemented as at least a portion of a storage system or storage.

That is, in the maintenance step S112 described above, an index-requested document is first piled up in a file having a structure or a form that has been defined in advance, and a predetermined amount of documents are read based on a certain capacity or a predetermined number of log files for the stacked files . Here, the operation of reading the file can be performed by the document logger. The document logger may be included in the index request processing unit.

Next, the index request processing unit periodically writes a document in the memory as a log file (S113). The index request processing unit may be included in the index server or the dynamic index module. The LOG file may have a file name of "indexlog " and may be stored in a format corresponding to the document in the segment, for example, 2321845350978868.

4 is a flowchart of an indexing procedure that can be employed in the indexing method according to the present embodiment.

Referring to FIG. 4, in the indexing method according to the present embodiment, the index scheduler can read the LOG file (S41). Then, a proper amount of documents can be selected (S42).

Appropriate amount of document selection can be implemented by reading an appropriate amount of log files. That is, the index scheduler is set according to the performance or capacity of the indexing device in performing indexing, or reads a log file having a preset number or capacity according to characteristics of data (big data) to be processed, May be implemented to select a document. It is preferable that the capacity or the number of the log files to be read once is 20 megabytes (Mbyte) or 10,000 in the case of the big data.

Next, the index scheduler can perform indexing on the selected document. In this embodiment, the term 'index' may be used in combination with 'dynamic index'. The indexed document may be classified according to the type, level, etc., and may be generated as a temporary segment by a predetermined amount (S43).

Once a temporary segment is created, the basic index can be viewed as complete. However, in the present embodiment, a further dynamic indexing process can be further performed for the improved dynamic indexing.

The index scheduler exposes the temporary segment to the search (S44). Here, exposing the search means storing the data in a memory or a storage system so that the indexing device is connected and the temporary segment can be searched by a device searching the database for an external query.

Next, the index scheduler can determine whether there is a merging document among the documents in the temporary segment exposed in the search (S45).

As a result of the determination, if there is a document to be merged, the index scheduler can store the identifier (document ID) of the document in the deletion request file (S46). The delete request file can be named 'DELETE.REQ'. The deletion request file refers to a file to be deleted when a predetermined condition is satisfied thereafter because of duplication or the like.

If it is determined that there is no document being merged among the documents in the temporary segment being exposed to the search, the index scheduler changes the predetermined condition (the range of exposure, level, etc.) You can re-expose it. This process may be a variant of the above step S44 to confirm that the document in the temporary segment is not related to merging or other operations.

On the other hand, if it is determined in step S45 that there is no document to be merged among the documents in the temporary segment or that there is no document related to another task among the documents in the temporary segment, The dynamic index can be completed (S47).

5 is a flow chart of a merging procedure that can be employed in the indexing method according to the present embodiment.

Referring to FIG. 5, in the indexing method according to the present embodiment, the merging scheduler can periodically check the segment list in the index data. Further, the merging scheduler can select a segment to be merged based on the checked segment list (S51).

The segment list may include information that distinguishes the merging target segment from other segments. In this embodiment, the term " merging " may be used in combination with the " merging index ".

As a result of the above checking and selecting step S51, if the segment to be merged does not exist (NO in S52), the merging scheduler can wait for the next cycle of the merging indexing operation.

On the other hand, if it is determined in step S51 that the segment to be merged exists (YES in step S52), the merging scheduler converts or segments the segments to be merged into temporary segments grouped according to the classification policy, (S53). Classification policies may include criteria to classify documents by document level, document type, and so on.

As a result of the merging operation, temporary segments bundled into groups can be generated. If a temporary segment grouped by group is created, the basic merging index operation can be viewed as completed. However, in the present embodiment, the following additional process can be further performed for real-time retrieval of data and the like.

The merging scheduler exposes the temporary segment to search (S54). Temporary segments that are exposed to the search may be stored, placed, or ordered in order along with the temporal flow with the temporary segments generated by the dynamic index.

Next, the merging scheduler can determine whether there is a deletion request file generated during the dynamic indexing operation or during the merging indexing operation (S55). As a result of the determination, if the answer is YES, the merging scheduler reads the deletion request file and deletes the temporary segment of the document ID or deletes the document corresponding to the temporary segment (S56). On the other hand, if the result of the determination is NO, the merging scheduler can omit the process of deleting a specific temporary segment.

Next, the merging scheduler can determine whether the deletion request file (e.g., DELETE.REQ) is not used elsewhere (job, etc.) (S57). If yes, the merging scheduler reads the deletion request file to delete the temporary segment corresponding to the document ID in the deletion request file or delete the document corresponding to the temporary segment (S58). On the other hand, if the determination result of step S58 is NO, the merging scheduler can omit the deletion process of the specific temporary segment.

The merging indexing operation can be completed according to the above-described procedure (S59).

Although the above-described two steps S55 and S57 are sequentially performed in the present embodiment, the present invention is not limited to such a configuration, and each step S55 or S57 may be performed independently or in parallel And may be implemented so that it is performed in the reverse order of the present embodiment (i.e., S57 is followed by S55) depending on the implementation.

According to the above-described embodiment, by performing the dynamic indexing operation and the merging indexing operation repeatedly, it is possible to perform an indexing operation for generating a small index data together with a quick index operation on the big data inputted in real time, Real-time retrieval of data such as < RTI ID = 0.0 >

6 is a view for explaining the operation principle of real-time indexing in the indexing method according to the present embodiment.

Referring to FIG. 6, the indexing method according to the present embodiment can be implemented by an indexing server including a dynamic indexing module 60 and an API listener 61. The index server can create and process jobs required for real-time indexing in a buffer or memory pool (Job Pool, 62). The dynamic index module 60 may include a document log 63, a dynamic index scheduler 65 and a merging index scheduler 66.

The operation process of the index server will be described in more detail as follows.

When the index request (/ service / index) is received (S61), the API listener 61 can transmit the index request to the preset work pool 62 (S62).

When the index request is forwarded, the work pool 62 creates an index document request job (IndexDocumentRequestJob) and provides an environment in which the job can be executed in the index server. That is, the index request job generated in the work pool 62 may be transmitted to the document logger 63 of the dynamic index module (S63). Here, the document logger 63 may be a TimeBaseRollingDocumentLogger that operates on a time basis.

The work pool 62 may be referred to as a memory pool. These pools of work 62 may include logical areas of main memory or storage devices that are reserved for processing one task or group of tasks. In the system of the index server, all main memory devices can be divided into logical allocations called memory pools. The system including the index server can manage the transmission of data and programs to the work pool 62 by default.

That is, the memory pool in which the user's work reserves memory is substantially the same as the pool that limits the activity level of this operation. The activity level of the memory pool may be the number of threads activated simultaneously in the memory pool. An exception to this is the system operation that gets memory from the default pool but uses the machine pool activity level. In addition, the subsystem monitor in the memory pool can free up memory in the first subsystem description pool, but can use the machine activity level. These subsystem monitors can always be run regardless of the activity level setting.

The above-described API listener 61 can basically perform a document insertion operation (InsertDocumentsAction), a document update operation (UpdateDocumentsAction), a document delete operation (DeleteDocumentsAction), or a combination thereof.

Next, the document logger 63 can generate an index log 71 for the document or segment in the file (S64). The index log 71 may include a log record having a numeric form. The log record may have the form of 2321845350978868 or the like.

In addition, the document logger 63 may operate every flush period to generate an index log file having a log file of a predetermined number or capacity, and insert the files into the file queue 64 of a predetermined format.

Next, when a file queue 64 containing an index log file is delivered in the document logger 63, the dynamic index scheduler (IndexFireScheduleWorker) 65 of the dynamic index module 60 sends a dynamic index A job is created (S66), and a segment can be created through a dynamic indexing operation (S67). The generated segment may be stored as a temporary segment in a specific area of memory and may be exposed as a search index segment.

Meanwhile, the merging index scheduler 66 of the dynamic indexing module 60 can periodically check the index segment (S68). The merging index scheduler 66 includes a first merging index job (Job-1), a second merging index job (Job-1), and a second merging index job 62. The merging index scheduler 66 includes a plurality of segments in the index segment, (Job-2) or the like (S69). The merged segments may be stored in a particular area of the memory and exposed to the search as another specific segment in the index segment (S69a).

Although the dynamic index module 60 has been described as including document logger 63, dynamic index scheduler 65 and merging index scheduler 66 in the present embodiment, the index server of the present invention is not limited to such a configuration, The merging index scheduler 66 may be implemented as a separate module outside the index server. Also, the document logger 63 can be implemented as a separate document request processing module outside the index server.

FIG. 7 is a diagram for explaining the operation principle of the document logger of FIG. 6; FIG.

Referring to FIG. 7, the document logger according to the present embodiment may be implemented by a software module or a program stored in a memory and a processor executing the program or the software module. The document logger basically reads the memory data by accessing the memory 71 or the memory area for storing the input document, converts the read document into index log files 75 and / or divides the read document into index log files 75, Can be inserted.

More specifically, the operation principle of the document logger will be described. First, the document logger accumulates input documents, that is, index-requested documents, in a file in the memory 71. The memory 71 may include a first memory 71a and a second memory 71b. Of course, the first memory 71a and the second memory 71b may correspond to the first memory area and the second memory area in a single memory.

Next, the document logger can replace and stack the input document stored in the memory 71 as memory data every time the input document is flushed by a double buffering method. In addition, the document logger may be configured to perform a flush check operation 72 (S72). The flush check operation refers to checking a document stored in a plurality of memories or a plurality of memory areas in a swap period alternately.

In one embodiment, the document logger may attach a document to the file at each flush period (S73). For example, if the flush period is set to 1 second, the indexed document (dynamic index scheduler, etc.) can receive updated documents every second.

In another embodiment, the document logger may replace the file every rolling cycle (S74). For example, when the rolling period is set to the basic 30 seconds or 1 minute, it is possible to prevent one file from becoming excessively large in the document logger as compared with the case where the rolling period is set to less than 30 seconds.

The index log file 75 generated by the file logger may store the index log record corresponding to each document and be designated with a predetermined file name. For example, you can use nanoTime (), a method that returns a time as an API, to generate a file name with a fixed number of time values (such as 2327473223121315).

The files 75 in which the index log is recorded can be delivered to the dynamic index scheduler through a file queue. The file queue 64 is a queue or a queue that reads out the contents of the file / object and outputs the contents when the file / . The file queue 64 may include a LogFileStatus object. The log file status object may contain information about the file object and whether it is closed (isClosed).

FIG. 8 is a view for explaining the operation principle of the index scheduler of FIG. 6, that is, the dynamic index scheduler.

8, the dynamic index scheduler according to the present embodiment reads a document from the file 75 transmitted through the file queue 64 (S81 and S82) and completes indexing within a few seconds (S84a, S84b , S84c). And exposed to the search node so that the indexed document is searched (S85a, S85b, S85c).

The operation principle of the dynamic index scheduler will be described in more detail as follows. When a file is received from the document logger via the file queue 64, the dynamic index scheduler closes the file of the queue and completes the reading of the document to be performed accordingly (S81) have.

Next, the dynamic index scheduler extracts an index document from among the read documents (S82). The extraction of an index document can be implemented to extract 20MB or less than 10,000 documents. The extraction condition of the index document may be stored in advance as a first setting (dynamic.max_log_count), a second setting (dynamic.max_log_size_MB), or the like. According to such extraction conditions, a large amount of data can be divided into a certain size (20 MB) or a certain amount (about 10,000) and indexed, thereby completing the indexing operation for the entire data within a few seconds.

Next, the dynamic index scheduler may distribute the extracted index document (index request document) for dynamic indexing operation and transmit it to the plurality of dynamic indexing operations (S84a, S84b, S84c) (S83). A plurality of dynamic indexing jobs (S84a, S84b, S84c) can perform each dynamic indexing operation and deliver the dynamically indexed segments to the respective search nodes S85a, S85b, S85c. The search nodes may include the dynamically indexed segment as a new segment in the index segment (S86). The dynamically indexed segment may be included in searchable data 82 as a newly added segment of the index segment 83 that is exposed to the search.

Here, when the segment of the search node is included in the segment list in the deletion request file, that is, when the segment is included in the merging operation, the dynamic index scheduler inserts the segment of the search node into a separate delete request file (delete.req) (84) (S87). The deletion request file 84 may be stored in a memory for later deletion of the corresponding file or may be transmitted to the outside.

The name of the deletion request file 84 may include a predetermined number before delete.req and may be combined with a reference number that sets a predetermined number of merging reference times. For example, the number (file name number) added to the file name of the deletion request file 84 and the reference number are as shown in [Table 1] below.

 File name number Reference count 1111111111 2 2222222222 3 3333333333 3

The deletion request file 84 may be set to count the reference number of the merging operation and delete the document if the reference number is zero. If the deletion request file 84 is used, a large number of deletion request files 84 are generated when the number of merging operations becomes longer and longer. However, when the merging operation ends, all the deletion request files 84 generated during the merging operation are deleted Can be configured.

The reason for creating the deletion request file 84 is to search for and delete a document in an existing segment if the newly indexed document is an update of an existing document. This deletion request file 84 may include the function of marking for deletion (deletion marking). In other words, according to the present embodiment, since the merging document can not be deleted, the data can be marked so that the merging process can be completed.

FIG. 9 is a diagram for explaining the operation principle of the dynamic index module of FIG. 6; FIG.

Referring to FIG. 9, the dynamic index module according to the present embodiment checks the number of live documents in the segments 83 continuously generated through the dynamic index and increases the number of segments, and performs a periodic merging index to reduce the number of segments .

This dynamic index module can be implemented by the collaboration of the document count confirmation module and the merging index scheduler in addition to the dynamic index scheduler. The document count verification module or the merging index scheduler may be implemented by a program (or a software module) stored in a memory and a processor executing the program. The above-described document number confirmation module can be included as a part of the merging index scheduler.

More specifically, the document count confirmation module may check the number of currently exposed documents included in the index segments (S91). The currently exposed document may be referred to as a live document.

The number of live documents can be checked according to a predetermined set of merging rules. That is, the document number check module can classify and check the number of live documents to include a log-likelihood range. The logarithmic range may include 1 to 6 in log base 10 (log ₁₀ ) units such as 0, 100, 1K, 10K, 100K, 1M, The number of live documents may include one or more segments per number of documents in each range.

The results of checking the number of live documents are shown in [Table 2] below.

Live document count range Number of Segments 0 2 100 6 1K 4 10K 2 100K One 1M One 5M One 5M + One

With the use of this logarithmic range, in a subsequent merging index operation, segments can be sorted according to the number of documents included in each segment and prepared to perform a plurality of merging operations in parallel (S92).

When the number of live documents is confirmed, the merging index scheduler may generate at least one merging job classified by the number of documents in the job pool according to a preset merging policy or merging rule, and execute the merging index (S93).

The merged documents may be prepared with a plurality of temporary segments 83a and 83b and exposed to search. Some of the merged documents may be deleted according to the updated package list or the document or segment list in the deletion request file 84 (S96). If you apply deletion using a delete request file, you can delete a file whose reference count is 0 by decrementing the reference count by one.

The non-deleted temporary segment may be registered as a new segment (Newer) in the index segment 83 included in the data 82 that is exposed to the search (S94, S95).

The foregoing merging rule is such that a segment having a specific field value of 0 performs merging unconditionally, a segment in which four or more segments are merged, a segment in which more than 5 million segments are excluded from merging, The merge may be performed without gathering four segments, or may be set such that segments not selected for merging are left as they are.

According to this embodiment, segments of similar size can be merged by executing the merging index divided by the number-of-documents range, whereby the input and output efficiency of the apparatus can be best used.

FIGS. 10A and 10B are diagrams for explaining the life cycle of the index segment, that is, the dynamic index segment of FIG.

Referring to FIG. 10A, the segment life cycle according to the present embodiment may have a change in the number of segments in the index segment and the number of documents in each segment by a dynamic index, a merging index, or a combination thereof.

For example, if there is a plurality of segments a0, a1, a2, and a3 as the index segments 83 of the data (first data, 82) exposed to the search, one segment 102 May be prepared and added as a new index segment a4 (S101). In that case, the index server may delete the documents that are duplicated with the documents in the new segment a4 dynamically indexed in the other segments (a0, a1, a2, a3) in the index segment of the data 82a that are exposed to the search S102). In Fig. 10, the dark shaded portions of each segment can indicate the deleted portion of the document.

10B, it is assumed that there are a plurality of segments a0, a1, a2, a3, a4, and a5 as index segments of data (first data, 82b) The index server includes the dynamically indexed segment 104 as the new segment in the first data 82b and the other segments a0, a1, a2 that are the same as the document in the dynamically indexed segment 104, a6 , a3, a4, a5) can be deleted. In this embodiment, the first segment a0 may overlap with the dynamic indexed segment a6 of all documents.

In addition, a merge indexing operation can be performed simultaneously with the dynamic indexing operation to generate the above dynamic indexed segment a6 (S105). That is, in parallel with the execution of a plurality of dynamic indexing operations, the merging index scheduler can perform a merge indexing operation on a plurality of segments a0, a1, a2, a3, a4, a5 in the index segment.

The merged indexed segment 106 may be included in the second data 82c as a new segment (S106). At this time, the merging index scheduler can delete the document by applying the deletion document 84a created during the merging to the merged indexed segment 106 (S107a).

In addition, the merging index scheduler may generate a document (deletion request file) for the subsequent deletion if there is a document of the job being merged in the merged indexed segment 106 (S107b).

In addition, if there are a plurality of different dynamic index operations, the merging index scheduler performs a dynamic indexing operation on each of the plurality of segments a0, a1, a2, a3, a4, a5, a6, The merged indexed segment 106 may be included in the second data 82c while adding the first segment 108a, 108b. The second data 82c may refer to data for which the index segment is increased, decreased or maintained in accordance with the time flow of the first data 82b to be exposed to the search.

The dynamically indexed segments 108a and 108b may be included in the second data 82c as new index segments a7 and a8 in the order described or as quickly as they are generated (S108a and S108b).

And two segments (a0, a1) in which the documents in the segment are all overlapped with the document in the segment that is dynamically indexed can be deleted.

FIG. 11 is a diagram illustrating a timeline of the life cycle of the index segment of FIG. 6. FIG.

Referring to FIG. 11, the data indexing method according to the present embodiment searches for segments (index segments) generated in each job for a plurality of index jobs and sequential merging jobs, that is, can do.

A first indexing segment a0 generated through a first indexing operation (index 1), a second indexing segment al generated through a second indexing operation (index 2), and a third indexing operation (index 3) The third index segment a2 generated through the first index segment may be exposed to the search in order according to the time flow.

Next, the index segments a0, a1, a2 may be merged indexed through a first merging operation (merging 1) to produce a sixth index segment a5.

Meanwhile, during the first merging operation (merging 1), the fourth index segment a3 may be generated through the fourth index operation (index 4). At this time, since the first merging operation (merging 1) for the previous index segments is being performed, the index server may give a temporary ID to the file name of the fourth index segment a3 and generate a deletion request file including the temporary ID together have.

Also, similar to the case of the above-mentioned fourth index operation (index 4), during the first merging operation (merging 1), the fifth index segment a4 may be generated through the fifth index operation (index 5) . At this time, since the first merging operation (merging 1) for some previous index segments is being performed, the index server may generate a deletion request file including a temporary ID in the file name of the fifth index segment a4.

On the other hand, after the first merging operation (merging 1) is completed, the second merging operation (merging 2) includes the sixth index segment a5 generated through the first merging operation (merging 1) and the second merging operation (A3, a4) dynamically indexed during the first and second index segments. The segment a7 which is the eighth index can be generated by the second merging operation (merging 2).

Of course, a seventh index segment a6 may be generated through a sixth index operation (index 6) during the second marching operation (merging 2). In this case, the index server can give a temporary ID to the seventh index segment a6 to generate a deletion request file including the temporary ID. The file name of the deletion request file may be given as "2330665693309899.delete.req ".

Since the IDs of segments are assigned sequentially at the end of the index, IDs can be granted late even if the index is first performed.

If there is a segment that is being merged at the end of the index, you can create your own <Temporary ID> .delete.req file (delete request file) so that the next merged segment can be referenced.

As described above, according to the present embodiment, duplicate documents can be effectively deleted by using a deletion request file while effectively performing a combination of a plurality of indexes and sequential mergings.

12 is a block diagram of a data indexing apparatus according to another embodiment of the present invention. 13 is a block diagram of a data indexing apparatus according to another embodiment of the present invention.

12, a computing device (hereinafter briefly referred to as an indexer or indexing device) that implements the data indexing method according to the present embodiment may include a control unit 1210 and a memory 1220. [ The control unit 1210 may further include a communication unit 1230 or a communication interface, and may be connected to the database 1240. At least a portion of the database 1240 may be included in the indexer.

The control unit 1210 may include an index request processing unit 1211, an index document extracting unit 1214, a dynamic indexing unit 1216, and a merging indexing unit 1218.

The index request processing unit 1211 may be implemented as an API listener or the like and may accept the index request and forward the received index request to the indexer's work pool. In the work pool, when an index request is received, an index request operation can be created.

The index document extracting unit 1214 may include means for starting the indexing operation according to the index request job created in the work pool or a component performing a function corresponding to this means. Index initiator 1214 may include a document logger that is coupled to a work pool.

The dynamic indexing unit 1216 may correspond to a dynamic indexing module or a dynamic indexing scheduler. The dynamic indexing unit 1216 can extract the index document based on the index log stored in the memory 1220 by the index document extracting unit 1214 and perform the dynamic indexing operation.

The merging index unit 1218 may be embodied as a merging index scheduler included in the dynamic indexing module or may be implemented with a separate merging indexing module. The merging indexing unit 1218 can check the number of live documents and perform a merging indexing operation.

The index request processing unit 1211, the index document extracting unit 1214, the dynamic indexing unit 1216 and the merging indexing unit 1218 are programs loaded into the control unit 1210, Lt; RTI ID = 0.0 &gt; 1224 &lt; / RTI &gt;

The controller 1210 may be implemented as a logic controller or a microprocessor. The controller 1210 may include one or more cores and a cache memory. When the control unit 1210 has a multi-core structure, a multi-core may refer to integrating two or more independent cores into a single package of a single integrated circuit.

13, when the indexing device of the present embodiment is implemented by the cooperation of the first computing device 120A and the second computing device 120B, the first computing device 120A May be implemented to perform part of the indexing method in the processor and to perform the remainder of the indexing method in the second processor of the second computing device 120B. In this case, the deletion request file 84c generated in the first processor and the deletion request file 84d generated in the second processor may have the same file name. Therefore, in this embodiment, a unique request ID or a number may be added to each of the plurality of processors at the end of the &quot; temporary ID &quot; to generate a deletion request file. For example, a unique identifier or number is set to 0 by adding 0 to the end of the temporary ID of the deletion request file 84c generated by the first processor and by appending 2 to the end of the temporary ID of the deletion request file 84d generated by the second processor Or the like.

Further, when the controller 1210 has a single core structure, a single core may include a central processing unit (CPU). The central processing unit may be implemented as a system on chip (SOC) in which a micro control unit (MCU) and a peripheral device (integrated circuit for external expansion device) are disposed together, but the present invention is not limited thereto. Here, the core includes registers for storing instructions to be processed, arithmetic logical units (ALUs) for comparisons, judgments, and arithmetic operations, internal controls for internally controlling CPUs for interpreting and executing instructions, A control unit, an internal bus, and the like.

In addition, the controller 1210 may include, but is not limited to, one or more data processors, image processors, CODECs, or a combination thereof. The control unit 1210 may include at least one electric control unit (ECU) mounted on the vehicle.

The controller 1210 may include a peripheral device interface and a memory interface. In this case, the peripheral device interface is responsible for connection and connection management between the controller 1210 and the input / output system and other peripheral devices (communication units, etc.) The interface may be responsible for connection and connection management between the controller 1210 and the memory 1220.

The control unit 1210 may execute various software programs to perform data input, data processing, and data output for performing the indexing method. In addition, the control unit 1210 may execute a specific software module (instruction set) stored in the memory 1220 to perform various specific functions corresponding to the corresponding module. That is, the controller 1210 can play a key role in performing the indexing method in the computing device by the modules 1224 stored in the memory 1220. [

The memory 1220 described above may be referred to as a memory system and may include high speed random access memory and / or nonvolatile memory such as one or more magnetic disk storage devices, one or more optical storage devices, and / or a flash memory. The memory 1220 may also store software, programs, a set of instructions, or a combination thereof.

Components of the software may include an operating system module, a communication module, a graphics module, a user interface module, a moving picture experts group (MPEG) module, a camera module, one or more application modules, and the like. Module 1224 may be represented as an instruction set or program as a collection of instructions.

The operating system includes built-in operating systems such as MS WINDOWS, LINUX, Darwin, RTXC, UNIX, OS X, iOS, Mac OS, VxWorks, Google OS, Android, And may include various components for controlling the system operation of the indexer. The above-described operating system may also include, but is not limited to, a function of performing communication between various hardware devices and software components (modules).

The communication unit 1230 can support one or more communication protocols so that the indexer can be connected to the search server or other device via the network. The communication unit 1230 may include one or more wireless communication subsystems. The wireless communication subsystem may include a radio frequency receiver and a transceiver and / or an optical (e.g., infrared) receiver or transceiver.

The network may be, for example, a Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), W-Code Division Multiple Access (W-CDMA), Long Term Evolution (LET-A), Orthogonal Frequency Division Multiple Access (OFDMA), WiMax, Wi-Fi (Wireless Fidelity), Bluetooth, and the like.

Meanwhile, in the present embodiment, the module 1224 stored in the memory 1220 of the indexer may be, but is not limited to, a functional block or module mounted on a computer device. The modules 1224 described above may be stored in a computer readable medium (recording medium) in the form of software for implementing a series of functions (indexing methods) they perform, or may be transmitted to a remote location in the form of a carrier to be implemented in various computing devices .

The computer readable medium may include a plurality of computing devices or cloud systems connected via a network, and at least one of the plurality of computing devices or the cloud system may be a program for performing the indexing method of the present embodiment Source code, and so on.

That is, the computer-readable medium may be embodied in the form of a program command, a data file, a data structure, or the like, alone or in combination. Programs recorded on a computer-readable medium may include those specifically designed and constructed for the present invention or those known and available to those skilled in the computer software arts.

The computer-readable medium may also include a hardware device specifically configured to store and execute program instructions, such as a ROM, a RAM, a flash memory, and the like. Program instructions may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. The hardware device may be configured to operate with at least one software module to perform the indexing method of the present embodiment, and vice versa.

In addition, the indexing method of this embodiment can be employed in a cloud computing based search engine system. A search engine may refer to a means for retrieving data in a network such as the Internet or a component that performs a function corresponding to such means. For example, the search engine system may be a system supporting Infrastructure as a Service (IaaS), Platform as a Service (PaaS), or Software as a Service (SaaS). Here, IasS is a service type in which a server resource, a communication protocol (IP, etc.), a network, a storage, and a power for operating a server are utilized in a virtual environment. PaaS is a form that provides a platform to develop a service and an API to develop an application that uses the environment. SaaS is a service type that provides applications running in the cloud environment.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

Claims (15)

1. A data indexing method for real-time searching performed in a computing device having a processor,
The index request processing unit of the processor recording a document of a memory as a log file;
Selecting an amount of a document including at least a portion of information read from the log file by an index scheduler of the processor;
The index scheduler generating at least one temporary segment for the document;
The index scheduler exposing the at least one temporary segment to a search of a search engine; And
If the index scheduler is merging a document included in the at least one temporary segment while the at least one temporary segment is exposed, generating an erase request file storing an identifier of the document, ,
The processor further performs the function of a merging index scheduler by a program stored in the memory,
The merging index scheduler merging a plurality of index segments dynamically indexed by the dynamic index scheduler of the processor,
Wherein the processor grants a temporary ID for an index segment that is dynamically indexed during a merge indexing operation of the merging index scheduler, wherein the deletion request file includes the temporary ID in a filename. The method according to claim 1,
Wherein the file name of the deletion request file includes a number determined by a temporal fixed value. The method of claim 2,
Wherein a reference reference number is set in the deletion request file, and the reference number is subtracted by a predetermined number when the deletion request file is referred to the merging. The method of claim 3,
Further comprising, after said generating, deleting a document corresponding to the deletion request file below the lower limit when the reference number of references is less than or equal to the lower limit by reference in the merging. 5. The method according to any one of claims 1 to 4,
After the step of exposing,
Further comprising the step of checking the number of live documents in the exposing step, wherein the range of live document counts is classified into log realistic ranges. The method of claim 5,
After the checking step,
And simultaneously processing a merging index for each of the plurality of merging target segments classified into the logarithmic ranges. The method of claim 6,
Wherein the logarithmic ranges include classification ranges in which the number of documents included in each segment is 0, 100, 1K, 10K, 100K or 1M. The method according to claim 1,
Wherein the writing step comprises attaching a document to a file each time the document logger flushes and replacing the file every rolling cycle to generate the log file. The method of claim 8,
Wherein the document logger is connected to two memories or memory areas that are replaced each time a flush of reading the log file is performed. The method of claim 8,
Wherein the log file is inserted into a file queue, the file queue includes a log file status object, and the log file status object includes a file object and a closed status. The method of claim 8,
Further comprising the step of obtaining a signal or information for an index request operation in the document logger prior to the step of recording. The method according to claim 1,
Wherein the selecting step extracts log files of a predetermined quantity or number according to the indexing capacity or performance of the computing device. As a data indexing device,
A processor, a memory coupled to the processor, and a program stored in the memory,
The processor performs the functions of the document logger and the dynamic index scheduler by the program,
The document logger records the document in the memory as a log file,
Wherein the dynamic index scheduler selects a quantity of documents containing at least a portion of the information read in the log file, generates at least one temporary segment for the document, exposes the at least one temporary segment to a search of a search engine Generating a deletion request file storing an identifier of the deletion candidate document when the deletion candidate document included in the at least one temporary segment is being merged,
The processor further performs the function of a merging index scheduler by a program stored in the memory,
The merging index scheduler merging a plurality of index segments dynamically indexed by the dynamic index scheduler of the processor,
Wherein the processor provides a temporary ID for an index segment that is dynamically indexed during a merging index operation of the merging index scheduler, wherein the deletion request file includes the temporary ID in a filename. delete 14. The method of claim 13,
Wherein the processor deletes a document included in a first index segment that has already been exposed to search using the deletion request file when exposing a second index segment generated by the merging index scheduler to the search, .

Images