KR102674954B1 - Method and apparatus for providing search service using large language model and deep learning - Google Patents

Abstract

A method of providing a search service, comprising: acquiring data from at least one external electronic device, extracting at least one entity from the obtained data through LLM, and learning a NER model using the extracted entity. and an inference operation, and an operation of identifying a search item through STS model inference using the inference result of the NER model. In addition to this, various embodiments identified through this document are possible.

Description

{Method and apparatus for providing search service using large language model and deep learning}

Various embodiments of the present invention relate to a method and device for providing a search service using a large language model and deep learning.

Due to the development of the Internet, search services that allow users to search for and purchase specific products or goods in a web or mobile environment are developing. This search service can receive information from multiple suppliers and help users easily find the information they want from the information provided. For example, in the case of a search service for accommodation information, accommodation information of accommodation establishments can be provided from a number of accommodation information providers to support users in finding desired accommodation information.

However, when providing a search service by receiving information from multiple suppliers, problems may occur where the same item has different information or the same item is exposed repeatedly. In addition, in order to determine whether or not duplicates of the same item exist, it was previously done simply manually, which inevitably required a lot of manpower and time.

Various embodiments of the present invention can provide a method and device for providing a search service using a large language model (LLM) and deep learning.

A method of providing a search service according to various embodiments of the present invention includes an operation of acquiring data from at least one external electronic device, an operation of extracting at least one entity from the obtained data through LLM, and the extracted entity It may include learning and inferring a NER model using , and identifying a search item through STS model inference using the inference result of the NER model.

The data according to various embodiments of the present invention may include information related to at least one accommodation.

The method according to various embodiments of the present invention includes the operation of removing unnecessary strings from the obtained data using regular expressions, and at least one unnecessary string from the data from which the unnecessary strings were removed in consideration of domain characteristics of the obtained data. An operation to separate the string may be further included.

A prompt is created to extract entities that can distinguish the properties of the search item, and the entities are extracted using LLM.

The method according to various embodiments of the present invention further includes an operation of performing verification on the extracted entity, and the operation of learning and inferring the NER model includes determining a true positive among the entities extracted by the LLM. ) may include the operation of learning and inferring the NER model by inspecting only the NER model.

The operation of verifying the extracted entity according to various embodiments of the present invention may include the operation of identifying the authenticity of the extracted entity based on user input.

The operation of learning and inferring the NER model according to various embodiments of the present invention includes the operation of fine tuning data to which an entity reflecting the domain characteristics of the data obtained through the LLM is added to the NER model, and the learning It may include an operation of inferring an entity for the acquired data through the NER model.

The operation of identifying the search item through the STS model inference according to various embodiments of the present invention includes outputting the output of the NER model that learned data to which entities reflecting domain characteristics of the data obtained through the LLM were added. It may include an inference operation using the input of the STS model.

The operation of identifying the search item through the STS model inference according to various embodiments of the present invention may include the operation of determining whether the search item corresponding to the acquired data is a duplicate.

The method according to various embodiments of the present invention may further include storing the search item in a database when it is determined that the search item is not duplicated.

An apparatus for providing a search service according to various embodiments of the present invention includes a memory including an LLM, a communication circuit for communicating with an external electronic device, and a processor operatively connected to the memory and the communication circuit, The processor acquires data from at least one external electronic device through the communication circuit, extracts at least one entity from the acquired data through the LLM, and learns and infers a NER model using the extracted entity. And, it can be set to identify the search item through STS model inference using the inference result of the NER model.

The data according to various embodiments of the present invention may include information related to at least one accommodation.

The processor according to various embodiments of the present invention removes unnecessary strings from the obtained data using regular expressions, and removes at least one string from the data from which the unnecessary strings have been removed in consideration of domain characteristics of the obtained data. can be set to separate.

The processor according to various embodiments of the present invention creates a prompt for extracting an entity that can distinguish the properties of the search item and extracts the entity using LLM.

The processor according to various embodiments of the present invention performs verification on the extracted entities, and learns and infers the NER model using entities that have only verified true positives among the entities extracted by the LLM. can be set.

The processor according to various embodiments of the present invention may be configured to perform verification of the extracted entity by identifying the authenticity of the extracted entity based on user input.

The processor according to various embodiments of the present invention fine-tunes data to which entities that reflect domain characteristics of the data acquired through the LLM are added to the NER model, and fine-tunes the data obtained through the learned NER model. It can be set to infer an entity.

The processor according to various embodiments of the present invention may be set to infer by using the output of the NER model that learned data to which entities that reflect the domain characteristics of the data obtained through the LLM were added as the input of the STS model. there is.

The processor according to various embodiments of the present invention may be set to identify the search item by determining whether the search item corresponding to the acquired data is a duplicate.

The processor according to various embodiments of the present invention may be set to store the search item in a database when it is determined that the search item is not duplicated.

According to various embodiments of the present invention, by providing a real-time search service using a large language model (LLM) and deep learning, it is possible to easily determine whether or not a search item is duplicated, and thus, the real-time search It can provide the effect of reducing user fatigue and time for searching and comparing items.

In addition, various effects that can be directly or indirectly identified through this document may be provided.

1 is a diagram for explaining the configuration of a device that provides a search service according to an embodiment of the present invention.
Figure 2 is a diagram for explaining a system for providing a search service according to an embodiment of the present invention.
Figure 3 is a diagram for explaining a method of providing a search service according to an embodiment of the present invention.
Figure 4 is a flow chart to explain a method of providing a search service according to an embodiment of the present invention.
In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components.

Hereinafter, various embodiments of the present invention will be described with reference to the attached drawings. For convenience of explanation, the sizes of components shown in the drawings may be exaggerated or reduced, and the present invention is not necessarily limited to what is shown.

FIG. 1 is a diagram for explaining the configuration of an apparatus for providing a search service according to an embodiment of the present invention, and FIG. 2 is a diagram for explaining a system for providing a search service according to an embodiment of the present invention.

Referring to FIGS. 1 and 2 , an electronic device 100 that provides a search service may include a communication circuit 110, a memory 120, and a processor 130. However, the configuration of the electronic device 100 is not limited to this. According to various embodiments, the electronic device 100 may further include at least one other component in addition to the components described above. As an example, the electronic device 100 may further include a display for displaying search results. The display may visually provide information to the outside of the electronic device 100 (eg, a user). The display may include, for example, a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The communication circuit 110 connects the electronic device 100 and the external electronic device 210 (e.g., the first external electronic device 211, the second external electronic device 212, and/or the n-th external electronic device 213). )) can support the establishment of a direct (e.g. wired) communication channel or wireless communication channel and the performance of communication through the established communication channel. According to one embodiment, the electronic device 100 may obtain data (or learning data) related to the provision of a search service from the external electronic device 210 through the communication circuit 110. According to one embodiment, the electronic device 100 may store data obtained from the external electronic device 210 through the communication circuit 110 in the database (DB) 220.

The memory 120 may store various data used by at least one component (eg, processor 130) of the electronic device 100. The data may include, for example, software (eg, program) and input data or output data for instructions related thereto. The memory 120 may include volatile memory or non-volatile memory. According to one embodiment, the memory 120 may include a large language model (LLM) 121 and 230. The LLM (121, 230) is a form of artificial intelligence and can be used to implement and generate text-based information. For example, the LLMs 121 and 230 develop language understanding and generation capabilities by learning large amounts of text data, and can be used in natural language processing tasks.

The processor 130 may execute software (e.g., a program) to control at least one other component (e.g., hardware or software component) of the electronic device 100 connected to the processor 130, A variety of data processing and calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 130 stores commands or data received from other components (e.g., communication circuit 110) in volatile memory, and stores the commands or data received from other components (e.g., communication circuit 110) in volatile memory. Commands or data can be processed and the resulting data can be stored in non-volatile memory.

On the other hand, like the electronic device 100, an artificial intelligence device (or system) or a device (or system) that provides a prediction model based on artificial intelligence is a computer system that implements human-level intelligence and allows the machine to learn and make decisions on its own. It is a device (or system) whose recognition rate improves as it is used. Artificial intelligence technology consists of machine learning (deep learning) technology that uses an algorithm that classifies/learns the characteristics of input data on its own, and element technologies that mimic the functions of the human brain such as cognition and judgment using machine learning algorithms. Element technologies include, for example, linguistic understanding technology that recognizes human language/characters, visual understanding technology that recognizes objects as if they were human eyes, reasoning/prediction technology that judges information and makes logical inferences and predictions, and human understanding technology. It may include at least one of a knowledge expression technology that processes experience information into knowledge data, and a motion control technology that controls the autonomous driving of a vehicle and the movement of a robot.

The processor 130 included in the electronic device 100, which provides a search service using the LLMs 121 and 230 and deep learning based on artificial intelligence, operates a program (and the database) stored in the memory 120. A neural network can be learned using the learning data stored in (220). The neural network may be designed to simulate human brain structure on a computer, and may include a plurality of network nodes with weights that simulate neurons of a human neural network. The plurality of network nodes may exchange data according to their respective connection relationships to simulate the synaptic activity of neurons sending and receiving signals through synapses. The neural network may include a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes are located in different layers and can exchange data according to convolutional connection relationships. Examples of the neural network models include deep neural networks (DNN), convolutional deep neural networks (CNN), Recurrent Boltzmann Machine (RNN), Restricted Boltzmann Machine (RBM), and deep neural networks (DNN). It includes various deep learning techniques such as deep belief networks (DBN) and Deep Q-Network, and can be applied to fields such as computer vision, speech recognition, natural language processing, and voice/signal processing. .

The processor 130 may acquire training data necessary for the neural network model. As an example, the processor 130 may obtain learning data from the database 220. Additionally, the processor 130 may perform training of the neural network model using the acquired training data. At this time, the processor 130 may learn the neural network model through supervised learning using at least some of the learning data as a decision standard. Alternatively, the processor 130 can learn the neural network model through unsupervised learning, which discovers a decision standard by learning on its own using the learning data without guidance. In addition, the processor 130 can learn the neural network model through reinforcement learning using feedback on whether the result of situational judgment based on learning is correct. Additionally, the processor 130 may train the neural network model using a learning algorithm including error back-propagation or gradient descent.

When the neural network model is learned, the processor 130 may store the learned neural network model (eg, LLM (121, 230)) in the memory 120. According to one embodiment, the processor 130 may store the learned neural network model in the memory of a server connected to the electronic device 100 through a wired or wireless network through the communication circuit 110.

According to one embodiment, the processor 130 is connected to at least one external electronic device 210 (e.g., a first external electronic device 211, a second external electronic device 212, and /Or data may be obtained from the nth external electronic device 213). For example, the external electronic device 210 may include a server that provides information about search items. As an example, the processor 130 may obtain accommodation information of an accommodation establishment from at least one accommodation information provider (eg, a server operated by the accommodation information provider) through the communication circuit 110. For example, the data may include information related to at least one accommodation. According to one embodiment, the processor 130 may store the obtained data in the database 220.

According to one embodiment, the processor 130 may perform pre-processing on the acquired data. For example, the processor 130 may remove unnecessary strings from the obtained data using regular expressions. Thereafter, the processor 130 may separate at least one string from the data from which the unnecessary strings have been removed by considering domain characteristics of the obtained data.

According to one embodiment, the processor 130 may extract at least one entity from the acquired data through the LLMs 121 and 230. For example, the entity may include an entity capable of distinguishing properties of a search item by considering domain characteristics of the acquired data. For example, accommodation information, the entity may include at least one of accommodation type, accommodation size (or room size), whether smoking is allowed, or type of view from the room (e.g., mountain view, ocean view, etc.). there is.

According to one embodiment, the processor 130 may train the LLMs 121 and 230 to increase the accuracy of entity extraction through the LLMs 121 and 230. As an example, the processor 130 sets an entity that can distinguish the properties of the search item by considering the domain characteristics of the data, and the set entity can be extracted from arbitrary data through the LLM (121, 230). The LLM (121, 230) can be trained by inputting a prompt to the LLM (121, 230). For example, the prompt may include example string learning data generated to extract entities that can distinguish properties of search items according to domain characteristics from arbitrary data.

According to one embodiment, the processor 130 may transmit the extracted entity to the item duplication determination module 240. The item duplication determination module 240 may perform a function of determining whether the search item corresponding to the acquired data is an item that overlaps with an existing search item (eg, a search item stored in the database 220). The item duplication determination module 240 may include a Named Entity Recognition (NER) model 241 and a Semantic Textual Similarity (STS) model 242.

According to one embodiment, the processor 130 may learn and infer the NER model 241 using the extracted entity. The NER model 241 can perform the task of recognizing and extracting named entities (named entities) from given text. Here, the named entity is an identifiable entity and may include, for example, at least one of the accommodation type, accommodation size (or room size), whether smoking is allowed, or the type of view seen from the room. Through the NER model 241, the processor 130 identifies an entity in the data and displays the start and end positions of the identified entity to accurately determine the location of the entity in the data. You can classify what type the identified entity belongs to.

According to one embodiment, the processor 130 may perform verification on the extracted entity before learning and inferring the NER model 241. As an example, the processor 130 may perform verification of the extracted entity by identifying the authenticity of the extracted entity based on user input. The authenticity of the entity can be expressed, for example, as a “true” value indicating that the entity was extracted correctly and a “false” value indicating that the entity was extracted incorrectly. Thereafter, as a result of verification of the extracted entity, if it is determined that the entity was extracted correctly, the processor 130 can learn and infer the NER model 241 using the verified entity.

According to one embodiment, the processor 130 fine tunes data to which entities reflecting domain characteristics of the data obtained through the LLMs 121 and 230 are added to the NER model 241. You can. To this end, the processor 130 may change the data into BIO (Beginning, Inside, Outside) format for learning the NER model 241. The BIO format is a data format used in entity recognition tasks, and includes labels indicating the beginning (B - beginning), the interior (I - inside), and the non-entity (O - outside) of the named entity in the data. ) may include. Afterwards, the processor 130 can digitize the data converted to the BIO format. For example, the processor 130 may define a label index for an entity type and map the BIO label to a unique number based on this. Thereafter, the processor 130 may fine tune the NER model 241 using the above-described labeling data. In this process, the NER model 241 can be trained to identify entities in given data. For example, the fine-tuned NER model 241 may have better performance in extracting entities from raw data. Accordingly, the processor 130 can infer an entity for the acquired data through the learned NER model 241.

According to one embodiment, the processor 130 may use the output of the learned NER model 241 as an input to the STS model 242 to identify a search item through inference. The STS model 242 can perform the task of measuring and comparing semantic similarity between two sentences or texts. For example, the STS model 242 outputs a similarity score for two given text inputs, and the score may indicate how semantically similar the input texts are. Through the STS model 242, the processor 130 can quantify the semantic similarity between two sentences or texts and generate a similarity score that expresses the similarity between the two input texts in numbers. According to one embodiment, the processor 130 may infer data to which an entity reflecting the domain characteristics of the data obtained through the LLMs 121 and 230 has been added as input to the STS model 242. .

According to one embodiment, the processor 130 may identify the search item by determining whether the search item corresponding to the acquired data is a duplicate. For example, the processor 130 determines whether the search item corresponding to the acquired data is an item that overlaps with an existing search item (e.g., a search item stored in the database 220) through the STS model 242. can be identified. According to one embodiment, if the processor 130 determines that the search items are not duplicates, the processor 130 may store the search items in the database 220. The database 220 may be included in the memory 120 or may be included in an external server.

Figure 3 is a diagram for explaining a method of providing a search service according to an embodiment of the present invention.

Referring to FIG. 3, a device that provides a search service (eg, the electronic device 100 of FIG. 1) may acquire data in operation 310. For example, the processor of the device (e.g., processor 130 in FIG. 1) communicates with at least one external electronic device (e.g., external electronic device in FIG. 2) through a communication circuit (e.g., communication circuit 110 in FIG. 1). Data can be obtained from the device 210). For example, the external electronic device may include a server that provides information about search items. As an example, the processor may obtain accommodation information of an accommodation establishment from at least one accommodation information provider (eg, a server operated by an accommodation information provider) through the communication circuit. For example, the data may include information related to at least one accommodation. According to one embodiment, the processor may store the acquired data in a database (eg, database 220 in FIG. 2).

In operation 320, the processor may extract at least one entity from the acquired data through LLM (eg, LLM (121, 230)). For example, the entity may include an entity capable of distinguishing properties of a search item by considering domain characteristics of the acquired data. For example, as accommodation information, the entity may include at least one of accommodation type, accommodation size (or room size), whether smoking is allowed, or the type of view seen from the room (e.g., mountain view, ocean view, etc.).

According to one embodiment, the processor may learn the LLM to increase the accuracy of entity extraction through the LLM. As an example, the processor sets an entity that can distinguish the properties of a search item by considering the domain characteristics of the data, and inputs a prompt to the LLM to enable the set entity to be extracted from arbitrary data through the LLM. Thus, the LLM can be learned. For example, the prompt may include example string learning data generated to extract entities that can distinguish properties of search items according to domain characteristics from arbitrary data.

At operation 330, the processor may perform verification on the extracted entity. As an example, the processor may perform verification of the extracted entity by identifying the authenticity of the extracted entity based on user input. The authenticity of the entity can be expressed, for example, as a “true” value indicating that the entity was extracted correctly and a “false” value indicating that the entity was extracted incorrectly. Thereafter, as a result of verification of the extracted entity, if it is determined that the entity has been extracted correctly, the processor creates a NER model (e.g., NER model 241 in FIG. 2) using the verified entity in operation 340. Can learn and reason. According to one embodiment, the processor may omit performing operation 330. In this case, the processor may learn and infer the NER model using the extracted entity in operation 340.

The NER model can perform the task of recognizing and extracting named entities (named entities) from given text. Here, the named entity is an identifiable entity and may include, for example, at least one of the accommodation type, accommodation size (or room size), whether smoking is allowed, or the type of view seen from the room. The NER model allows the processor to identify entities in the data, mark the start and end positions of the identified entities to accurately locate the entities in the data, and determine what type the identified entities are. It can be classified as belonging to.

According to one embodiment, the processor may fine-tune data to which an entity reflecting domain characteristics of the data obtained through the LLM is added to the NER model. To this end, the processor can change the data into BIO format for learning the NER model. The BIO format is a data format used in entity recognition tasks and may include labels indicating the beginning, interior, and non-entity parts of a named entity in the data. Afterwards, the processor can digitize the data converted to the BIO format. For example, the processor may define a label index for an entity type and map the BIO label to a unique number based on this. Thereafter, the processor can fine tune the NER model using the above-described labeling data. In this process, the NER model can be trained to identify entities in given data. For example, a fine-tuned NER model may have better performance at extracting entities from raw data. Accordingly, the processor can infer an entity for the acquired data through the learned NER model.

At operation 350, the processor may identify a search item through STS model inference. For example, the processor 130 may use the output of the learned NER model 241 as an input to the STS model 242 to identify a search item through inference. The STS model can perform the task of measuring and comparing semantic similarity between two sentences or texts. For example, the STS model outputs a similarity score for two given text inputs, and the score may indicate how semantically similar the input texts are. Through the STS model, the processor can quantify the semantic similarity between two sentences or texts and generate a similarity score that expresses the similarity between the two input texts in numbers. According to one embodiment, the processor may infer data to which an entity reflecting the domain characteristics of the data obtained through the LLM has been added as input to the STS model.

According to one embodiment, the processor may identify the search item by determining whether the search item corresponding to the acquired data is a duplicate. For example, the processor may identify whether the search item corresponding to the acquired data is an item that overlaps with an existing search item (eg, a search item stored in the database 220) through the STS model. According to one embodiment, if the processor 130 determines that the search item is not duplicated, the processor 130 may store the search item in the database.

Figure 4 is a flow chart to explain a method of providing a search service according to an embodiment of the present invention.

Referring to FIG. 4, a device that provides a search service (eg, the electronic device 100 of FIG. 1) may acquire data from at least one provision server in the data acquisition step 410. The provision server may include a server that provides information about search items. According to one embodiment, the provision server may include a server operated by an accommodation information provider. In this case, the data may include information related to at least one accommodation.

In the preprocessing step 420, the device may remove unnecessary strings from the obtained data using regular expressions. Thereafter, the device may separate at least one string from the data from which the unnecessary strings have been removed by considering the domain characteristics of the acquired data.

Additionally, the device can extract entities using LLM (eg, LLM (121, 230)). For example, the device may extract at least one entity from the acquired data through the LLM. For example, the entity may include an entity capable of distinguishing properties of a search item by considering domain characteristics of the acquired data. For example, as accommodation information, the entity may include at least one of accommodation type, accommodation size (or room size), whether smoking is allowed, or the type of view seen from the room (e.g., mountain view, ocean view, etc.).

Afterwards, the device can change the data into BIO format for learning a NER model (e.g., NER model 241 in FIG. 2). The BIO format is a data format used in entity recognition tasks and may include labels indicating the beginning, interior, and non-entity parts of a named entity in the data. Afterwards, the device can digitize the data converted to the BIO format. For example, the device may define a label index for an entity type and map the BIO label to a unique number based on this.

In the NER model learning and inference step 430, the device may fine-tune data to which entities reflecting domain characteristics of the data obtained through the LLM are added to the NER model. For example, the device can fine tune the NER model using the above-described labeling data. In this process, the NER model can be trained to identify entities in given data. For example, a fine-tuned NER model may have better performance at extracting entities from raw data. Accordingly, the device can infer an entity for the acquired data through the learned NER model. For example, the device may extract entities from the acquired data through the learned NER model.

According to one embodiment, the device may perform verification on the extracted entity before learning and inferring the NER model. As an example, the device may perform verification of the extracted entity by identifying the authenticity of the extracted entity based on user input. The authenticity of the entity can be expressed, for example, as a “true” value indicating that the entity was extracted correctly and a “false” value indicating that the entity was extracted incorrectly. Thereafter, as a result of verification of the extracted entity, if it is determined that the entity was extracted correctly, the device can learn and infer the NER model using the verified entity.

In the STS model inference step 440, the device may determine whether search items are duplicates. For example, the device may determine whether search items corresponding to the acquired data are duplicates. Through the STS model, the device can identify whether the search item corresponding to the acquired data is an item that overlaps with an existing search item (eg, a search item stored in the database 220). According to one embodiment, if the device determines that the search item is not duplicated, the device may store the search item in the database.

A device that provides a search service according to various embodiments disclosed in this document (eg, the electronic device 100 of FIG. 1) may be of various types. The device may include, for example, a portable communication device (eg, a smartphone, PDA, or tablet), a computer device, a portable multimedia device, or a wearable device. Devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components.

According to various embodiments, each of the above-described components may include a single or a plurality of entities, and some of the plurality of entities may be separately arranged in other components. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. . The operations in the above description may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims (20)

In a method of providing a search service,
Obtaining data from at least one external electronic device;
Extracting at least one entity from the acquired data through a Large Language Model (LLM);
Learning and inferring a Named Entity Recognition (NER) model using the extracted entity;
Identifying a search item through STS (Semantic Textual Similarity) model inference using the inference result of the NER model; and
A method comprising extracting an entity using LLM by creating a prompt for extracting an entity capable of distinguishing properties of the search item in consideration of domain characteristics of the acquired data. In claim 1,
A method wherein the data includes information related to at least one accommodation. In claim 1,
removing unnecessary character strings from the obtained data using regular expressions; and
The method further includes an operation of separating at least one string from the data from which the unnecessary string has been removed in consideration of domain characteristics of the obtained data. delete In claim 1,
Further comprising the operation of performing verification on the extracted entity,
The operation of learning and inferring the NER model is,
A method including the operation of learning and inferring the NER model by checking the entities extracted by the LLM for true positives. In claim 5,
The operation of performing verification on the extracted entity is:
A method comprising identifying whether the extracted entity is authentic based on user input. In claim 1,
The operation of learning and inferring the NER model is,
An operation of fine tuning data to which an entity reflecting domain characteristics of the data obtained through the LLM is added to the NER model; and
A method comprising inferring an entity for the acquired data through the learned NER model. In claim 1,
The operation of identifying the search item through the STS model inference is:
A method including an operation of inferring the output of the NER model, which has learned data to which an entity reflecting domain characteristics of the data obtained through the LLM, has been added, as an input to the STS model. In claim 1,
The operation of identifying the search item through the STS model inference is:
A method comprising determining whether the search item corresponding to the acquired data is a duplicate. In claim 9,
If it is determined that the search items are not duplicates, the method further includes storing the search items in a database. In a device that provides a search service,
Memory with Large Language Model (LLM);
Communication circuitry for communicating with external electronic devices; and
a processor operatively coupled to the memory and the communication circuit;
The processor,
Obtaining data from at least one external electronic device through the communication circuit,
Extracting at least one entity from the acquired data through the LLM,
Learn and infer a Named Entity Recognition (NER) model using the extracted entities,
It is set to identify search items through STS (Semantic Textual Similarity) model inference using the inference results of the NER model,
The processor,
Setting an entity that can distinguish the properties of the search item by considering the domain characteristics of the obtained data,
A device configured to extract entities using LLM by creating a prompt for extracting entities that can distinguish the properties of the search item. In claim 11,
A device wherein the data includes information related to at least one accommodation. In claim 11,
The processor,
Remove unnecessary strings from the obtained data using regular expressions,
A device configured to separate at least one string from the data from which the unnecessary strings have been removed in consideration of domain characteristics of the acquired data. delete In claim 11,
The processor,
Perform verification on the extracted entity,
A device configured to learn and infer the NER model using entities extracted by the LLM and verified using only true positive entities. In claim 15,
The processor,
A device configured to perform verification of the extracted entity by identifying the authenticity of the extracted entity based on user input. In claim 11,
The processor,
Fine tuning the data to which entities that reflect the domain characteristics of the data obtained through the LLM were added to the NER model,
A device configured to infer an entity for the acquired data through the learned NER model. In claim 11,
The processor,
A device configured to infer by using the output of the NER model, which learned data to which an entity reflecting the domain characteristics of the data obtained through the LLM, was added, as an input to the STS model. In claim 11,
The processor,
A device configured to identify the search item by determining whether the search item corresponding to the acquired data is a duplicate. In claim 19,
The processor,
A device configured to store the search item in a database when it is determined that the search item is not duplicated.