KR20220001064A - Method and apparatus for assessing relevancy between contents and asset - Google Patents

Abstract

A technical problem to be solved through some embodiments of the present invention is to provide a method and device for automatically evaluating magnitude and direction of influence of content on a specific asset. Disclosed is a method for evaluating relevance of content and asset. According to an embodiment of the present invention, the method for evaluating the relevance of content and asset comprises: a step of obtaining a relevance score indicating the relevance between a learning target content and the asset; and a step of inputting learning target content and the obtained relevance score to a content evaluation model, and supervising and training the content evaluation model.

Description

Method and apparatus for evaluating the relevance of content and assets

The present invention relates to a method and apparatus for evaluating the relevance of content and assets. More particularly, it relates to a method and apparatus for automatically evaluating the magnitude and direction of the effect of content (eg, an event occurring in reality, etc.) of content such as a news article on a specific asset.

Stocks and real estate have traditionally been the most representative and popular investment assets. The stock price reflects the company's potential value, and its value is sensitively determined according to domestic and foreign political, economic and industrial trends. On the other hand, real estate such as buildings, land, and houses, as well as their intrinsic value of use, as investment assets that can expect a profit from the sale, are attractive targets for transactions, and real estate transactions are increasing significantly around the world. Real estate prices are also greatly affected by domestic and foreign political, economic and industrial trends. Accordingly, investors consider these information to select an investment target asset, and determine the timing of acquisition and sale of the asset.

Investors can collect various news articles through various media such as the Internet and establish investment strategies based on the collected information. However, since the price of assets such as stocks is sensitive to various and diverse information, and tens of thousands of news articles about various events are generated on the Internet every day, the events covered in these news articles have a positive effect on certain assets. It is very difficult to assess whether it has an impact or a negative effect. Furthermore, it is almost impossible to provide such an evaluation in a timely manner for the vast amount of news articles made every day, even for experts with the judgment ability necessary for such evaluation.

On the other hand, the development of artificial neural network-based machine learning technology allows computers to perform various types of human thinking and judgment processes that could not be replaced with conventional rule-based automation technology. have. Using a model learned through supervised learning, unsupervised learning, reinforcement learning, etc., human thinking and judgment, such as image recognition, text summarization, translation, and composition, as well as creation Artificial neural network-based machine learning technology is increasingly being used in the field of

Nevertheless, a method and apparatus for automatically evaluating how the content of news articles and the like affects specific assets by using machine learning technology based on artificial neural networks has not been provided.

Korean Patent No. 10-1708444 (2017.02.22. Announcement)

A technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for automatically evaluating the magnitude and direction of the effect of content on a specific asset.

Another technical problem to be solved through some embodiments of the present invention is to provide a method and an apparatus for learning a model for evaluating the relevance of content and assets.

Another technical problem to be solved through some embodiments of the present invention is to provide a method and an apparatus for efficiently obtaining a label for supervised learning a model for evaluating the relevance of content and assets from an evaluator.

Another technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for assigning an appropriate reliability score to a relevance score of content and an asset obtained from a plurality of raters.

Another technical problem to be solved through some embodiments of the present invention is to properly preprocess the relevance score before training the artificial neural network-based evaluation model using the relevance score of the content and the asset obtained from a plurality of evaluators. To provide a method and apparatus for doing so.

Another technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for automatically generating and sending a report including a news article dealing with an event affecting a specific asset.

Another technical problem to be solved through some embodiments of the present invention is to provide a method and apparatus for automatically generating and sending a report including the frequency of occurrence of an event affecting a specific asset.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, the method for evaluating the relevance of content and assets according to an embodiment of the present invention includes the steps of: obtaining a relevance score indicating the degree of relevance between the learning target content and the asset, and the learning target content and inputting the obtained relevance score to a content evaluation model to supervise learning the content evaluation model.

In an embodiment, the method for evaluating the relevance between the content and the asset may further include calculating a relevance score between the content to be evaluated and the asset by using the content evaluation model.

In an embodiment, the relevance score may include a first relevance score indicating a direction of an influence on an asset price and a second relevance score indicating a magnitude of an influence on an asset price.

In an embodiment, the obtaining of the relevance score includes displaying at least a portion of the learning target content, displaying an identifier of an asset matched with the learning target content, and inputting the relevance score The method may include displaying a user interface for receiving on the same screen as the learning target content.

In an embodiment, the matched asset may be an asset whose relevance to the content is identified as a result of text analysis of the learning target content.

In an embodiment, the relevance of the matched asset to the content may be determined based on whether an identifier of the matched asset is included in the text of the content.

In an embodiment, the relevance of the matched asset to the content may be determined based on whether an identifier of an asset belonging to the same category as the matched asset is included in the text of the content.

In an embodiment, the matched assets are a plurality, and the user interface for receiving the relevance scores may be a user interface for sequentially receiving the relevance scores respectively corresponding to the plurality of matched assets. .

In one embodiment, the matched asset is plural, and the user interface for receiving the relevance score input includes a separate user input when the relevance score corresponding to a first asset among the plurality of matched assets is input. Without it, an interface for receiving the relevance score corresponding to the second asset among the plurality of matched assets may be automatically displayed.

In one embodiment, the matched asset is a plurality, and the step of displaying the identifier of the matched asset includes grouping and displaying the plurality of matched assets based on the analysis result of the learning target content In addition, the user interface for receiving the relevance score may be a user interface for receiving the relevance score corresponding to the matched asset group.

In an embodiment, obtaining the relevance score may include obtaining the relevance score based on at least one of a direction and a magnitude of a gesture input from an evaluator.

In an embodiment, the acquiring of the relevance score includes displaying a plurality of questions including at least a part of the learning target content, and based on answers obtained from an evaluator for the plurality of questions, the learning It may include calculating the relevance score between the target content and the asset.

In an embodiment, the plurality of questions may be a dichotomous question (Yes/No Question).

In an embodiment, the supervised learning of the content evaluation model may include inputting a confidence score indicating the reliability of the obtained relevance score to the content evaluation model.

In an embodiment, the method for evaluating the relevance of the content and the asset includes: displaying a user interface for receiving a relevance score corresponding to any one of a plurality of learning target contents; the user interface is displayed The method may further include calculating the reliability score based on the time taken for the relevance score to be input after it is received.

In an embodiment, the method for evaluating the relevance of the content and the asset includes: displaying a user interface for receiving a relevance score corresponding to any one of a plurality of learning target contents; the user interface is displayed The method may further include: monitoring a behavior while the relevance score is input after being used; and calculating the confidence score based on the monitored behavior of the evaluator.

In one embodiment, monitoring the behavior of the rater may include tracking movements of the rater's eye of the rater using an imaging device.

In one embodiment, monitoring the behavior of the rater may comprise tracking the movement of the pointing means.

In an embodiment, the method for evaluating the relevance of the content and the asset may further include receiving the reliability score from an evaluator.

In an embodiment, the obtaining of the relevance score includes distributing a plurality of learning target contents to a plurality of raters, and obtaining the relevance scores for the plurality of learning target contents from the plurality of raters. It may include obtaining, and adjusting a relevance score obtained from a first rater among the plurality of raters.

In an embodiment, the adjusting of the relevance score obtained from a first rater among the plurality of raters includes: based on a relevance score obtained from a second rater among the plurality of raters, from the first rater removing bias in the obtained relevance score.

In an embodiment, at least a portion of the content allocated to the first rater among the plurality of raters is also allocated to a second rater among the plurality of raters in duplicate, and the relevance score obtained from the first rater is used. The adjusting may include a relevance score obtained from the first rater with respect to all the contents assigned to the first rater, based on the relevance score obtained for the at least some contents that are repeatedly assigned to the second rater. may include adjusting the

In an embodiment, the obtaining of the relevance score includes: obtaining a relevance score corresponding to a first group of contents among a plurality of learning target contents from a first rater among a plurality of raters; and displaying a relevance score corresponding to the reference content to the first evaluator, and obtaining, from the first evaluator, a relevance score corresponding to a content of a second group among the plurality of learning target contents. , based on the relevance score obtained with respect to the content of the second group, adjusting the relevance score obtained with respect to the content of the first group, wherein the relevance score corresponding to the reference content is evaluated It may be obtained from an administrator.

In an embodiment, the method for evaluating the relevance of the content and the asset may further include generating an asset analysis report including the calculated relevance score.

In an embodiment, the content to be evaluated is a news article, and the report may include at least a portion of a news article related to an asset and a magnitude and direction of an influence of the news article on the asset.

In an embodiment, the report may further include a graph indicating a trend in the occurrence of positive news articles or negative news articles about the asset.

In an embodiment, the report may further include an index index value calculated based on trends in the occurrence of positive news articles and negative news articles about the asset.

In an embodiment, the calculating of the relevance score between the content to be evaluated and the asset by using the content evaluation model includes: collecting a plurality of documents for a preset asset; The method may include calculating a relevance score of a preset asset, and automatically generating an asset analysis report for the asset based on a determination that the calculation result satisfies a preset condition.

In one embodiment, in the method for evaluating the relevance of the content and the asset, the step of calculating the relevance score between the content to be evaluated and the asset by using the content evaluation model includes: In response, the method may further include sending the asset analysis report.

The apparatus for evaluating the relevance of content and assets according to another embodiment of the present invention for solving the above technical problem includes the steps of: obtaining a relevance score indicating a degree of relevance between a learning target content and an asset, and the learning target content and inputting the obtained relevance score to a content evaluation model, and performing supervised learning of the content evaluation model.

A computer-readable non-transitory recording medium according to another embodiment of the present invention for solving the above technical problem comprises the steps of: obtaining a relevance score indicating a degree of relevance between a learning target content and an asset, and the learning target content and inputting the obtained relevance score to a content evaluation model, and supervising and learning the content evaluation model.

1 is a diagram for explaining input and output of an apparatus for evaluating the relevance of content and assets according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating the function and configuration of the apparatus for evaluating the relevance of content and assets described with reference to FIG. 1 .
FIG. 3 is a view for explaining an apparatus for learning the evaluation model of the relevance score evaluation unit described with reference to FIG. 2 .
4 is a flowchart of a method for evaluating the relevance of content and assets according to another embodiment of the present invention.
5 to 10 are diagrams illustrating exemplary user interfaces for obtaining a relevance score between a learning target content and an asset in some embodiments of the present invention.
11 is a diagram for explaining in more detail some steps of the relevance evaluation method described with reference to FIG. 4 .
12 is a diagram for describing in more detail some steps of the relevance evaluation method described with reference to FIG. 4 .
13 is a diagram illustrating an exemplary user interface for obtaining a confidence score for a relevance score between a learning target content and an asset in some embodiments of the present invention.
14 is a diagram for explaining in more detail some steps of the relevance evaluation method described with reference to FIG. 4 .
15 is a reference diagram for explaining an exemplary asset analysis report generated based on a degree of relevance evaluated according to some embodiments of the present disclosure.
16 is a diagram for explaining a hardware configuration of an apparatus for evaluating the relevance of content and assets according to some embodiments of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the technical spirit of the present invention is not limited to the following embodiments, but may be implemented in various different forms, and only the following embodiments complete the technical spirit of the present invention, and in the technical field to which the present invention belongs It is provided to fully inform those of ordinary skill in the art of the scope of the present invention, and the technical spirit of the present invention is only defined by the scope of the claims.

In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly defined in particular. The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is “connected”, “coupled” or “connected” to another component, the component may be directly connected or connected to the other component, but another component is formed between each component. It should be understood that elements may also be “connected,” “coupled,” or “connected.”

As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, operations and/or elements mentioned. or addition is not excluded.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining input and output of an apparatus for evaluating the relevance of content and assets according to an embodiment of the present invention.

As shown in FIG. 1 , the apparatus 10 for evaluating the relevance of the content and the asset acquires the content 1 such as a news article, the identifier 5 of the asset related to the content 1, and the content A computing device that provides a score (7) indicative of the relevance of the asset. According to some embodiments of the present invention, the relevance evaluation apparatus 10 may also provide an analysis report 9 on the asset based on the relevance between the content and the asset.

The computing device may be a notebook, desktop, or laptop, but is not limited thereto and may include any type of device having a computing function. An example of the computing device is further referenced to FIG. 16 .

1 shows as an example that the relevance evaluation device 10 is implemented as a single computing device, the first function of the relevance evaluation device 10 is implemented in the first computing device, and the second function is the second function. It may be implemented in a computing device.

According to various embodiments of the present disclosure, the relevance evaluation device 10 includes an identifier 5 of a related asset from the content 1 , a relevance score 7 indicating the effect on the related asset, and an asset analysis report ( 9) can be created.

The content 1 may be, for example, text content such as an announcement document, an IR document, and a news document. For example, announcements on the performance of specific companies, economic issues such as development plans for specific regions, issues related to trade disputes between countries, and news articles about incidents and accidents that affect the economy by shrinking consumer sentiment, etc. Content may be input to the relevance evaluation device 10 . The content 1 may be collected and obtained from a data source such as a web server of an institution that provides news articles, such as a public announcement agency, a press company, or a portal site. However, the scope of the present disclosure is not limited thereto.

The identifier 5 of the asset related to the content 1 includes, for example, a name or stock code of a stock item, a name of a specific area where real estate is located, and unique identifiers for distinguishing various assets, such as the name of an apartment complex.

The relevance score 7 representing the impact of the content 1 on the asset may be a continuous numerical value or discontinuous values or ratings representing how an event or event addressed in the content 1 affects the asset. The relevance score 7 may include a first relevance score indicating the magnitude of the impact on the asset and a second relevance score indicating the direction of the impact on the asset (positive, negative, or neutral, etc.) However, the scope of the present disclosure is not limited thereto. For example, in some embodiments of the present invention, the relevance score 7 may represent the direction and magnitude of the influence of the content 1 on the asset as an integer or character.

The asset analysis report 9 may include content content (title and/or text of a news article) about an event affecting a specific asset, and information about the magnitude and direction of impact of each content on the specific asset have. In other words, content dealing with an event that will have a strong positive impact on a specific asset registered in advance as a user's interest, content dealing with an event that will have a somewhat negative impact on a specific asset, and the frequency of occurrence of an event affecting a specific asset And information such as trends may be included in the asset analysis report for a specific asset.

The relevance evaluation apparatus 10 may generate the asset identifier 5 , the relevance score 7 , and the asset analysis report 9 from the content 1 using an artificial neural network-based model.

Hereinafter, the configuration and operation of the relevance evaluation apparatus 10 according to an embodiment of the present invention will be described with reference to FIG. 2 . As shown in FIG. 2 , the relevance evaluation apparatus 10 according to the present embodiment may include a related asset matching unit 150 , a relevance score evaluation unit 170 , and a report generating unit 190 . . However, only the components related to the embodiment of the present invention are illustrated in FIG. 2 . Accordingly, one of ordinary skill in the art to which the present invention pertains can see that other general-purpose components other than those shown in FIG. 2 may be further included. In addition, it should be noted that each of the components of the relevance evaluation apparatus 10 shown in FIG. 2 represents functionally distinct functional elements, and a plurality of components may be implemented in a form that is integrated with each other in an actual physical environment. do. Hereinafter, each component will be described.

The related asset matching unit 150 receives the content 1 . The content 1 may be text content such as publication documents, IR documents, and news documents. The related asset matching unit 150 analyzes the contents of the contents 1 and matches related assets matching the contents of the corresponding contents 1 . For example, if the content (1) is news about the performance announcement of Company S, the relevant assets are not only the stock of Company S, but also the stock of Company L, which is in competition with Company S, and T, which supplies parts to Company S. It may include stocks of companies, etc. In the process of matching the related assets from the contents of the content 1, various methods and techniques known in the prior art may be utilized. For example, in the content (1), directly or indirectly to detect a company mentioned by the name or stock code, or to companies related to the company mentioned in the content (1) or in the field mentioned in the content (1) A method of inquiring the names of affiliated companies in a database may be used. Furthermore, by using the artificial neural network model trained in advance with the learning target data in which the content (1) and the identifier of the asset related to the content (1) are labeled, the name of the related asset matching the content (1) may be obtained. have.

The related asset matching unit 150 obtains the related asset identifier 5 matching the content 1 and provides it to the relevance score evaluation unit 170 to be described later, and the related asset identifier 5 is the relevance evaluation device It can be provided as an output value of (10).

In some other embodiments, the relevance evaluation device 10 may be configured without the related asset matching unit 150 . In this case, the identifier 5 of the related asset may be given as an input value to the relevance evaluation apparatus 10 in a state in which the content is previously tagged, and may be processed by the relevance score evaluation unit 170 to be described later.

The relevance score evaluation unit 170 receives the content (1) and the related asset identifier (5), and outputs the relevance score (7) between the content (1) and the asset. The relevance score 7 may include a first relevance score indicating the magnitude of the impact on the asset and a second relevance score indicating the direction of the impact on the asset.

Although not shown, in some embodiments, the relevance score evaluation unit 170 may include a content evaluation model. The content evaluation model may be an artificial neural network-based model. The content evaluation model may be a model supervised through an evaluation model learning apparatus using the learning object content and the relevance score labeled on the learning object content 1 . The learning of the content evaluation model will be described in more detail with reference to FIGS. 3 to 12 .

The identifier 5 of the asset related to the content 1 includes, for example, a name or stock code of a stock item, a name of a specific area where real estate is located, and unique identifiers for distinguishing various assets, such as the name of an apartment complex.

The relevance score 7 representing the impact of the content 1 on the asset may be a continuous numerical value or a discontinuous value representing how an event or event handled in the content affects the asset. The relevance score 7 may include a first relevance score indicating the magnitude of the impact on the asset and a second relevance score indicating the direction of the impact on the asset (positive, negative, or neutral, etc.) However, the scope of the present disclosure is not limited thereto. example

The report generator 190 automatically generates at least a part of the report 9 analyzing a specific asset based on the content 1 and the relevance score 7 . The asset analysis report 9 includes information such as content related to events affecting a specific asset, the magnitude and direction of each content's influence on the specific asset, and the frequency and trend of events affecting a specific asset can do.

So far, the configuration and operation of the relevance evaluation apparatus 10 according to an embodiment of the present invention has been described with reference to FIG. 2 .

According to this embodiment, announcements about the performance of a specific company covered in contents such as disclosure documents, IR documents and news documents, development plans for specific regions, issues related to trade disputes between countries, social events, etc., are related to specific assets. Whether it has a positive or negative influence, a strong influence, or no influence can be provided through the relevance score (7). In other words, in addition to the facts contained in disclosure documents, IR documents, news documents, etc. and/or the opinion of the author, the relevance score (7), which objectifies and quantifies the magnitude and direction of the effect of the facts on the assets, is calculated by the general investor. can be provided to others. Based on this information, even investors without advanced expertise or insight can be helped in selecting an investment target asset or deciding when to acquire or sell an asset.

In addition, according to this embodiment, content such as news articles about events affecting a specific asset, the magnitude and direction of the influence of each content on the specific asset, frequency and trend of occurrence of events affecting a specific asset, etc. The information will be further included in the asset analysis report and made available to investors. In addition to the analyst's opinion on the asset, investors can also refer to the contents and occurrence trends of events analyzed to have an impact on the asset. be able to

Hereinafter, a process of pre-learning the content evaluation model of the relevance score evaluation unit 170 described with reference to FIG. 2 will be described with reference to FIG. 3 .

The content evaluation model may be a model based on an artificial neural network. For example, Deep Neural Network (DNN), Linear Neural Network, Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN) Networks) or a neural network model constructed by combining them may be used as the content evaluation model, but is not limited thereto.

3 shows that the evaluation model learning apparatus 30 can supervise the content evaluation model by using a plurality of learning target contents 11 and the relevance scoring result 13 corresponding to each content 11 It is a drawing. In particular, FIG. 3 shows that a plurality of learning object contents 11 are distributed to a plurality of rater devices 20a to 20n, and a relevance scoring result 13 can be obtained from the plurality of raters. In order to supervise the artificial neural network of the content evaluation model, a considerable amount of learning target data (content 11 labeled with the relevance score 13) is required, so in this embodiment, the learning target content 11 is evaluated by a plurality of evaluators. by distributing it to users, so that each evaluator can review the content of the content and assign a relevance score (13) indicating its impact on a specific asset. In this way, the relevance score result 13 and the learning target content 11 are collected and provided to the evaluation model learning apparatus 30, and the relevance score evaluation unit 170 by the evaluation model learning apparatus 30 is A content evaluation model is trained.

The relevance scoring result 13 may be understood as a label given as an input value to the supervised learning. That is, the evaluation model learning apparatus 30 uses the plurality of learning target contents 11 and the relevance scoring result 13 to update the parameters of the artificial neural network of the content evaluation model, so that the contents evaluation model can be learned. .

The content evaluation model learned in this way is used by the relevance evaluation device 10 to calculate a relevance score 7 indicating the degree of relevance between the evaluation target content 1 and a specific asset.

Hereinafter, a method for evaluating the relevance of content and assets according to another embodiment of the present invention will be described in more detail with reference to FIG. 4 .

4 is an exemplary flowchart illustrating a series of processes for evaluating the relevance of content and assets according to the present embodiment. However, this is only a preferred embodiment for achieving the object of the present invention, and it goes without saying that some steps may be added or deleted as needed.

Each step of the method for evaluating the relevance between content and assets illustrated in FIG. 4 may be performed by, for example, a computing device such as the relevance evaluation device 10 . In other words, each step of the method for evaluating the relevance of the content and the asset may be implemented with one or more instructions executed by a processor of a computing device. All steps included in the method for evaluating the relevance of content and assets may be executed by one physical computing device, but the first steps of the method are performed by a first computing device, and the second step of the method These may be performed by the second computing device. Hereinafter, it is assumed that each step of the method for evaluating the relevance of the content and the asset is performed by the relevance evaluation device 10 will be described. However, for convenience of explanation, the description of the operating subject of each step included in the method for evaluating the relevance of the content and the asset may be omitted.

Of course, the technical ideas of the embodiments described with reference to FIGS. 1 to 3 may be reflected in each operation of the method for evaluating the relevance of content and assets according to the present embodiment, even if not separately mentioned. In addition, on the contrary, the technical idea reflected in each operation of the method for evaluating the relevance of content and assets according to the present embodiment is also described with reference to FIGS. And it may be reflected in the configuration and operation of the evaluation model learning apparatus 30 .

Referring to FIG. 4 , the method for evaluating the relevance of content and assets according to the present embodiment includes the steps of supervising and learning the content evaluation model (S10 to S30) and evaluating the relevance using the learned content evaluation model and reporting It may include the step of generating (S40 to S50).

In step S10, a plurality of learning target content is obtained, and an asset matching each content is identified.

The content may be, for example, text content such as public documents, IR documents, and news documents, and may be collected and obtained from data sources such as a web server of an institution that provides news articles such as public institutions, media companies, and portal sites.

The asset matching the content is an asset that is related to the content of the content. For example, the related asset matching the news content related to the performance announcement of Company S is the stock of Company S, the stock of Company L in competition with Company S. Includes stocks and shares of company T that supplies parts to company S. As another example, related assets matching news content about the construction plan of a new railway that significantly reduces the travel time between areas A and B include apartment complexes in areas A and B.

Various methods and techniques known in the prior art may be used for identification of assets matching each content, but the present invention is not limited thereto.

In step S20, a relevance score between the plurality of learning target contents obtained in step S10 and the related asset identified in step S10 is obtained. The relevance score may include a first relevance score indicating a magnitude of an influence of the content of the learning target content on an asset and a second relevance score indicating a direction of an influence on an asset. The relevance score may be a relevance score assigned by an evaluator.

In some embodiments, a plurality of learning target contents may be distributed to a plurality of raters, and a relevance score for each of the learning target contents may be obtained from the plurality of raters. More specifically, the plurality of evaluators review the contents of the learning target content distributed to each of the evaluators using their respective evaluator devices, evaluate the relevance between the content and the related assets, and evaluate the relevance evaluation result through the evaluator device can be entered. An exemplary user interface for receiving a relevance score for content from each evaluator device will be described in more detail below with reference to FIGS. 5 to 9 .

In step S30, the plurality of learning target contents obtained in step S10, the related assets identified in step S10, and the relevance score between them obtained in step S20 are input to the content evaluation model and , the content evaluation model may be supervised learning. Specifically, the content evaluation model can be supervised learning by gradually updating the parameters of the artificial neural network of the content evaluation model by using the identifiers of the plurality of learning target contents and related assets labeled with the relevance score given by the evaluator. .

When the identifier of the content to be evaluated and the specific asset are given as input values to the content evaluation model for which supervised learning has been completed through step S30, a relevance score indicating the effect of the content to be evaluated on the specific asset is evaluated for the content. It can be output from the model.

In step S40, a relevance score between the content to be evaluated and a specific asset may be calculated using the content evaluation model learned in step S30.

Specifically, the evaluation target content and the identifier of a specific asset related thereto are input to the content evaluation model as input values, and a relevance score between the content and the asset is output from the content evaluation model. In other words, a degree of relevance indicating whether the content of an event, event, announcement, or news addressed in the content has a positive or negative impact on the asset, and whether such impact is high, low, or negligible. A score may be obtained from an artificial neural network of the content evaluation model.

In step S50, an analysis report on the asset including the relevance score is generated.

Specifically, the asset analysis report may include information about one or more contents for which a relevance score with the asset was obtained in step S40 together with the analyst's analysis opinion on the asset of interest. For example, the asset analysis report provides information such as content related to an event affecting a specific asset, the magnitude and direction of each content's effect on the specific asset, and the frequency and trend of events affecting a specific asset. It may include a report, and at least a part of the asset analysis report may be automatically generated based on the content and relevance score information.

So far, with reference to FIG. 4, a method for evaluating the relevance of content and assets according to an embodiment of the present invention has been described.

According to this embodiment, after the content evaluation model is built through the supervised learning process using the learning target content and the relevance score scored by the evaluator, news articles, public documents, and It becomes possible to automatically calculate the relevance score between the new content to be evaluated, such as an IR document, and the asset without the intervention of an evaluator. Accordingly, the coverage of the content to be analyzed and the coverage of the related assets are expanded to a degree that is incomparable with the conventional method in which evaluators with professional judgment ability directly analyze and evaluate new content generated every day.

In addition, according to this embodiment, since the time required for content analysis is shortened, when news that affects a specific asset occurs, a relevance score indicating the direction and magnitude of the influence on the asset is calculated without delay, so that investors It can be provided in a timely manner for reference. Based on this information, even investors without advanced expertise or insight can be helped in selecting an investment target asset or deciding when to acquire or sell an asset.

In addition, according to this embodiment, content such as news articles about events affecting a specific asset, the magnitude and direction of the influence of each content on the specific asset, frequency and trend of occurrence of events affecting a specific asset, etc. The information will be further included in the asset analysis report and made available to investors. In addition to the opinions of analysts on assets, investors can also refer to the contents and occurrence trends of events analyzed to have an impact on the asset. be able to

Hereinafter, exemplary user interfaces that can be used in the process of obtaining a relevance score between the learning target content and the asset from the evaluator in step S20 of FIG. 4 will be described with reference to FIGS. 5 to 10 .

5 and 6 are examples of a user interface 500 that may be provided, for example, in the rater devices 20a - 20n shown in FIG. 3 .

5 and 6 , the user interface 500 may include a first area 501 , a second area 503 , a third area 505 , and a fourth area 507 . All or at least a portion of the first area 501 to the fourth area 507 of the user interface 500 may be displayed on the same screen.

One learning target content may be displayed in the first area 501 . The learning target content includes news articles, public documents, IR documents, and the like. In some embodiments, a summary text may be displayed in the first area 501 instead of the full text of the content to be learned.

In the second area 503 , an identifier of an asset matched with the learning target content may be displayed. The matched asset is an asset related to the learning target content, and may be an asset mentioned in the learning target content or another asset belonging to the same category as the mentioned asset (eg, in the case of stocks, a company's industry, etc.). The identifier of the asset may be any type of identifier that identifies the asset, such as a name of a stock stock, a stock stock code, a name of a specific area where real estate is located, a name of an apartment complex, and the like.

On the other hand, there may be two or more assets matched with the learning target content. In other words, there may be more than one asset that is relevant to the content to be learned. For example, if the content to be learned is content that includes information about the performance announcement of Company S, the matched assets are the stock of Company S, the stock of Company L that competes with Company S, and parts for Company S. It may include stocks of the company T that supplies it. In an embodiment, when there are a plurality of assets matched with the learning target content, the user interface 500 may be configured to sequentially receive the learning target content and the relevance score of each of the plurality of assets can

Referring to the second area 503 shown in FIG. 5 , an interface that allows a relevance score for each asset to be input while changing a plurality of matched assets using the navigation buttons 551 and 553 . may be provided.

Referring to the second area 503 illustrated in FIG. 6 , all or at least some identifiers 561 to 567 of a plurality of matched assets may be displayed together in the second area 503 . Among the identifiers of the plurality of assets displayed in the second region 503 , the display attribute of the identifier 561 of the asset to which the current relevance score is to be input through the third region 505 and the fourth region 507 is different. By displaying, it can be distinguished from the identifiers 562 to 567 of other assets.

Compared with the interface shown in Fig. 5, in the case of the interface shown in Fig. 6, since the evaluator can refer to a list of a plurality of assets related to the content of the learning target content, the influence exchanged between the plurality of assets can be reduced. It has the advantage that the evaluator can assign a relevance score to the asset by considering it comprehensively. For example, the news of A itself can be judged to have a somewhat positive effect on Company S, but the news of A has a very large positive effect on Company L, which is a competitor of Company S, and consequently has a negative effect on Company S. There may be cases where it is affected. In such a case, rather than receiving the relevance score for S company from the evaluator in the state that only the news of A and the name of S company are displayed on the user interface 500, the names of all companies related to the news A are displayed on the user interface ( 500), receiving the relevance score for the company S from the evaluator while displayed together on the screen will enable the acquisition of more accurate and comprehensive evaluation results.

An input interface for receiving a relevance score may be provided in the third area 505 and the fourth area 507 . In an embodiment, the relevance score may include a first relevance score indicating a direction of an influence on an asset price and a second relevance score indicating a magnitude of an influence on the asset price.

In the third area 505 , the first area indicating whether the effect of the learning target content currently displayed in the first area 501 on the assets currently displayed in the second area 503 is positive, negative, or neutral An interface through which a relevance score can be input is provided. The evaluator may input the first relevance score by using the input interface exemplarily shown in the third region 505 of FIGS. 5 and 6 .

In the fourth area 507 , it is determined whether the learning target content currently displayed in the first area 501 has a large (important), moderate (moderate), or influence on the assets currently displayed in the second area 503 . An interface is provided for receiving a second relevance score indicating whether there are few (exclusions). The evaluator may input the second relevance score by using the input interface exemplarily shown in the fourth region 507 of FIGS. 5 and 6 .

In one embodiment, when the input of the first and second relevance scores for the asset currently displayed in the second area 503 is completed, the relation related to the next asset among the plurality of matched assets without a separate user input An interface through which a degree score can be input may be automatically provided. For example, when the input of the first relevance score and the second relevance score for the first asset among the plurality of matched assets is completed, the identifier of the second asset is displayed on the second area 503 without a separate user input. Alternatively, the display attribute of the identifier of the second asset that has already been displayed may be changed. Accordingly, the evaluator can input the relevance score between the learning target content and the second asset through the third area 505 and the fourth area 507 without a separate user input.

Hereinafter, with reference to FIG. 7 , an exemplary user interface for collectively receiving relevance scores for a plurality of assets related to learning target content in a group unit will be described.

Referring to FIG. 7 , in the present embodiment, a plurality of assets related to the learning target content displayed in the first area 501 are classified into one or more groups 571 and 572 and clustered in the second area 503 . can be displayed.

For example, when the learning target content displayed in the first area 501 is news about the performance announcement of company S, the stocks of company S as assets related to the content and stocks of company L competing with company S , shares of company T that supplies parts to company S, shares of company E that distributes products of company S, etc. may be identified as related assets. In this case, the stock of company S, the stock of company T, and the stock of company E, which are highly likely to be similar in the direction and magnitude of the impact of the news on the performance of company S, are grouped into the first group, and the The shares of potential L company may be grouped into a second group.

In the present embodiment, for the related assets displayed in the second area 503 and belonging to the currently selected group (eg, the first group 571 shown in FIG. 7 ), the third area 505 and the fourth area 507 are ), the relevance score can be entered collectively through the interface provided. Also, when the input of the relevance score for the currently selected first group 571 is completed, the second group 572 is selected without a separate user input, and the relevance of the related assets belonging to the second group 572 is completed. Scores can be entered.

Hereinafter, another exemplary user interface through which a first relevance score and a second relevance score can be input will be described with reference to FIGS. 8 and 9 , in some embodiments of the present invention.

In some embodiments of the present invention, pointing means used by the evaluator, for example, clicking a specific area on the display screen with a mouse, moving the mouse pointer in a specific direction on the display screen, or a touch gesture on a specific location of the touch screen , or through a touch and drag gesture in a specific direction on the touch screen, a user interface through which the first relevance score and the second relevance score can be input at once may be provided. Interfaces to be described below can improve the work efficiency of the evaluator and reduce the time required for the work, especially when the amount of learning target content to be scored by the evaluator is huge.

In one embodiment, an input interface such as that shown in FIG. 8 may be displayed along with the user interface 500 . The evaluator selects any one of the plurality of regions 581 to 589 shown in FIG. 8 with a pointing means such as a mouse or a touch, thereby indicating a first relevance score indicating the direction of influence (positive, neutral, or negative influence) And, a second relevance score indicating the magnitude of the influence (important, moderate, or negative) can be input with one operation.

In another embodiment, within any area of the user interface 501, the evaluator selects the direction of influence (positive, neutral, or negative influence) by moving the mouse pointer in a predetermined direction or by inputting a touch-and-drag gesture. The first relevance score to indicate and the second relevance score to indicate the magnitude of the influence (important, moderate, or negative) can be input through a single operation. For example, when a touch and drag gesture is input in the first direction 591 shown in FIG. 9 , the gesture may indicate that the first relevance score is “negative” and the second relevance score is “important”. have. When a touch and drag gesture is input in the ninth direction 599 shown in FIG. 9 , the gesture may indicate that the first relevance score is “positive” and the second relevance score is “excluded”. As described above, an interface through which the evaluator can input the first relevance score and the second relevance score through a single operation through the moving direction of the pointing means may be provided.

Hereinafter, another exemplary user interface for receiving an input of a relevance score in some embodiments of the present invention will be described with reference to FIG. 10 .

In this embodiment, the relevance score indicating the direction and magnitude of the influence of the content included in the learning target content (eg, news on a specific event, news on the current quarter's performance, etc.) on the related asset is directly input from the evaluator. Instead of receiving, at least a part of a preset algorithm is used to derive the direction and magnitude of the influence. If the evaluator directly assigns the relevance score between the learning target content and the asset, the evaluator's subjective opinion and bias may have a greater effect on the relevance score. Therefore, instead of receiving the relevance score directly from the evaluator, the relevance score is provided by presenting the evaluator to questions asking for confirmation of some objective facts to be input into the algorithm for calculating the relevance score, and receiving the answers. can be obtained. Referring to FIG. 10 , a plurality of questions may be sequentially presented in an area 591 of the user interface 500 .

In this case, by configuring the questions as binary questions (Y/N questions) rather than open-ended questions, it can be expected that the time required for answering the questions will be shortened. Referring to FIG. 10 , an interface 592 for selecting yes (Y) and an interface 593 for selecting no (N) may be provided in the user interface 500 .

For example, if the news about Company S's performance improvement is the learning content, and you want to calculate a relevance score that indicates the impact on Company L, "Are Company S and Company L competitors?", "Last quarter Was the direction of company S's earnings change and the direction of company L's earnings the same?" etc., short questions about some objective facts that need to be confirmed for the implementation of the algorithm for estimating the impact of the performance improvement of Company S on Company L are provided to the evaluator, answering yes (Y) or no (N) By obtaining an answer, it is possible to calculate an objective relevance score that involves less subjectivity of the evaluator. In addition, in this embodiment, the evaluator does not review the entire text of the learning content, but provides only answers to simple questions provided, thereby completing the provision of information necessary for calculating the relevance score, thereby reducing the evaluator's work required time and fatigue. can be expected to decrease.

So far, exemplary user interfaces that can be used in the process of obtaining a relevance score between the learning target content and the asset from the evaluator have been described with reference to FIGS. 5 to 10 .

Hereinafter, some embodiments used for learning the content evaluation model after adjusting the relevance score obtained from the evaluator will be described below with reference to FIG. 11 . Steps S210 to S230 shown in FIG. 11 may be performed as a part of step S20 described with reference to FIG. 4 .

In order to improve the quality of the content evaluation model, a significant amount of content to be learned should be labeled. Therefore, it may not be realistic for one or a small number of raters to label a vast amount of content to be learned. Accordingly, as described above with reference to FIG. 3 , a set of learning target content is distributed to a plurality of evaluators (step S210), and a relevance score for the content assigned to each evaluator is obtained (step S220). ), these can be collected and used for learning the content evaluation model. However, the evaluation criteria of each evaluator are different from each other, and each evaluator may have a certain level of bias. Therefore, in this embodiment, after each evaluator adjusts the result of scoring the relevance score (step S230), it is used for learning the content evaluation model.

In some embodiments, the scoring result of the first rater may be adjusted by comparing the scoring result of the first rater among the plurality of raters with the scoring result of the remaining raters. For example, the scoring result of the first rater may be adjusted by collectively adjusting the score of the first rater to have a mean and variance value similar to the mean and variance values of the scores of all raters other than the first rater.

In some other exemplary embodiments, in a process of distributing a set of learning target content to a plurality of raters, some content may be redundantly allocated to a plurality of raters. In other words, the predetermined learning target content may be assigned redundantly so that a relevance score is given by all evaluators. The predetermined learning target content may be understood as a kind of verification sample for estimating the bias of each evaluator. Based on the difference in the relevance scores given by each evaluator to the content duplicately assigned to all evaluators, the relevance score given by each appraiser to the remaining content that is not duplicated may be adjusted.

In some other embodiments, while receiving the relevance score for the learning content distributed to a plurality of evaluators, respectively, the content for reference and the relevance score corresponding to the content for reference may be presented (eg, displayed) to each evaluator. have. The content for reference and the relevance score are provided by the evaluation manager, and may be understood to serve as a criterion for evaluating the relevance score. After the reference content and the relevance score are presented to the evaluator, relevance scores for the remaining learning content are continuously input from the evaluator. When the acquisition of the relevance score for the learning content assigned to the evaluator is completed, the relevance score (relevance score for the content of the first group) given before the content for reference and the relevance score therefor are presented; After the reference content and its relevance score are presented, the relevance score (relevance score for the content of the second group) is compared, and the relevance to the entire content evaluated by the evaluator according to the comparison result The degree score may be adjusted. For example, the relevance score for the content of the first group may be adjusted to have a distribution similar to the distribution of the relevance score for the content of the second group.

So far, with reference to FIG. 11 , some embodiments for removing the bias of the evaluator by adjusting the relevance score obtained from the evaluator before being used for learning the content evaluation model have been described. According to the above-described embodiments, by adjusting the relevance score obtained from the evaluator, the bias or error of the relevance score that may occur when the relevance score for different content to be learned is given by a plurality of evaluators is minimized. do.

Hereinafter, with reference to FIGS. 12 and 13 , some embodiments of additionally acquiring a confidence score indicating a confidence level for a relevance score obtained from an evaluator and using the obtained confidence score for learning a content evaluation model will be described.

Even if the relevance score responded to the relevance between the learning target content and the asset is the same, sometimes the evaluator's confidence in the same response or the degree of trustworthiness of the response may be different from each other. For example, even if one and the same rater answered that the impact of news A on Company S was “positive” and that the impact of news B on Company L was “positive”, the rater himself may have different degrees of certainty. Accordingly, the quality of the content evaluation model can be further improved by learning the content evaluation model by additionally reflecting such a difference.

12 is a flowchart illustrating a process of using a confidence score for a relevance score obtained from an evaluator for learning a content evaluation model, according to some embodiments. Steps S311 to S315 and S320 shown in FIG. 12 may be performed as a part of step S30 described with reference to FIG. 4 .

In some embodiments, a confidence score may be selected based on the input required time of each relevance score (step S311). Specifically, the reliability of the obtained relevance score with respect to the impact of news A on S company is the time taken until the relevance score is input after the interface for inputting the relevance score is displayed to the rater. It can be calculated based on time. If the time taken for inputting the relevance score between News A and Company S is short, it is highly likely that the evaluator has greater confidence in the input relevance score. On the other hand, if the evaluator inputs a relevance score after thinking for a long time about the impact of news A on S company, the evaluator's confidence in the input relevance score will be relatively weak. Therefore, the longer the time required to input the relevance score, the lower the reliability score, and the shorter the relevance score input time is, the higher the reliability score is calculated. Available.

In some other embodiments, a confidence may be calculated based on the monitored evaluator's behavior while entering the relevance score (step S313). For example, while inputting the relevance score, the movement of the eyeball of the rater is tracked through an imaging device installed in the rater device, and a confidence score may be calculated based on a pattern observed in the tracked movement. As another example, the reliability score may be calculated based on a movement pattern of a pointing means (eg, a mouse) used by the rater while inputting the relevance score.

In some other embodiments, the reliability score may be directly input from the evaluator (step S315). 13 illustrates an exemplary user interface 500 that may directly receive a confidence score from an evaluator. The evaluator can measure the relevance score input through the third area 505 and the fourth area 507 through the input interface provided in the fifth area 509 of the user interface 500 shown in FIG. 13 . You can explicitly input your level of confidence or confidence directly.

The reliability score obtained by the above-described exemplary methods may be input to the content evaluation model together with the relevance score, and may be used for learning the content evaluation model. By additionally using the reliability score to train the content evaluation model, the quality of the content evaluation model may be further improved.

So far, with reference to FIGS. 12 and 13 , some embodiments have been described in which a reliability score for the relevance score obtained from the evaluator is additionally obtained and used for learning the content evaluation model.

Hereinafter, with reference to FIGS. 14 and 15 , a process of calculating a relevance score for an evaluation target content and an asset using a content evaluation model that has been trained and generating an asset analysis report including the same will be described.

14 is a view for explaining in more detail steps S40 and S50 of the relevance evaluation method described with reference to FIG. 4 .

Referring to FIG. 14 , first, a plurality of contents identified as being related to a preset asset in step S410 are collected. For a specific method of analyzing the degree of relevance between a preset asset and content, for example, reference may be made to the description of the related asset matching unit 150 of FIG. 2 and the step S10 of FIG. 4 .

In step S420, a relevance score between the plurality of contents and a preset asset is calculated. Each of the plurality of contents and the identifier of the preset asset may be input to a contents evaluation model, and a relevance score output from the contents evaluation model may be obtained. The content evaluation model is an artificial neural network-based model, and may be a model supervised through an evaluation model learning apparatus using the learning object content and a relevance score labeled on the learning object content. As described above, the relevance score may include a first relevance score indicating the direction of the influence on the asset price and a second relevance score indicating the magnitude of the influence on the asset price.

The relevance score calculated in step S420 indicates whether the content (event, accident, announcement, change, performance, etc.) handled in specific content (eg, news article) has a positive or negative effect on the preset asset. may indicate whether the magnitude of the influence is strong important content, or whether the magnitude of the influence is moderate, or whether the magnitude of the influence is insignificant. As described above, the relevance score calculated by the content evaluation model is provided to general investors along with the content itself, so that investors who do not have advanced expertise or insight can understand what the content in the content implies, And with an understanding of how it will affect the price of a specific asset, you will be able to select an investment target or decide when to acquire or sell an asset.

Referring back to FIG. 14 , steps S510 and S520 may be performed as part of the asset analysis report generating step S50 including the relevance score shown in FIG. 4 .

First, in step S510, it is determined whether the relevance score calculated in step S420 satisfies a preset condition, and if the condition is satisfied, an asset analysis report may be automatically generated. Here, the preset condition refers to whether the sum or average of the relevance scores for the asset exceeds a preset value as a result of calculating the relevance score between the content collected for a certain period and a specific asset (eg, interest asset) can be For example, content with a relevance score indicating a positive impact on Company S (eg, news articles) was collected more than usual at a certain level, or, conversely, content evaluated to have a negative impact on Company S was usually Whether more than a certain level or more has been collected may be used as the preset condition.

The asset analysis report may include information such as content related to an event that affects a specific asset, the magnitude and direction of the influence of each content on the specific asset, and the frequency and trend of occurrence of events affecting the specific asset. . 15 illustrates an example asset analysis report 1401 . In the asset analysis report 1401, a region 1410 in which contents 1411 to 1413 related to an event having a positive effect on a specific asset are described, and contents 1421 to 1422 related to an event having a negative influence on the asset. This described region 1420 may be included. In addition, the asset analysis report 1401 may further include information on the magnitude of the impact of the events on the asset. In addition, in the asset analysis report 1401, a graph 1430 indicating the occurrence frequency and trend of events that have a positive effect on the asset and a graph 1440 indicating the occurrence frequency and trend of events that have a negative impact on the asset are included. more may be included. In addition, the asset analysis report 1401 may further include an index index calculated based on the occurrence frequency or trend of events that have a positive effect on the asset and the occurrence frequency or trend of events that have a negative effect on the asset. The index index is a numerical value that comprehensively reflects positive events and negative events, and may be understood as a number that can be referenced in the process of determining whether to invest in the asset.

Referring back to FIG. 14 , in step S520 , the report automatically generated in step S510 is displayed to the manager, and when an approval input is obtained from the manager, the report is registered to users (eg, general investors) can be sent to

As in the embodiment described with reference to FIG. 14 so far, if the relevance score calculated based on the automatically collected content for the preset asset satisfies the preset condition, the asset analysis report is automatically generated without human intervention. By creating such an event, it is possible to provide users with an asset analysis report indicating the content related to the event and the impact that the event will have on the asset within a minimum time from the occurrence of a specific event affecting the asset.

So far, an apparatus and method for evaluating the relevance of content and assets according to various embodiments of the present invention have been described with reference to FIGS. 1 to 15 .

According to embodiments of the present invention, after establishing a content evaluation model through a supervised learning process using the learning target content and the relevance score scored by the evaluator in relation thereto, news articles continuously generated every day in addition to the learning target content; It will be possible to automatically calculate the relevance score between assets and new content to be evaluated, such as disclosure documents and IR documents, without the intervention of an evaluator. Accordingly, the coverage of the content to be analyzed and the coverage of the related assets are expanded to a degree that is incomparable with the conventional method in which evaluators with professional judgment ability directly analyze and evaluate new content generated every day.

In addition, according to embodiments of the present invention, in addition to facts and/or opinions of authors included in content such as disclosure documents, IR documents, and news documents, the magnitude and direction of the influence of the facts on assets are objectified and quantified A relevance score can be provided to general investors and the like. Based on this information, even investors without advanced expertise or insight can be helped in selecting an investment target asset or deciding when to acquire or sell an asset.

In addition, according to embodiments of the present invention, since the time required for content analysis is shortened, when news that affects a specific asset occurs, a relevance score indicating the direction and magnitude of the influence on the asset is calculated without delay. , it will be provided in a timely manner for investors to refer to.

In addition, according to embodiments of the present invention, content such as news articles about an event affecting a specific asset, the magnitude and direction of the influence of each content on the specific asset, the frequency of occurrence of an event affecting the specific asset, and Information such as trends can be additionally included in the asset analysis report and provided to investors. In addition to the analyst's opinion on the asset, investors can also refer to the contents and occurrence trends of events analyzed to have an impact on the asset. be able to

In addition, according to embodiments of the present invention, if the relevance score calculated based on the automatically collected content for a predetermined specific asset satisfies a predetermined condition, the asset analysis report is automatically generated without human intervention, Asset analysis reports can be provided to users within a minimum period of time from the occurrence of a specific event affecting the asset.

Hereinafter, with reference to FIG. 16 , an exemplary computing device 1500 capable of implementing the apparatus 10 for evaluating the relevance of content and assets in some embodiments of the present invention will be described.

16 is a hardware configuration diagram illustrating an exemplary computing device 1500 capable of implementing the apparatus 10 for evaluating the relevance of content and assets according to some embodiments of the present invention.

16 , the computing device 1500 loads one or more processors 1510 , a bus 1550 , a communication interface 1570 , and a computer program 1591 executed by the processor 1510 . ) may include a memory 1530 and a storage 1590 for storing the computer program 1591 . However, only the components related to the embodiment of the present invention are illustrated in FIG. 10 . Accordingly, one of ordinary skill in the art to which the present invention pertains can know that other general-purpose components other than the components shown in FIG. 16 may be further included.

The processor 1510 controls the overall operation of each component of the computing device 1500 . The processor 1510 includes a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphic processing unit (GPU), or any type of processor well known in the art. can be In addition, the processor 1510 may perform an operation on at least one application or program for executing the method according to the embodiments of the present invention. The computing device 1500 may include one or more processors.

The memory 1530 stores various data, commands, and/or information. The memory 1530 may load one or more programs 1591 from the storage 1590 to execute a method according to embodiments of the present invention. The memory 1530 may be implemented as a volatile memory such as RAM, but the technical scope of the present invention is not limited thereto.

The bus 1550 provides a communication function between components of the computing device 1500 . The bus 1550 may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

The communication interface 1570 supports wired/wireless Internet communication of the computing device 1500 . Also, the communication interface 1570 may support various communication methods other than Internet communication. To this end, the communication interface 1570 may be configured to include a communication module well-known in the art.

According to some embodiments, the communication interface 1570 may be omitted.

The storage 1590 may non-temporarily store the one or more programs 1591 and various data.

The storage 1590 is a non-volatile memory such as a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, or well in the art to which the present invention pertains. It may be configured to include any known computer-readable recording medium.

The computer program 1591 may include one or more instructions that, when loaded into the memory 1530 , cause the processor 1510 to perform methods/operations according to various embodiments of the present invention. That is, the processor 1510 may perform the methods/operations according to various embodiments of the present disclosure by executing the one or more instructions.

In this case, the apparatus 10 for evaluating the relevance of content and assets according to some embodiments of the present invention may be implemented through the computing device 1500 .

So far, various embodiments of the present invention and effects according to the embodiments have been described with reference to FIGS. 1 to 16 . Effects according to the technical spirit of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

The technical ideas of the present invention described with reference to FIGS. 1 to 16 may be implemented as computer-readable codes on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). can The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

In the above, even though all the components constituting the embodiment of the present invention are described as being combined or operating in combination, the technical spirit of the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more.

Although acts are shown in a particular order in the drawings, it should not be understood that the acts must be performed in the specific order or sequential order shown, or that all illustrated acts must be performed to obtain a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can practice the present invention in other specific forms without changing the technical spirit or essential features. can understand that there is Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the technical ideas defined by the present invention.

Claims (12)

A method performed on a computing device, comprising:
obtaining a relevance score indicating relevance between the learning target content and the asset; and
Injecting the learning target content and the obtained relevance score to a content evaluation model, supervising learning the content evaluation model
including,
The step of obtaining the relevance score comprises:
distributing a plurality of learning target contents to a plurality of evaluators;
obtaining the relevance score for the plurality of learning target contents from the plurality of raters; and
adjusting a relevance score obtained from a first rater among the plurality of raters;
containing,
How to evaluate the relevance of content and assets. According to claim 1,
Adjusting the relevance score obtained from a first rater among the plurality of raters includes:
removing bias in the relevance score obtained from the first rater based on the relevance score obtained from a second rater among the plurality of raters;
How to evaluate the relevance of content and assets. According to claim 1,
At least a portion of the content allocated to the first rater among the plurality of raters is also redundantly allocated to a second rater among the plurality of raters,
Adjusting the relevance score obtained from the first rater comprises:
adjusting the relevance score obtained from the first rater with respect to all the contents assigned to the first rater based on the relevance score obtained with respect to the at least some contents that are repeatedly assigned to the second rater; containing,
How to evaluate the relevance of content and assets. According to claim 1,
calculating a relevance score between the content to be evaluated and the asset by using the content evaluation model
further comprising,
How to evaluate the relevance of content and assets. According to claim 1,
The relevance score includes a first relevance score indicating the magnitude of the influence on the asset price and a second relevance score indicating the direction of the influence on the asset price,
How to evaluate the relevance of content and assets. According to claim 1,
The step of obtaining the relevance score comprises:
displaying at least a portion of the learning target content;
displaying an identifier of an asset matched with the learning target content; and
Displaying a user interface for receiving the relevance score on the same screen as the learning target content
containing,
How to evaluate the relevance of content and assets. According to claim 1,
The step of obtaining the relevance score comprises:
displaying a plurality of questions including at least a part of the learning target content; and
Calculating the relevance score between the learning target content and the asset based on the answers obtained from the evaluator for the plurality of questions
containing,
How to evaluate the relevance of content and assets. According to claim 1,
The step of supervising the content evaluation model is,
Including the step of inputting a reliability score indicating the reliability of the obtained relevance score to the content evaluation model,
How to evaluate the relevance of content and assets. 5. The method of claim 4,
generating an asset analysis report including the calculated relevance score
further comprising,
How to evaluate the relevance of content and assets. A method performed on a computing device, comprising:
obtaining a relevance score indicating relevance between the learning target content and the asset; and
Injecting the learning target content and the obtained relevance score to a content evaluation model, supervising learning the content evaluation model
including,
The step of obtaining the relevance score comprises:
obtaining a relevance score corresponding to a content of a first group among a plurality of learning target contents from a first rater among a plurality of raters;
displaying a reference content and a relevance score corresponding to the reference content to the first evaluator;
obtaining, from the first evaluator, a relevance score corresponding to a second group of contents among the plurality of learning target contents; and
adjusting a relevance score obtained with respect to the content of the first group based on the relevance score obtained with respect to the content of the second group;
including,
The relevance score corresponding to the reference content is obtained from the evaluation manager,
How to evaluate the relevance of content and assets. An apparatus for evaluating the relevance of content and assets, performing the method according to any one of claims 1 to 10. A computer program for causing a computer to perform the method for evaluating the relevance of content and assets according to any one of claims 1 to 10 is stored therein, a computer-readable non-transitory recording medium.

Images