JP3465815B2 - Text filtering system - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a text filtering system for delivering electronic text data sent from a news agency or a newspaper company to a plurality of users.

[0002]

2. Description of the Related Art Conventionally, there is a system for filtering a large amount of electronic text data such as electronic news and delivering it to a plurality of users. Monitoring of full-text data in these systems is a search by batch processing at a fixed time, and should be called clipping. On the other hand, the conventional so-called real-time monitoring is monitoring only a part of a sentence, for example, a headline of a newspaper or the like, and is not monitoring the whole sentence. That is, in the conventional system,
Full-text based text monitoring with a time delay, or only part of the text was monitored even in real time.

[0003]

However, users (operation managers) such as institutional investors and financial institutions involved in asset management need to acquire data as early as one second and make a decision whether to sell or buy. is there. Even now, with regard to English text data, required text data can be acquired in real time by setting predetermined conditions. On the other hand, in the case of text data in a language such as Japanese where there is no break between words, full-text search is performed for only the necessary text data that is sent from time to time, and real-time search is performed. It was difficult and time-consuming to deliver. Here, the full-text search refers to searching or filtering all character strings from the beginning to the end of text data.

Further, it is conventionally possible to collate all search terms with one scan of text data by creating one finite automaton with all search data included in filtering conditional expressions of many users. is there. However, the conventional filtering method using the automaton is not practical because of a lot of noise.
For noisy data, for users who have to make an instant judgment on the acquired text data,
It will be a heavy burden. Further, as the number of users increases and the number of keywords for filtering increases, the size of the automaton increases, and the processing takes time. Therefore, even if it is called real-time processing, the conventional method may take tens of minutes. Since the automaton expands into memory, the larger the size, the larger the amount of memory used and the pressure on the memory, resulting in a decrease in the overall processing speed.
As described above, the filtering process using the automaton can process the text data at a high speed, but when the size of the automaton becomes large, the processing speed becomes slow, and the filtering using the automaton has a problem that it is noisy.

The present invention has been made based on the above circumstances, and a text data filtering system and a text data filtering system capable of filtering text data in real time with respect to text data in a language in which words and words are not separated from each other, such as Japanese. The purpose is to provide a text filtering method.

[0006]

In order to achieve the above object, a text filtering system according to the present invention has a dictionary in which a predetermined character string and an identification code corresponding to the character string are written in an automaton. If the corresponding text string is present in the text data and filtering is performed on the input text data, an identification code assigning means for assigning a corresponding identification code to the text data, and a text to which the identification code is assigned A noise removing unit that searches the data for characters before and after the character string and, when a predetermined character is added, deletes the assigned identification code to fix the identification code. It is characterized by doing.

[0007] The filtering condition created and registered for each user in advance by using the free keyword, the identification code and the logical operator is expanded into the automaton, and the text data to which the identification code is added is scanned and filtered. It is desirable to have filtering means for outputting the results. To achieve the above object, the text filtering method according to the present invention includes, as a primary process, a step of adding an identification code by pre-filtering, and a step of removing noise from the added identification code. The present invention is characterized by including a step of performing a filtering process according to a filtering condition set by a user and a step of removing noise of an identification code.

In the present invention, as for the items that can be narrowed down in common, the filtering of the automaton is used, the target profile is narrowed down by this, and further the individual search is performed for each profile to Real-time processing with less noise in the data is now possible. Here, the profile is a search condition as well as a filtering condition set by the user.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [Configuration of Embodiment] FIG. 1 is a block diagram of a text filtering system according to an embodiment of the present invention. The text data system shown in FIG. 1 manages the input of electronic text data from various electronic information sources 10, and a primary receiving code for the text data sent from the data reception server 12. A filtering server 16, a storage unit 18 for backing up text data with a stock code, a search server 20 for registering text data with a stock code in a database 22 and performing full-text search. A secondary filtering server 24 that filters text data according to a filtering condition set by the user, and a recording unit 26 that stores the result of filtering by the secondary filtering server.
And a web server 28 for controlling the exchange of data with the terminal
With. Each server has a transmission / reception unit (not shown) for transmitting / receiving data to / from another server. The text data of this embodiment is assumed to be data used in financial securities business such as corporate information and newspaper nice. Further, the entire system fetches and updates the data about the user set by the system administrator once a day, for example, at 12:00 midnight. Further, the text data sent from each electronic information source is a few kbytes, and the system of the present embodiment delivers the text data of such a size in real time. The system of this embodiment uses Windows NT as the OS of each server.

[Data receiving server] Data receiving server 1
Second, the text data sent from the electronic information source 10 is temporarily recorded in the news queue file storage unit 14 and then sent to the primary filtering server. When sending the text data sent from moment to moment to the primary filtering server, the text data is temporarily stored in the queue file recording part in order to match the timing of sending and to send it according to the processing of the primary filtering server. It is necessary. Further, when the processing of the primary filtering server is delayed for some reason, the received text data is temporarily received in the news queue file storage unit 1 so that the text data is not missed from the electronic information source 10.
Store in 4.

[Primary Filtering Server] In the primary filtering server, a classification code is added by using text pattern matching processing as preprocessing, a stock code is added by filtering of an automaton, and a noise removal dictionary (table) is referred to. , To remove noise. The primary filtering server 12 that performs this processing includes a text editing unit 121 that removes extra blanks from the text data and shapes the text data, and a classification code addition unit 12 that adds a classification code to the text data.
2, a brand code assigning unit 123 that assigns a brand code that is an identification code to the text data, a noise removing unit 124 that removes noise from the text data to which the brand code is assigned, and the text data once. It has a text search engine 125 that can match many filtering conditions only by scanning. In addition to this, the primary filtering server records, in the backup recording unit 18, text data in which one record is data sent to the search server or the secondary filtering server as backup data. If your system crashes or you want to use your backup data elsewhere,
Use this backup data. Text editor 1
21 shapes the received text data. That is, usually, in the news text, spaces may be inserted on both sides of the text in order to make it easy to see when displayed on the screen. If this kind of unique formatting is done in the space, words may be separated at the time of searching or filtering, or HT may occur on the client terminal.
It is inconvenient when displaying LM. Therefore, it is necessary to format the text data. In addition, the text editing unit removes line breaks other than sentence breaks from the news text and converts half-width katakana to full-width katakana. Further, since the data format handled by the search engine is the CSV format in which each item is separated by a comma, the text editing unit converts the text data and the code into the CSV format and outputs the converted data.

[Category Code Addition Unit] Next, the processing in the classification code addition unit 122 will be described. Classification codes include, for example, from the standpoint of an investor, the source code, the information category code, the personnel information code,
Rating information code etc. are required. The classification code is the source (information source) number from the genre code and news title (headline) according to the instructions of the classification conversion table,
Convert to genre number, (genre) detailed number, Japanese-English type. The classification conversion table to which the classification code is added is shown in FIG.
According to the rule shown in (A), the items are CSV (see FIG. 2B) separated by commas. The genre code of the item name is a code attached to the news sent in advance. In the target field, the search target is divided into three categories: "no setting", "title", and "body". As operators, exact match, forward match, backward match, intermediate match, and in-box intermediate match are provided. Enclosed character 1 is when you use "Inside enclosing character" in the operator,
It defines the character to start the enclosing, and the enclosing character 2 is
It defines the end character. The source number is the number corresponding to the genre code. This code number is used to determine where the information source is. The genre number is a number assigned in NAA (Nikkei Asahi All) according to the condition of an operator. The detailed number is used when the genre number is further subdivided and used. By looking at the genre number and the detailed number, it is possible to judge whether the text data is personnel information or rating information. The Japanese-English number is "0" for Japanese news and "1" for English news.
Attach. The number of valid days indicates the period during which news can be searched by the number of days from the issue date. Next, a specific example of CSV will be described with reference to FIG.
The first line in the same figure (B) is a genre called NAA, and if there is a prefix <Asahi> in the title, the source (information source) number is "1", the genre number is "2", and the Japanese-English number is " It means that "0" and "0" are given as the number of effective days. In this example, detailed numbers are omitted. Further, the last line in the figure (B) is a genre called NAA, and if there is a character string of NQN in the characters enclosed in black brackets, the source number is "1", the genre number is "8", and the details. This means that "5" is assigned as the number, "0" is assigned as the Japanese-English number, and "0" is assigned as the number of valid days.

Using such a classification conversion table, text pattern matching is performed on the sent text data, and a classification code is added. Also, in order to reduce the processing time as much as possible and to simplify the processing procedure, matching is performed in order from the top of the classification conversion table, and if there is a hit, the classification number is given and no further search is performed. . Therefore, it is necessary to consider that the classification numbers to be given priority should be placed at the top of this table. In this process, most of the operators used are prefix match or suffix match, so only the character strings at the beginning or end of the title or body are matched, so this processing time is extremely short.

[Brand Code Assigning Unit] Next, the assigning of a brand code will be described. FIG. 3 is a diagram showing a part of a brand dictionary used when a brand code is assigned. Stock dictionary 4
2a is a stock code, for example, 660, as shown in FIG.
1 and a brand name, for example, Japan, Japan Manufacturing Co., Ltd., Nihon. A stock dictionary is created and recorded for each stock code in advance. The stock dictionary is expanded in the automaton when the system is started,
It can be rewritten by reactivating the system once a day, for example, at 12 o'clock in the middle of the night, in order to capture information on newly listed and abolished brands. Since the system is started up instantly, it does not affect the distribution service.

If the title includes the character string "personnel", the text in the title is not searched, so that noise when assigning the stock code can be reduced. Personnel names such as Honda, Takeda, Matsushita, etc. are often described in personnel news, and since these names are the same as company names such as Honda Motor Co., Takeda Pharmaceutical Company, Matsushita Electric Industrial, etc., Stock codes are often added to HR news. In this way, many people's names are the same as the stock names, so if the text data title has the character string "HR" in advance, when assigning the classification code, add a code indicating that it is data related to personnel. To do. By not using the text data to which this code is added as the text data to be filtered by the automaton, it is possible to improve the processing speed of assigning the stock code and reduce noise.

The brand code assigning unit 123 expands the brand dictionary into an automaton, inputs the received text data to the text search engine in real time, and scans it. If there is a corresponding character string in the text data, the issue code corresponding to that character string is output. The output brand code is described by a half-width numerical value at a predetermined position in the text data. For example, in the case of Japan Manufacturing Co., Ltd., the code number of 6601 is assigned separately from the text data. The text data is divided into several items such as the information of the delivery source, the title, the text, the date, and the time by the tab code. The item of the stock code is created there, and the corresponding stock code is given numerically. In this way, the next noise removal process is performed based on the text data to which the brand code is added.

[Noise Removal Unit] Next, noise removal will be described. The filtering process of the automaton can be processed in real time, but since there is a lot of noise, it is necessary to devise a way to reduce that noise. It is also possible to remove noise for each user. However, it is not easy for the user to create a conditional expression for reliable noise removal, and the size of the automaton becomes too large, which is not realistic. Therefore, in this embodiment,
The noise is removed by the noise removing unit after the brand code is added by the filtering process of the automaton. FIG. 4A is a diagram showing a description method of the noise removal dictionary, and FIG. 4B is a diagram showing a part of the noise removal dictionary described by the description method of FIG. 4A. Noise removal dictionary 4
2b is based on the rule shown in FIG.
mma Separated Value) Created by text. That is, a numerical value, for example, 6601 in the case of Nippon Seisakusho, is described in the first column for each issue code as the issue code. In the second column, the stock name to be checked,
For example, Japan is described in the third column with the required number of operation codes and character strings. For example, @ 1 conversion, @
Described as 1 train, @ 1 information, @ 1 city. After that, these three columns are set as 1, that is, the stock code, the stock name that needs to be checked, and the combination of the operation and the character string are set as 1, and the description of the set is repeated. Create a noise dictionary. Once created, also register the noise removal dictionary.

When the stock dictionary of the present embodiment is used, for example, the stock code 6601 can be used if the text data contains a character string of "Japan", "Japan Works", or "Nihon" as a target character string. The stock code 6051 is added to the text data. Therefore, even when the text data includes the character strings “Nippon Kasei”, “Japan Information Engineering”, “Nippon Electric Wire”, and “Japan City”, the stock code 6601 is given. in this way,
When a brand code is assigned using the brand dictionary shown in FIG. 4, noise is included. Of course, if the target character string is "Japan Seisakusho" only, it is possible to assign a stock code without noise. However, if the stock dictionary is created in this way, it is not possible to search for the Japanese manufacturing company whose abbreviated name is "Japan" in the text data. That is, the filtering result becomes unreliable. Therefore, in this embodiment,
In the first stage, a string of characters including such abbreviations is searched for by using automaton filtering, and a stock code is assigned. In the second step, the noise is removed by using the noise removing unit.

The noise removing unit 124 carries out a process of fixing the brand code provisionally assigned by the brand code assigning unit 123. That is, it is determined whether or not the issue code is removed by referring to the noise removal dictionary for each issue code provisionally provided. For example, with respect to the issue code 6601, if there is "ka", "den" or "city" after the character string "Japan" in the text data, the issue code 6610 given to this text data is removed. Thus, by referring to the dictionary created based on the rule shown in FIG. 4, noise is removed and the stock code is determined.

By growing this noise removal dictionary,
The accuracy of noise removal can be improved. This noise dictionary does not affect the processing time even if the size increases. Since it suffices to make a necessary reference only for the assigned brand code, this noise processing takes a long processing time, and real-time processing is not disabled.

In addition, in the case of news to which a certain classification code is added, for example, news relating to personnel, since the title (headline) is usually "Ox company personnel", only the title is searched. , Assign a stock code.
As a result, it is possible to reduce noise that occurs when a brand code is added and to perform a quick filtering process.

[Secondary Filtering Server] The secondary filtering server 24 includes a filtering unit 241 for filtering an automaton under a filtering condition set by a user, and a code checking unit 242 for checking a code for a profile hit by the filtering of the automaton. , A text search engine 243 capable of matching many filtering conditions by scanning the text data once. Further, the secondary filtering server stores the filtering condition set by the user in the profile information recording unit 263, and responds to the inquiry about the hit data from the user by referring to the filter result recording unit 262. To do. Furthermore, since a new user is added or a user's option is changed, the updated user information is referred to by referring to the user table 41 set by the administrator once a day. To record. The profile information storage unit 263 records filtering conditions for each user. The filtering result recording unit 262 records the heading of the text data that is the result of the secondary filtering. The headline that is the filtering result is instructed by the client via the filtering unit of the web server,
Displayed on the client display. The text is recorded in the database 22 and displayed on the display device of the client via the search unit of the web server. 30 filtering results are recorded for each profile.
In the present embodiment, the profile is 9 in advance for each user.
You can register the case. By recording the filtering result in this way, the filtering result can be reused. That is, not only can the filtering result be displayed in real time, but the filtering result can be displayed at any time when the user needs it. The system of the present embodiment is more convenient than the conventional system in which the filtering result is not recorded.

The text data to which the fixed brand code is added is filtered by the secondary filtering server. In the profile information recording unit 263, filtering conditions created in advance by each user using a free keyword, a stock code, and a logical operator are registered. The filtering unit 241 expands this filtering condition into an automaton, inputs the received text data with brand code into the text search engine in real time, and scans it. If there is one that meets the filtering conditions, the corresponding user is notified of that fact via the web server. This allows the text data to be delivered to the user in real time.

By the way, when the user sets, for example, 1998 as the stock code, this text data is hit even if the year data 1998 is included in the text data to be filtered. Therefore, for the issue code, after the text data is hit, it is confirmed whether the issue code, in this case 1998, is in the item of the issue of the text data. Look at the item of the brand delimited by the tab, and if it is not there, determine that it is noise. Such noise will not occur if the stock code is searched by looking only at the stock item when searching. However, this method is time consuming because it has to look up all stock codes. Therefore, it is better in terms of processing speed to collate with other free keywords in the automaton in one scan and individually check only the stock code that is hit later.

Each time the user changes the filtering conditions, the automaton of the secondary filtering server redeploys the changed filtering conditions in the automaton. However, since the size of this automaton is smaller than that of the conventional one by only the sentences of the stock dictionary, the processing time when re-developing into the automaton can be shortened as compared with the conventional system.

In the user table, news sources that can be searched for are set individually for each user. If the news source is charged, it is also determined whether or not the news source has been paid for a hit by the filtering process of the automaton. In this way, the noise-free data is distributed to the user by gradually narrowing it down.

As in the present embodiment, the search processing is performed by gradually narrowing down the stock code confirmation processing, the table reference processing, and the like based on the result of filtering using the automaton, so that efficient and accurate processing can be performed. You can search. In other words, real-time filtering processing with less noise can be performed.

The text search engine used in the primary filtering server and the secondary filtering server of this embodiment is a commercially available one. This text search engine is provided as several functions. Here, detailed description of these functions is omitted. [Search Server] The search server 20 is a full-text search engine 2
01 and a thesaurus function not shown. The full-text search engine is an engine that can register and search text data in real time. There are various methods such as n-gram index method and bitmap index method for full-text search. However, this full-text search engine has the same or higher speed as the processing speed at the time of text data registration, that is, filtering processing. It uses a full-text search engine with processing speed of. This enables real-time registration / search as well as filtering. Further, this full-text search engine has basic performance required for a database in addition to searching text data. For example, date, numeric and string items can be defined. In this embodiment,
This full-text search engine uses a commercially available one. Therefore, detailed description of this full-text search engine is omitted.

[Web Server] The web server 28 includes a search unit 281, a filtering unit 282, and a main unit 2.
83 is a server that controls the sending and receiving of text data and the like with the client. The search unit 281 displays a search screen on the display device of the client and controls the search performed by the user for the past text data including the result of real-time filtering. The filtering unit displays the filtering screen on the display device of the client, informs the client of the filtering result in real time, and sends the filtering condition set by the client to the secondary filtering server. The main unit 283 controls the name and ID number of the accessing user, confirms the option the user is buying, and determines whether or not to grant access. The client side uses a browser. In this embodiment, the web server and the browser send and receive information between the system and the client, but the present invention is not limited to this, and other client / server type or mail server may be used. You may use it.

The system administrator sets the contents of the user table 41 and the dictionary table 42. [Operation of Embodiment] FIG. 5 is a flow chart when a stock code is added in the primary filtering server. This flow is executed when the system is started up. In step 1, the stock dictionary is read from the dictionary table 42 and developed in the automaton. Step two
Then, similarly, the noise removal dictionary is read from the dictionary table 42 and developed in the memory. In this state, it waits for the text data to be transmitted from the data receiving server (step 3). When the text data is transmitted from the data receiving server, the process proceeds to step 4, and the text data is provided with a classification code indicating the above-mentioned information source by text pattern matching. If there is no text data, the process proceeds to step 9 to determine whether or not to end the system. Next, the issue dictionary is referenced to perform filtering, and the issue code corresponding to the text data is provisionally added. This issue code is not limited to one, and if there are multiple corresponding issues, all issue codes are assigned.

In step 6, the noise dictionary is referred to for each provisionally assigned brand code, and if it is noise described in the noise dictionary, the brand code is deleted. In this way, the stock code provisionally given is fixed. The text data to which the decided brand code is added is sent to the search server (step 7) and the secondary filtering server (step 8). Next, in step 9, it is determined whether or not the system is to be terminated. If not, the process proceeds to step 3 and waits for reception of text data. If it is finished, the processing in the primary filtering server is finished.

FIG. 6 is a flow chart when the secondary filtering server performs filtering under the filtering condition set by the user. This flow is executed when the system is started up. In step 1, user information including a user name and user ID information is expanded on a database and a memory. In step 2, the profile information set by the user is developed in the automaton, and the reception of the text data to which the classification code and the brand code are added from the primary filtering server is waited. If there is no text data, the process proceeds to step 10 and it is determined whether or not the process ends. If not completed, the process returns to step 3 and waits for reception of text data. If text data is sent from the primary filtering server, step 4
Filter by automaton with. If there is a profile that has been hit by filtering, check the stock code for noise only for that profile. For example, if a profile contains 1998 as a stock code, even if the year code of 1998 is included in the text of the text data, the stock code 19
It is falsely detected as 98. Therefore, it is confirmed whether or not the brand code in the hit profile is described in the brand code item in the text data. If 1998 is not a brand code item but is described in the text, it is determined to be noise. In step 6, if there are 10 hit profiles, for example, if there are 10 hit profiles, it is judged whether or not the processing for the 10 hits has been performed, and if completed, the reception of the next news is waited. If not completed, the process proceeds to step 7 and it is determined whether or not the other code conditions are met. That is, it is determined whether the classification code specified by the user is applicable, for example, the specified field or the specified language (Japanese or English). Since the classification code information cannot be judged by the filtering of the automaton, the judgment of the classification code needs to be examined separately from the filtering. In step 8, the user
Whether or not there is an access right to the hit information source, that is, whether or not the user is buying the information source is determined by referring to the information in the user table. The processing in step 7 and step 8 can be performed in real time because only the hit profile needs to be performed. If it is determined in step 8 that there is an access right, in step 9 the filtering result is recorded in the filtering result recording unit, and the user is notified of the hit through the filtering unit of the web server. If NO in steps 7 and 8, this text data is not distributed to the user who registered the profile.

[Search Screen] Next, the case of setting search conditions will be described with reference to FIG. After starting the client terminal and establishing communication with the web server, search conditions are set using the screen of FIG. In the search word column arranged in the window 51 of the search screen shown in FIG. 7, a character (free keyword) to be searched and a logical operator are input. Enter the brand code numerically in the brand code field. By inputting a plurality of stock codes separated by a predetermined symbol, the AND condition of the search word and the stock code can be specified. In the storage condition field, enter a name for saving the created search condition. If you enter a name in this field and click the save button, the name entered in the pull-down menu of saved search conditions will be registered. In this embodiment, a total of 10 search conditions can be registered. The registered search conditions are saved in the search server. In the period specification field, enter the period to search. In the display option column, the display order and the number of displays are set. In the Source column, click the information source to be searched and specify. The number of information sources displayed here is determined by the contract with the user.

When the search conditions are set using the screen of FIG. 7, for example, when characters such as Japan and the prime minister are entered and the search button directly below is clicked, the search is executed, and the search is executed between the database and Japan. The text data including the characters is searched, and the list of the headings is displayed in the window 53. Here, when the user clicks on a headline, the text corresponding to the headline is displayed in the window 54. As a result, it is possible to retrieve the necessary text data from the past text data. Incidentally, even if it is past, the registration of the text data in the database in this embodiment is processed in real time.
New text data input in real time is also searched. Further, the search condition can be reused by pressing the save button and registered, or the registered search condition can be called and changed.

[Filtering Screen] Next, the case of setting filtering conditions will be described with reference to FIG. When the filtering button in the uppermost window 51 on the screen of FIG. 7 is clicked, the screen display is changed to that of FIG. The functions of the window 52, the window 53 and the window 54 are basically the same as those shown in FIG.
The storage profile column in the window 54 is a column for registering the set filtering conditions. If you enter a name here and click the save button, the filtering conditions set on this screen will be registered in the secondary filtering server. The name of the registered profile is displayed in the notification area at the bottom of the uppermost window 52. In this embodiment, up to 9 profiles can be registered in total.

Filtering conditions are set using the screen of FIG. For example, enter tie-up in the search term field, enter the stock code in the same way as in the search screen, specify the information source in the source field, enter the profile name in the save profile field, and click the save button. To do. Then, the profile name is displayed after the numerical value "1" in the notification area at the bottom of the window 52, and the text data received by the system of the present embodiment is filtered in real time. If the text data received by the system meets the registered filtering conditions,
A mark is added to the registered profile name in the notification area of the window 52 so that it flashes to notify the user that news corresponding to the profile name has hit.
When the user clicks the blinking profile name, the headline of the hit text data is displayed in the window 53. When the displayed headline is clicked, the body of the text data corresponding to the headline is displayed in the window 54. By thus registering the filtering conditions in advance, it is possible to capture and read the news that occurs momentarily in real time. The registered filtering condition can be called and changed. Further, by clicking the confirmation search button before registering the set filtering condition, it is possible to perform a search and verify with this condition.

In the screens shown in FIG. 7 and FIG. 8, the case where only the headline is displayed when hit is explained, but the body of the latest text data is automatically displayed in the window 54 when hit. You may In addition, in FIG. 7 and FIG. 8, the classification code setting field is omitted for simplification of the drawing, but in the same manner as the stock code,
The classification code can be set.

[Effects of the Embodiment] In the above-described embodiment, by using the issue code, it is possible to specify a large number of issue codes and perform high-speed filtering. In the method using the conventional automaton, when creating a filtering condition for one issue, in addition to the official name, multiple variant names, for example, abbreviations, uppercase letters, lowercase letters, katakana, hiragana, etc. are considered,
The filtering conditional expression becomes long. Therefore, each of the plurality of users may have 100 to 2 out of about 3000 stocks.
If a brand close to 00 is specified and filtering conditions are registered, the conventional method using an automaton takes a long processing time and cannot process text data in real time. On the other hand, in the present embodiment, since the branding code is added to the text data in advance, the subsequent filtering process and the like can be sped up, so that the filtering process as a whole can be sped up and real-time processing can be performed. .

Further, in the conventional method using the automaton, it is necessary to recreate a large automaton each time the user adds, deletes, or changes a brand code, which is also an obstacle to real-time processing. On the contrary,
In the present embodiment, even if each user changes the brand, it is not necessary to rearrange the automaton on the side that gives the brand code. By expanding the profile set by the user, which may be recombined frequently, in the secondary automaton having a small size, it becomes possible to respond in real time to a change in the brand by the user.

In the above-described embodiment, since about 3000 brand codes are assigned on the primary side, even if the user changes the brand code, only a small size automaton on the secondary side needs to be recombined. Real-time processing is now possible. In the conventional system, the noise must be removed for each user. Therefore, in the conventional system, each user sets a process for removing noise, and the system automaton is forced to perform duplicate noise removal. For this reason, the size of the automaton becomes large, making real-time processing difficult. On the other hand, in the present embodiment, noise is removed after the primary filtering, but this noise removal processing can be shared by all users. As a result, noise can be reliably removed and the processing speed can be improved.

According to this embodiment described above, in addition to full-text search by keywords, search by brand code (up to 225 brands / profile) or classification code can be performed. In addition, if the user sets the automatic monitoring conditions by keywords in advance, the news received by the system will be monitored in real time, and when the corresponding news occurs,
It automatically notifies the user, who can see its contents.

[Other Embodiments] The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the invention. For example, in the above embodiment, the case where the search screen and the filtering screen are switched and displayed has been described, but both screens may be displayed on one screen. Further, in the above embodiment, the case where a separate server is used for each function has been described, but the processing may be performed by one server. Further, in the above embodiment, the case where the text data is related to financial securities information has been described, but the text data may be data related to sports and entertainment or company internal information.

[0043]

As described above, according to the present invention, Japanese text data input in real time is scanned in real time by filtering using an automaton, and a stock code is automatically added. It is possible to provide a text filtering system and a text filtering method capable of searching and filtering Japanese text data according to a stock code.

[Brief description of drawings]

FIG. 1 is a block diagram of a text filtering system according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining a classification conversion table.

FIG. 3 shows a stock dictionary.

FIG. 4 is a diagram for explaining a noise removal dictionary.

FIG. 5 is a flowchart when assigning a stock code in the primary filtering server.

FIG. 6 is a flowchart when filtering is performed in the secondary filtering server under filtering conditions set by the user.

FIG. 7 is a diagram showing a search screen.

FIG. 8 is a diagram showing a filtering screen.

[Explanation of symbols]

10 electronic resources 12 Data receiving server 14 News queue file storage 16 Primary filtering server 18 Recording section 20 Search server 22 Database 24 Secondary filtering server 26 Record Department 28 Web servers

Front page continued (72) Inventor Kunio Sato 15-15 Minamidairacho, Shibuya-ku, Tokyo Minamidaira Imai Building, 5th floor, Queues Creative Co., Ltd. L. & SCHWART Z, L. Written by Yoshiyuki Kondo, Perl Programming, Japan, SoftBank Corp., February 28, 1993, p. 45-48 (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06F 17/30 JISST file (JOIS)

Claims (6)

(57) [Claims] 1. A dictionary in which a predetermined character string and an identification code corresponding to the character string are described is expanded in an automaton, and filtering is performed on input text data, and the corresponding character string is the text. If in the data, an identification code assigning means for assigning a corresponding identification code to the text data, and for the text data to which the identification code is assigned,
And a noise removing unit that searches for characters before and after the character string and, when a predetermined character is attached, fixes the identification code by deleting the assigned identification code. Text filtering system to do. 2. A filtering condition created and registered in advance for each user by using a free keyword, the identification code and a logical operator is expanded into an automaton, and the text data provided with the identification code is scanned, The text filtering system according to claim 1, further comprising filtering means for outputting a result of filtering. 3. The identification code given to the text data output in claim 2, when the identification code is not described at a predetermined position in the text data, the identification code is regarded as noise, A text filtering system comprising a noise removing means for removing. 4. The text filtering system according to claim 1, further comprising a full-text search engine having a function of registering and managing the text data provided with the identification code in a database in real time. . 5. The text filtering system according to claim 1, wherein the identification code is a stock code. 6. For input text data, at least one operator among forward match, backward match, exact match and intermediate match operators is applied to at least one of a title and a body of the text data. A classification code assigning means for assigning at least one classification code of a code indicating an information source, a code indicating an information category, and a code indicating Japanese or English to the text data by performing a search using The text filtering system according to claim 1, 2, 3, 4, or 5.