JP5068350B2 - Data processing apparatus, method and program - Google Patents

Description

  The present invention relates to time-series data extraction processing.

  Conventionally, database technology that handles large-scale data has been utilized in various fields (see, for example, Patent Document 1), and one of such fields is miniblogging such as Twitter (see Non-Patent Document 1).

  In the typical example, when each user of a miniblog posts as a contributor by inputting a message (called “tweet”, “tweet”, etc.) up to a predetermined number of characters from a client system such as a PC to the server, it is posted. Each message is accumulated as time-series data in time order. When each user registers (follows) other contributors that he / she wants to read a message as a target person in the selection list like “Favorites”, it is accumulated at the time of subsequent server access or update From the series data, messages posted by the subject are extracted and displayed in a time series (timeline) form.

  More generally speaking, in the time-series data to be extracted as described above, a record (for example, a message) can be added at an arbitrary timing as time passes (addition is not essential). The message includes the time of occurrence and the contributor (attribute value is, for example, “Class A” or “B”) as an attribute. As an extraction condition, there is a selection list of contributors for each user who selects and holds several values that can be taken by attributes of the message (for example, the contributor is instep and heel). As a sort criterion for extraction results, all messages corresponding to the selection list are sorted in the latest occurrence time order and displayed as a new arrival order list up to a predetermined maximum number.

In the mini blog service as described above, for example, a user who has 100,000 contributors, who has 1000 messages per contributor, that is, tweets, and who has registered 100 favorite display targets, That is, it is assumed that tweets of the display target person are displayed as a list in the order of arrival of arrival times. In this case, tweets for 100 people extracted from 100 million tweets

1000 tweets x 100 people = 100,000 cases

Consider a case in which a predetermined maximum number of 500 new arrivals is displayed.

  As one method for efficiently performing such processing, there is an approach for creating a search dictionary called an index in a database (DB). This index can be used to speed up the processing particularly in two applications of “selection” and “sort”. In many cases, B-Tree (balance tree) is used to construct an index. B-Tree guarantees that all leaves can be reached with the number of times around log (n) order to reach the leaf from the root of the tree structure, where n is the number of leaves (records) held. To do. This allows the index to be used to speed up the process of “selecting” the leaf. Also, because of the tree structure, leaves can be acquired in the “sorted” order by using the branch search algorithm.

  An index is generated based on a specific attribute for a DB record. This criterion is based on the attributes included in the record, but may be a single attribute or a plurality of attributes. For example, when an index is created based on the order of a certain attribute 1 and attribute 2, records with the same attribute 1 belong to a subtree under the same node, and a leaf is assigned according to the value of attribute 2. By performing a branch search on this index, records sorted in the order of attribute 1 and attribute 2 can be acquired. For example, if playing cards have attribute 1 as a suit (heart, diamond, etc.) and attribute 2 as a number, the ace of spades, 2 of spades, 3 of spades ... the ace of clubs, 2 of clubs, 3 of clubs ... Records can be obtained efficiently.

Twitter, Inc., “Twitter”, [online], [Search June 9, 2010], Internet <URL: http://twitter.com/>

Japanese Patent Laid-Open No. 3-108064

  However, in the conventional technology as described above, the data extracted under a first condition (for example, a contributor) from a large amount of data is another sort criterion that is not related to the first condition (for example, in time order, that is, posting). In the general case where it is desired to sort and display up to a predetermined maximum number, the processing load may increase. In particular, when a record occurs at an arbitrary timing, there is a problem that the load is excessive because it is not possible to judge and limit the time range to be searched retroactively in order to satisfy the condition of displaying up to the predetermined maximum number if applicable. was there.

  For example, in the above-mentioned miniblog service, when displaying the list of messages of favorite contributors in the order of the latest posting time, the above index is set with attributes in the order of (1) “posting time” and (2) “poster”. If it is created, it seems that it can be sorted in order of posting time and can efficiently acquire tweets corresponding to the favorite contributor, but that is not the case. This is because, since 100,000 users have posted 1000 tweets each, 100 million tweets are searched in order from the top.

  So, for example, if there is a user who likes only the contributor who has stopped tweeting about the middle of the period since the service, in order to display the new arrival of that user's favorite, from the present to the past The index has to be traced about 50 million items, which is an excessive load.

  On the other hand, considering the case of creating an index in the attribute order of (1) “Contributor” and (2) “Posting time”, as described above, 1000 tweets for 100 favorites, and 100,000 tweets in total. Will get. However, the first attribute of the index is “Contributor”. If you are a single contributor, you can get tweets from the index in order based on the “post time” of the second attribute, but if there are multiple favorites, you can re-sort the tweets you got in time order. Without it, you can't put them out in order of arrival. In other words, it is necessary to sort 100,000 items. This is also a very expensive process.

  As another approach, it is also conceivable to hold a maximum number N of new arrival order lists corresponding to the selection list for each selection list designating a display target person for each user. However, this means that the latest tweets of favorite contributors are held for each user, for example, 500, and there is a problem that the data is redundant and the required storage capacity is a burden.

  In view of the above problems, an object of the present invention is to reduce the load without redundant data retention in the process of obtaining a new arrival list from a time series data group to a predetermined maximum number by specifying a selection list as a condition. .

  Based on the above object, the data processing apparatus according to one aspect (1) of the present invention includes a data storage unit that stores a record with an attribute as time series data together with an occurrence time, and one or two of the predetermined attribute in the record. Based on the selection list storage means storing the selection list for selecting the above values and the specified selection list, the records up to a predetermined maximum number are extracted from the time series data and output as a new arrival order list. Extracting means, and time processing means for storing the generation time of the oldest record output as the new arrival order list as the oldest time and storing it in a predetermined oldest time storage means in association with each selection list; When the oldest time corresponding to the designated selection list is stored in the oldest time storage means, the extracting means And performing the extraction targeting new records after the earliest time.

  A data processing method according to another aspect (3) of the present invention is a data storage means in which the above aspect is captured by a method category, and a computer stores a record with attributes as time series data together with an occurrence time; A selection list storage means storing a selection list for selecting one or more values for a predetermined attribute in the record, and the computer is configured to execute the time based on the specified selection list. An extraction step of extracting a record up to a predetermined maximum number from the series data and outputting it as a new arrival order list, and the computer selecting the generation time of the oldest record output as the new arrival order list as the oldest time And a time processing step for storing the information in a predetermined oldest time storage means in association with each list. In the step, when the oldest time corresponding to the specified selection list is stored in the oldest time storage means, the extraction is performed on a new record after the oldest time. To do.

  Another aspect (4) of the present invention is a data processing program that captures the above aspect in the category of a computer program, and performs data processing by controlling the computer. And a selection list storage unit that stores a selection list that selects one or more values for a predetermined attribute in the record, Based on the specified selection list, the computer extracts up to a predetermined maximum number of records from the time series data and outputs the records as a new arrival order list, and causes the computer to output the oldest records output as the new arrival order list. As the earliest time of occurrence, a predetermined oldest time record is associated with each selection list. And when the oldest time corresponding to the designated selection list is stored in the oldest time storage means, the extraction is performed for new records after the oldest time. It is made to perform.

  In this way, by storing the occurrence time corresponding to the maximum number of new arrival order lists based on the selection list as the oldest time for each selection list, the next time the selection list is obtained for the selection list next time, The processing target can be limited by the old time. For this reason, it is possible to reduce the load without redundant data retention in the process of obtaining a new arrival order list from a time-series data group up to a predetermined maximum number by specifying a selection list as a condition.

  According to another aspect (2) of the present invention, in any one of the above aspects, the time processing means is based on the oldest time stored in the oldest time storage means in association with the selection list. When a record is extracted by the extraction means and newly output as a new arrival order list, it corresponds to the selection list stored in the oldest time storage means according to the generation time of the oldest newly output record The oldest time to be updated is updated.

  In this way, when a new arrival list is output using the stored oldest time, the oldest time is updated when the number of records such as messages increases by updating the oldest time. The waste of processing extra records on the basis can be avoided.

  It should be noted that a category (method for the apparatus, program for the method, etc.) different from each of the above-described modes and more specific modes described below are also included in the present invention. For different categories, “means” shall be appropriately read as “process” or “step”.

  According to the present invention, it is possible to reduce a load without redundant data holding in a process of obtaining a list of new arrivals up to a predetermined maximum number from a time-series data group by specifying a selection list as a condition.

The functional block diagram which shows the structure of embodiment of this invention. The figure which illustrates the information (data) used by embodiment of this invention. The flowchart which shows the process sequence in embodiment of this invention. The figure which shows the example of a screen display in embodiment of this invention.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described with reference to the drawings. It should be noted that assumptions common to those already described in the background art and problems are omitted as appropriate.

〔Constitution〕
The present embodiment relates to the data processing apparatus 1 shown in FIG. 1 (hereinafter abbreviated as “this apparatus 1” or “this apparatus”). This apparatus 1 is connected to a terminal T used by a user via a communication network W. This is a web server device that accepts and displays an input for a miniblog message in response to access.

  The apparatus 1 has at least an arithmetic control unit 6 such as a CPU, a storage device 7 such as an external storage device (HDD or the like) or a main memory, and a communication network W (Internet, cellular phone network, Communication means 8 (LAN adapter etc.). A terminal T is an information processing apparatus used by a user, such as a personal computer (PC), a smartphone, or a mobile phone terminal device, and one of them is schematically illustrated in FIG.

  In the apparatus 1, a predetermined computer program (not shown) stored (installed) in advance in the storage device 7 controls the arithmetic control unit 6 so that elements (10, 20, etc.) shown in FIG. Is realized. Among these elements, the information storage means can be realized in the storage device 7 in any format such as various databases (also referred to as “DB”), variables such as files and arrays, various stacks and registers, and system setting values.

  Among such storage means, as illustrated in FIG. 2A, the data storage means 10 sets the miniblog message M as a record with attributes (contributors: former, second etc.) and the occurrence time B It is a means for storing as time series data together with (date and time in the example of the figure). The selection list storage means 15 is a means for storing, as a selection list, other contributors registered as favorite display subjects who the user wants to read messages for each user of the miniblog service.

  For example, the selection list A illustrated in FIG. 2 (2) has one or more values (for example, for a predetermined attribute (contributor) in the time-series data record for each miniblog user (（in the example). , Class A, Class B ...) are listed. Each means other than the storage means is a processing means for realizing and executing the following information processing functions and operations.

[Basic effects]
The operation procedure of the apparatus 1 configured as described above is shown in the flowchart of FIG. First, when there is a display request due to access from a user (step S1), a selection list in which the user registers a display target person is designated. Then, the extracting unit 20 determines whether or not the oldest time corresponding to the designated selection list is stored in the oldest time storage unit 35 (step S2). Does not store the oldest time (step S2: “NO”).

  In this case (step S2: “NO”), the extraction unit 20 extracts records from the time series data stored in the data storage unit 10 to a predetermined maximum number based on the designated selection list (step S2). S3) It is unknown how often the message of the display target person included in the selection list is included in the time-series data. For this reason, starting with the latest message stored in the data storage means 10 as a starting point, a sufficient number of items with a considerable margin are extracted from the past, or the number of extracted items corresponding to the display target person is confirmed. The extraction process is advanced by a predetermined amount.

  At this time, if the number or ratio of messages of the display target person is small and the predetermined maximum number is not reached, the process of checking the poster with the display target person is performed over all the messages since the start of the service. . In the selection list shown in FIG. 2 (2), the user 指定 designates A and B as display target persons (broken line ellipses in the figure), and based on this selection list, the time series shown in FIG. 2 (1). From the data, the records of the posters A and B (shown by a dashed ellipse in FIG. 2 (1)) are extracted (FIG. 2 (4)).

  In the example of FIG. 2 (4), the extracted data extracted as described above is arranged for each contributor (the former and the second), and is not in the order of occurrence time, as indicated by a dashed ellipse. Then, the records of the display target messages extracted in this way (for example, FIG. 2 (4)) are sorted in order of occurrence time up to a predetermined maximum number N (step S4, for example, FIG. 2 (5)). As illustrated in FIG. 4, the list is output as a new arrival list of miniblogs (step S4).

  Subsequently, the time processing means 30 sets the generation time B of the oldest record (that is, the maximum number N records) output as the new arrival order list as the oldest time Z, and associates each selection list with a predetermined maximum time. It is stored in the old time storage means 35 (step S6, for example, FIG. 2 (3)). In the example of FIG. 2, the selection list A in the selection list storage unit 15 and the oldest time Z in the oldest time storage unit 35 are illustrated separately, but the display target person and the oldest time are shown for each user. For example, the selection list storage unit 15 and the oldest time storage unit 35 may be integrally configured by associating Z with each other.

  Then, when the oldest time Z corresponding to the designated selection list is stored in the oldest time storage means 35 (step S2: “YES”), the extraction means 20 selects a new record after the oldest time. The extraction is performed as a target (step S5).

  As described above, the occurrence time B corresponding to the maximum number (Nth) of the new arrival list based on the selection list A is stored as the oldest time Z for each selection list A (step S6 in FIG. 3). When the new arrival list is obtained for the selection list A next time, the processing target can be limited to the corresponding oldest time Z (step S5). For this reason, it is possible to reduce the load without redundant data retention in the process of obtaining a new arrival order list from a time-series data group up to a predetermined maximum number by specifying a selection list as a condition.

[Example of SQL]
The following SQL is conceivable as an aspect for realizing the record extraction process (step S5) using the oldest time as described above. However, it is assumed that the record of each message M included in the time series data includes the fields of the contributor who has the attribute P and the occurrence time B. However, this SQL shows a conceptual example rather than an accurate description example, and actually uses a description according to the configuration of the system, database table, field, and the like.

<SQL1>
SELECT message, time B, message, content
FROM selection list A INNNER JOIN message M
ON selection list A. attribute P = message M. attribute P
WHERE selection list A. set = (set specification)
AND message M. Occurrence time B> = oldest time Z

  Although the result of the above SQL1 is not yet arranged in the order of the occurrence time B (for example, FIG. 2 (4)), if “attribute P” and “occurrence time B” are set in the message M as an index, the message M For each attribute P in the selection list, a message M whose generation time B is newer than the oldest time Z can be efficiently acquired while referring to the index.

  This is because when searching for a branch of the index B-Tree, the leaves in the attribute P can be obtained in the latest order of the occurrence time B, and when a leaf older than the oldest time Z is encountered, the search for the subtree of the attribute P is terminated. Since the search of the subtree of the next attribute P can be started, and when the subtree of the attribute P not in the selection list is encountered, the subtree of the next attribute P is searched by ignoring the subtree. It is because it can move. If the earliest time Z is appropriate, the number of messages M acquired is the maximum number N as a result of SQL1.

[Example of sorting by SQL]
When sorting in the order of occurrence time is newly performed in the DB with respect to the extraction result by the above SQL1, it is described as follows after SQL1.

<SQL2>
(SQL1) ORDER BY message M. Occurrence time B DESC

Here, in general, since the DB executes the sort in the on-memory if the number of items to be sorted is small, if the maximum number N is sufficiently small, a sufficient processing speed is obtained with SQL2. However, when the maximum number N increases, the DB server starts writing records to the disk before sorting, and the processing speed decreases.

  In order to deal with the above case, the result executed in SQL1 is held in the memory on the application program side that realizes the function of the present embodiment, for example, and sorted on-memory according to the occurrence time B on the application program side Can be considered. The maximum number N can be increased to a reasonable number for sorting on-memory.

[Update of oldest time]
Since the message M is added irregularly, the oldest time Z is preferably initialized and updated at some timing. Although the timing can be freely selected, the most preferable timing is a timing for newly acquiring a new arrival order list as shown in the flowchart of FIG. At this timing, the latest new arrival order list based on the selection list can be acquired, the maximum number N of occurrence times B can be known, and the oldest time Z can be updated.

  In other words, in this case, the time processing unit 30 extracts the record by the extraction unit 20 based on the oldest time stored in the oldest time storage unit 35 in association with the selection list (step S2: “YES”). (Step S5) When newly output as the new arrival order list (Step S4), the generation time of the oldest record newly output is stored as the oldest time, and stored in the oldest time storage means 35. The oldest time corresponding to the selected list is updated (step S6).

  In this way, when the number of records such as messages increases by updating the oldest time when a newly arrived list is output using the stored oldest time, the oldest time is used. Therefore, waste of processing extra records can be avoided.

[Other Embodiments]
In addition, said each embodiment is only an illustration, and this invention includes what is illustrated below and other embodiment other than that. For example, the application target of the present invention is not limited to mini-blogs, but can be arbitrarily determined such as electronic bulletin boards, SNS (social networking services), voice message systems for disasters, various electronic commerce and security management, etc. .

  In addition, each element such as means may be realized by other information processing mechanisms such as an electronic circuit based on a wired logic or the like without being limited to an arithmetic control unit of a computer. Further, each configuration diagram, data diagram, flowchart diagram, and the like are merely examples, and the presence / absence of each element, its order, specific contents, and the like can be changed as appropriate. For example, the apparatus of the present invention may be realized by using a plurality of apparatuses such as servers, and a configuration in which each storage unit is realized by a separate and independent server apparatus or system is also common. Depending on the function, the configuration can be flexibly changed, for example, by calling an external platform or the like with an API (application program interface) or network computing (so-called cloud or the like).

DESCRIPTION OF SYMBOLS 1 Data processing device 6 Arithmetic control part 7 Storage device 8 Communication means 10 Data storage means 15 Selection list storage means 20 Extraction means 30 Time processing means 35 The oldest time storage means A Selection list B Occurrence time M Message W Communication network P Attribute T Terminal Z oldest time

Claims (4)

Data storage means for storing records with attributes as time series data together with the time of occurrence;
Selection list storage means for storing a selection list for selecting one or more values for a predetermined attribute in the record;
Extraction means for extracting records up to a predetermined maximum number from the time-series data based on the specified selection list, and outputting as a new arrival order list;
A time processing unit for storing the generation time of the oldest record output as the new arrival order list as the oldest time, and storing it in a predetermined oldest time storage unit in association with each selection list;
Have
The extraction means, when the oldest time corresponding to the designated selection list is stored in the oldest time storage means, performs the extraction for a new record after the oldest time. Characteristic data processing device.   The time processing means, when a record is extracted by the extraction means based on the oldest time stored in the oldest time storage means in association with the selection list, and newly output as a new arrival order list, 2. The data according to claim 1, wherein the oldest time corresponding to the selection list stored in the oldest time storage means is updated with the occurrence time of the oldest record that has been newly output. Processing equipment. A data storage means in which a computer stores a record with attributes as time series data together with an occurrence time, and a selection list storage that stores a selection list for selecting one or more values for a predetermined attribute in the record Means, and
An extraction step in which the computer extracts records up to a predetermined maximum number from the time series data based on the specified selection list, and outputs it as a new arrival order list;
A time processing step in which the computer stores the occurrence time of the oldest record output as the new arrival order list as the earliest time in association with each selection list in a predetermined oldest time storage unit;
Including
In the extraction step, when the oldest time corresponding to the designated selection list is stored in the oldest time storage means, the extraction is performed on a new record after the oldest time. The data processing method characterized by performing. A data processing program for performing data processing by controlling a computer,
Data storage means for storing a record with attributes as time series data together with the time of occurrence in a computer, and a selection list storage for storing a selection list for selecting one or more values for a predetermined attribute in the record Means, and
Based on the specified selection list, the computer extracts a record up to a predetermined maximum number from the time series data, and outputs it as a new arrival list,
In the computer, the generation time of the oldest record output as the new arrival order list is stored as an oldest time in association with each selection list in a predetermined oldest time storage unit,
When the earliest time corresponding to the designated selection list is stored in the earliest time storage means, the computer is made to perform the extraction on a new record after the earliest time. A data processing program.

Images