JP2013073266A - Information processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate a content of transmission data without referring to data parts in packets when a data transmission device has divided the transmission data into units of packets and transmitted the divided pieces of transmission data.SOLUTION: An information processing device estimates a content of transmission data transmitted from a data transmission device. The information processing device comprises: first information holding means which holds first feature information including a feature amount related to the data flow when the data transmission device holding a plurality of pieces of transmission data transmits the respective pieces of transmission data; second information holding means which holds second feature information including a feature amount related to the data flow when the data transmission device transmits any piece of transmission data; and means which collates the first feature information held by the first information holding means with the second feature information held by the second information holding means, and estimates transmission data transmitted by the data transmission device by using a result of the collation.

Description

  The present invention relates to an information processing apparatus and a program, and can be applied to, for example, a system that analyzes a user's behavior tendency from a network access status.

  Conventionally, when a user uses a user terminal to access content on a WWW server on the Internet, a history related to the access is analyzed and used for marketing, user matching advertisement, or the like.

  As a conventional technique for analyzing the access history of the user (user terminal) as described above, there is a technique described in Patent Document 1.

  In the technique described in Patent Document 1, in order to present more useful information to the user, a word that is considered important for the user is statistically estimated from the access history to the content of the user (user terminal), It describes the provision of content that seems more useful to the user from the words.

Japanese Patent Laid-Open No. 10-162011

  However, the technique described in Patent Document 1 is based on the premise that the content viewed by the user can be uniquely specified by the URL or the like used by the user (user terminal) to specify the content.

  However, in an actual network environment, there is a case where information such as a URL that can uniquely specify content trained by a user terminal cannot be acquired. For example, when a packet transmitted and received by the user terminal is observed on a network path between the user terminal and the WWW server, and the URL of the content accessed by the user terminal is acquired, the data of the payload portion of the packet is read. In this case, the following problems exist.

  The first problem is that when the payload portion of the packet is encrypted, the URL of the content accessed by the user terminal cannot be obtained without the decryption being canceled. is there.

  As a second problem, when there are a large number of packets to be transmitted / received by the user terminal, it is determined which of the packets contains information on the URL of the content accessed by the user terminal. An enormous cost (processing amount, storage capacity, etc.) is required to grasp.

  In view of the above problems, when a data transmission device (for example, a WEB server) divides into packet units and transmits transmission data (for example, WEB content), without referring to the data portion of the packet, There is a demand for an information processing apparatus and program that can estimate the contents of the transmission data.

  The information processing apparatus according to the first aspect of the present invention provides (1) first feature information including a feature amount relating to a data flow when each transmission data is transmitted from a data transmission apparatus holding a plurality of transmission data. First information holding means for holding, and (2) second information for holding second feature information including a feature quantity relating to a data flow when any of the transmission data is transmitted from the data transmission device. A collating process that performs a process of collating the holding means, and (3) the second feature information held by the second information holding means and the first feature information held by the first information holding means. And (4) estimation processing means for estimating transmission data transmitted by the data transmission device using the collation result of the collation processing means.

  An information processing program according to a second aspect of the invention includes a computer that includes (1) a feature amount relating to a data flow when each transmission data is transmitted from a data transmission device that holds a plurality of transmission data. A first information holding means for holding feature information; and (2) a second feature information for holding second feature information including a feature amount relating to a data flow when any of the transmission data is transmitted from the data transmission device. And (3) a process of collating the second feature information held by the second information holding unit and the first feature information held by the first information holding unit. And (4) using the collation result of the collation processing means to function as estimation processing means for estimating the transmission data transmitted by the data transmitting apparatus.

  According to the present invention, when the data transmitting apparatus transmits transmission data divided in units of packets, the contents of the transmission data can be estimated without referring to the data portion of the packet.

It is explanatory drawing, such as connection relation of the various apparatuses (including the content estimation apparatus of embodiment) related to embodiment. It is the block diagram shown about the structural example of the content processed with the content estimation apparatus which concerns on embodiment. It is explanatory drawing shown about the example of the content of the index management table which concerns on embodiment. It is the flowchart shown about operation | movement of the information generation part which concerns on embodiment. It is the flowchart shown about operation | movement of the estimation process part which concerns on embodiment. It is explanatory drawing shown about the example of the content of the index management table which concerns on the modification of embodiment.

(A) Main Embodiment Hereinafter, an information processing apparatus and a program according to an embodiment of the present invention will be described in detail with reference to the drawings. Note that the information processing apparatus of this embodiment is a content estimation apparatus.

(A-1) Configuration of Embodiment FIG. 1 is an explanatory diagram of the connection relationship of various devices (including the content estimation device 10 of the embodiment) related to this embodiment.

  The content estimation apparatus 10 shown in FIG. 1 observes (captures) packets transmitted and received by the user terminal 30, and estimates the content (for example, the URL of the content) on the WEB server 20 accessed by the user terminal 30. Is.

  The user terminal 30 is assumed to be a terminal including a WEB browser such as a PC, a mobile phone terminal, or a PDA. As the user terminal 30, a terminal including an existing Web browser can be applied.

  Further, it is assumed that the WEB server 20 supplies any data of the contents C1 to C4 in response to an access request from the user terminal 30. Since an existing WEB server can be applied to the WEB server 20 as well, detailed description is omitted. In the WEB server 20, U 1 to U 4 are defined as URLs for the contents C 1 to C 4, and content data corresponding to the URL notified from the user terminal 30 is supplied to the user terminal 30. The configuration of the WEB server 20 and the content configuration and the number of contents to be stored are not limited, and the same ones as those of various existing WEB servers can be applied.

  The user terminal 30 accesses the WEB server system 20 via the access network N2 and the Internet N1, and receives (downloads) any of the contents C1 to C4 on the WEB server system 20. In this embodiment, the number of WEB servers 20, the number of user terminals 30, and the network configuration between the devices are not limited.

  Hereinafter, the IP address of the WEB server 20 is represented as S1, and the IP address of the user terminal 30 is represented as T1.

  Next, a configuration example of each content stored in the WEB server 20 will be described.

  FIG. 2 is a block diagram showing the configuration of the content C1 stored in the WEB server 20.

  As described above, the configuration of the content stored in the WEB server 20 is not limited. However, in this embodiment, the content C1 has a configuration as shown in FIG. 2 in order to simplify the description. The following will be described.

  Each content has element content (hereinafter also referred to as “EC”) that is one or a plurality of constituent elements. In the case of the content C1, as shown in FIG. 2, a text (for example, an HTML sentence or XML) Element content EC11 of a sentence or the like, banner advertisement (image data) element content EC12 called from the text (EC11), and style sheet element content EC13 called from the text (EC11).

  In the content estimation apparatus 10, it is assumed that identifiers are managed for each element content constituting each content. In this embodiment, it is assumed that identifiers EC11 to EC13 are assigned to the element contents EC11 to EC13, respectively. Specifically, the content estimation apparatus 10 may manage the above-described identifier in association with a URL for accessing each element content. Further, the content estimation apparatus 10 may manage the URL itself as an identifier, and do not assign a separate identifier to each element content.

  Next, an outline of the configuration of the content estimation device 10 will be described.

  The content estimation apparatus 10 includes an information generation unit 11, a content information storage unit 12, and an estimation processing unit 13.

  The communication device 120 includes, for example, an information processing program according to the embodiment in an execution configuration of a program such as a CPU, ROM, RAM, EEPROM, and hard disk, and a device (computer) having an interface for communicating with other communication devices. It is constructed by installing etc.

  The information generation unit 11 accesses each content (C1 to C4) on the WEB server 20, and flows data (packet string) observed when downloading data of element contents constituting each content (flow). Information (hereinafter referred to as “flow information”) is acquired. In this embodiment, the flow information includes statistical information for each flow (for example, (the IP address of the server facing the user terminal 30, the number of packets in the packet sequence constituting the flow, etc.), etc. Based on the acquired flow information, the information generation unit 11 generates information (hereinafter referred to as “index information”) that serves as an index (heading) for searching for content corresponding to the statistical information. In other words, the index information includes information indicating the feature amount regarding the data flow when the corresponding content is downloaded.

  Further, the information generation unit 11 reads the contents of the contents (C1 to C4) on the WEB server 20 and generates information related to the contents (hereinafter referred to as “content information”). The content information may include information on the content (keywords, words, etc.) and information such as URLs. In this embodiment, in order to simplify the description, it is assumed that the content information includes at least a URL for accessing the content.

  Then, the information generation unit 11 records the index information and the content information in the index management table 121 of the content information storage unit 12 in association with each other. In other words, the content information storage unit 12 is configured as a database capable of searching for content information using some or all items of index information as keys. In this embodiment, the content information storage unit 12 is described as managing the content information corresponding to each index information in a table format for simplicity of explanation, but a specific data management method is a table format. The present invention is not limited to this, and various database formats can be applied.

  Then, the estimation processing unit 13 acquires flow information between the user terminal 30 and the WEB server 20, and based on the acquired flow information, the index information and a part or all of the index information are processed by the same process as the index creation unit 113. Information with common items (hereinafter referred to as “search target index information”) is generated. In other words, similar to the index information, the search target index information includes information indicating the characteristic amount related to the observed data flow.

And the estimation process part 13 performs the process which collates the produced | generated search object index information and each index information of the content information storage part 12 (index management table 121). The estimation processing unit 13 performs processing for estimating the content accessed by the user terminal 30 based on the collation result. In other words, the estimation processing unit 13 performs a process of detecting index information in which the search target index information matches the flow-related feature amount within a predetermined range.

  Next, details of the configuration of the content estimation apparatus 10 will be described.

  First, the information generation unit 11 will be described.

  The information generation unit 11 includes a flow information acquisition unit 111, a crawling processing unit 112, and an index creation unit 113.

  The crawling processing unit 112 performs processing of sequentially specifying and specifying a URL related to predetermined content (for example, content that is a target for acquiring a user's behavior tendency). That is, the crawling processing unit 112 performs the same processing (processing that generates a flow (traffic) on the network) when the user terminal 30 accesses each content on the WEB server 20.

  In this embodiment, the description will be made assuming that a list of URLs to be accessed is registered in advance in the crawling processing unit 112, but the method of holding the URLs to be accessed by the crawling processing unit 112 is not limited. For example, only one URL may be set in the crawling processing unit 112, and the crawling processing unit 112 may sequentially access content linked by the content of the URL.

  When the crawling processing unit 112 accesses one content (page), the crawling processing unit 112 first downloads the element content related to the body of the content, analyzes the description of the body and analyzes other element contents (for example, the content) , A banner, a style sheet, etc.) and a process of accessing (downloading) the searched element contents one by one. When one content includes a plurality of element contents such as a document other than the main body, a format, and an image at this time, the crawling processing unit 112 has a sufficient interval for separating and acquiring them as a plurality of flows. To access each element of the content. Further, when accessing the content, the crawling processing unit 112 holds content information related to the content and supplies the content information to the index creating unit 113. In this embodiment, it is assumed that the content information held by the crawling processing unit 112 includes at least a URL.

  The flow information acquisition unit 111 observes the flow (flow) of data (packet string) that flows from the crawling processing unit 11 to the WEB server 20 in response to a content access request from the crawling processing unit 112, and based on the observation result. Flow information. Then, the flow information acquisition unit 111 passes the obtained flow information to the index creation unit 113.

  Specifically, the flow information acquisition unit 111 observes a flow when the crawling processing unit 112 downloads each element content, and acquires flow information for each flow. Therefore, the flow information acquisition unit 111 acquires one flow information for one element content. The number of information items included in the flow information and the combination of types are not limited, but here, the identifier (address) of the data transmission source of the flow, the information on the number of packets of the packets constituting the flow, Will be described as being included.

  The specific configuration in which the flow information acquisition unit 111 acquires the flow information is not limited. For example, NetFlow (see Reference Document 1 (IETF RFC3954)), IPFIX (see Reference Document 2 (IETF RFC5101)) Since the processing configuration of the exporter (agent) and the collector in the prior art such as the above can be applied, detailed description is omitted.

  Note that the flow information acquisition unit 111 need not be installed in the content estimation apparatus 10 itself. For example, the flow information acquisition unit 111 is mounted on a relay device (router or the like) installed on a network path between the crawling processing unit 112 and the WEB server 20, and the content estimation device 10 receives supply of statistical information. You may do it.

  The index creation unit 113 generates one or a plurality of index information based on the flow information supplied from the flow information acquisition unit 111, and associates the content information with the content information storage unit 12 (index management table 121). sign up.

  It is assumed that each index information generated by the index creation unit 113 includes information on items constituting the flow information, the number of flows, and the like as key information related to the search.

  FIG. 3 shows an example of the contents of the index management table to which the index information and content information generated by the index creation unit 113 of this embodiment are input.

  In FIG. 3, among the contents registered in the index management table 121, the contents registered based on the flow information acquired when the crawling processing unit 112 accesses the contents C1 are shown.

  As shown in FIG. 3, in the index management table 121, one or a plurality of index information generated based on one flow information is registered in association with content information. Specifically, in the index management table 121 shown in FIG. 3, information including one index information, an ID as an identifier of the index information, and content information corresponding to the index information (hereinafter referred to as “index management information”). ") Is represented by one line. That is, the index creation unit 113 of this embodiment performs processing for generating index management information based on the index information and registering it in the index management table 121 for each generated index information.

  FIG. 3 shows an example in which seven index management information items are registered in seven rows, and reference numerals K10 to K16 are assigned to the index management information items having IDs R0 to R6, respectively. For example, the index management information K10 whose ID is R0.

  Next, details of the index management information constituting the index management table 121 in the index management table 121 will be described.

  As shown in FIG. 3, each index information constituting the index management table 121 includes information on items of “address”, “number of flows”, “number of packets”, “derived type”, and “number of original flows”. It is included.

  “Address” indicates an identifier (address) of the data supply source of the flow related to the index information. Here, since all the index information relates to the data flow with the WEB server 20 as the supply source, the address “S1” for accessing the WEB server 20 is input to all the address items. .

  The item “number of flows” indicates the number of flows related to the index information.

  The item “number of packets” indicates the total number of packets for each flow.

  In the following, the number of packets when the element content EC11 is downloaded as a single flow is P11, the number of packets when the element content EC12 is downloaded as a single flow is P12, and the element content EC13 is downloaded as a single flow. Let P13 be the number of packets.

  The item “derivation type” indicates whether or not the index information is index information that directly reflects the acquired flow information.

  As described above, the crawling processing unit 112 accesses each element content constituting each content independently. That is, in the crawling processing unit 112, access control is performed so that each element content is observed as a separate flow. However, when the user terminal 30 actually accesses the content, one or all elements are accessed. It is also possible to access (download) content at the same time. When the user terminal 30 accesses (downloads) a plurality of element contents at the same time, these flows are combined and observed as one flow. This is because when the user terminal 30 actually accesses the content, the flow information acquisition unit 131 may observe the flow as a single flow as a result of continuous access to a plurality of element contents.

  Furthermore, when there is an element content (for example, a style sheet) that is commonly used for a plurality of contents, the element content may be cached on the user terminal 30 side. When there is element content cached on the user terminal 30 side, even if the user terminal 30 accesses content including the element content, the cached element content is not downloaded. That is, the observed flow information may differ depending on the presence / absence of element content cached on the user terminal 30 side.

  Therefore, in the index creation unit 113 of this embodiment, as an example, for content including a style sheet in element content, index information when the download of the element content is omitted is also generated. In the index creation unit 113, whether or not each element content is a style sheet is confirmed by, for example, an extension or from which component part of the original content is called (for example, from a style sheet designation part) It can be easily determined by confirming that the link has been made.

  Therefore, in the information generation unit 11 of this embodiment, the index information when all element contents are downloaded in a single flow is used as reference index information (hereinafter referred to as “reference index information”) and the reference index information. In order to manage the derived index information (hereinafter referred to as “derived index information”), the above-described “derived type” item is provided in the index management table 121.

  In FIG. 3, “reference data” is input in the derived type field for the reference index information, and information indicating a combination of element contents constituting each flow is input in the derived type item for the derived index information. ing.

  For example, in FIG. 3, the index information constituting the index management information K10 with the ID “R0” is managed as the reference index information. In the index information (reference index information) of the index management information K10 shown in FIG. 3, the number of packets “P11”, “P12”, “P13” relating to the three flows corresponding to the element contents EC11 to EC13 is included in the packet number item. Is entered.

  On the other hand, in FIG. 3, the index information constituting the index management information K11 with the ID “R1” is managed as a derived index. Specifically, the index information of the index management information K11 is registered as derived index information when the element contents EC11 and EC12 are downloaded in one flow and the element content EC13 is downloaded in a single flow. In the index information (reference index information) of the index management information K11 shown in FIG. 3, information “P11 + P12” and “P3” on the number of packets related to two flows are input in the item of the number of packets. Among the items of the number of packets, “P11 + P12” indicates the number of packets constituting the flow when the element contents EC11 and EC12 are downloaded in one flow. Also, “P3” in the item of the number of packets indicates the number of packets constituting the flow when the element content EC13 is downloaded as a single flow. Then, in the derived type item, two pieces of information “EC11 + EC12” and “EC13” are input as information relating to the combination of element contents constituting each of the two flows described above. Of the derived type items, “EC11 + EC12” indicates that the flow is composed of data of two element contents EC11 and EC12. In the derivation type item, “EC13” indicates that the flow includes data of one element content EC13.

  In addition, regarding the correspondence relationship between the index management information related to the reference index information and the index management information related to the derived index information, management may be performed in which separate items are provided and linked to each other using a pointer or the like. In FIG. 3, the above-mentioned link is illustrated by arrows between the index management information. In FIG. 3, the index management information K10 is the center, and a link is established between the index management information K11 to K16. This facilitates maintaining consistency when updating the index management table 121 (index management information).

  The item “original flow” indicates the number of flows of reference index information related to the index information. In other words, the item of the original flow number is a value that directly represents the number of flows when the flow of the corresponding content is observed by the flow information acquisition unit 111. Accordingly, for the reference index information, the item “number of flows” and the item “original number of flows” have the same value.

  Next, details of the estimation processing unit 13 will be described.

  The estimation processing unit 13 includes a flow information acquisition unit 131, a content information search unit 132, a reliability calculation unit 133, and an output unit 134.

  The flow information acquisition unit 131 observes the flow of data (packet string) flowing between the WEB server 20 and the user terminal 30, acquires flow information, and supplies the flow information to the content information search unit 132. When an access request for any content is made from the user terminal 30 to the WEB server 20, data of the content is supplied from the WEB server 20 to the user terminal 30 to the user terminal 30. Observe the flow of data and obtain flow information. Note that the flow information acquisition unit 131 preferably acquires flow information by the same processing as that of the flow information acquisition unit 111 described above. Further, in the flow information acquisition unit 131, for example, the protocol for which the flow information is acquired may be narrowed down to a predetermined protocol. For example, when content data is supplied from the WEB server 20 to the user terminal 30 only by the HTTP protocol, the flow information acquisition unit 131 may acquire the flow information by focusing on the HTTP protocol. In addition, the flow information acquisition unit 131 may narrow down and observe a packet to be observed in an address unit (for example, a transmission source address and / or a transmission destination address of a packet).

  Based on the flow information obtained by the flow information acquisition unit 131, the content information search unit 132 retrieves information on items similar to the index information (except for items of “derived type” and “original flow number”). Generated as “search target index information”. That is, the search target index information of this embodiment includes information on items of “address”, “flow information”, and “number of packets”.

  Then, the content information search unit 132 performs a process of comparing the generated search target index information with each index information of the index management table 121 of the content information storage unit 12, and index information recognized as matching within a predetermined range. Content information corresponding to is detected.

  At this time, the content information search unit 132 detects only the index information in the index management table 121 that is found to match the search target index information, the address, the number of flows, and the number of packets of each flow. Even if the number of packets in each flow does not completely match, in anticipation of the possibility of fluctuation in dynamic content, an error within a predetermined tolerance is detected as matching. And In this embodiment, as an example, the content information search unit 132 determines to match if the number of packets of the search target index information is within ± 3% of the number of packets in the index management table 121. You may make it do. It should be noted that the percentage of the error range allowed for the number of packets by the content information search unit 132 is not limited to the above example.

  As described above, when the content information search unit 132 detects, from the index management table 121, index information that is found to match the search target index information, the content information search unit 132 converts the index information and the content information corresponding to the index information to reliability. It supplies to the calculation part 133.

  The reliability calculation unit 133 calculates a value indicating the reliability that the content related to the search target index information is content information corresponding to the detected index information. Since the content information search unit 132 may detect a plurality of pieces of content information (index information), the reliability calculation unit 133 detects the content information search unit 132 in order to cope with such a case. A process of calculating the reliability according to the result is performed. For example, the IP addresses are the same if they are on the same WEB server, and if the documents have the same size and the same number of flows per content, the number of packets may be the same.

  In this embodiment, as an example, the reliability calculation unit 133 determines the difference between the number of detected content information (index information) and the number of detected index information flows (the “original flow number” of the content information). The value weighted according to the difference between the value and the “number of flows”) is handled as the reliability.

  When a large number of corresponding content information (index information) is detected by the content information search unit 132, information other than the content actually accessed by the user terminal 30 compared to the case where only one corresponding content information is detected. In such a case, it is estimated that the reliability is low. Further, the closer the number of flows acquired by the flow information acquisition unit 111 and the number of flows acquired by the flow information acquisition unit 131 are, the more likely that the user terminal 30 has accessed the same content (the same URL). It is considered that the nature is high. In view of this, the content information search unit 132 calculates a value that takes the above factors into consideration as a value indicating reliability, and performs subsequent processing using the detected content information (index information) (for example, data mining). Processing).

  In the following, the number of content information (index information) detected by the content information search unit 132 is “df”, and the “original flow number” related to the content information (index information) whose reliability is to be calculated is “f0”. When the “number of flows” (number of flows of search target index information) related to the content information (index information) for which reliability is calculated is “f1”, for example, the value A indicating the reliability is (1 ) Expression. The following equation (1) indicates that the higher the reliability A, the higher the reliability related to the content information (index information).

Reliability A = f1 / (f0 × √df) (1)
In the above equation (1), the reliability A tends to be smaller as df is larger.

  Further, in the above equation (1), the reliability A tends to be smaller as f1 is smaller. Since f0 indicates the number of flows when all element contents are downloaded in a separate flow for the content, f0 ≧ f1 is basically satisfied. In the case of f0> f1, since the flows related to at least one element content are combined into one flow, the value of f1 decreases as the combined flows increase. become. Therefore, in the above equation (1), in consideration of the fact that the reliability associated with the index information decreases as the number of combined flows increases, the reliability A tends to decrease as f1 decreases. Yes.

  Note that the above formula (1) is an example for calculating the reliability value, and if it corresponds to the above-mentioned tendency, the specific calculation formula is not limited and other calculation formulas should be used. May be.

  The output unit 134 outputs the obtained content information (URL information) and reliability values. The format output by the output unit 134 is not limited, but simply outputs the data of the obtained content information (URL information) and a list of reliability information (for example, spreadsheet, text data, etc.). May be. The destination to which the output unit 134 outputs data is not limited. For example, the output unit 134 may be a recording medium such as a hard disk or a DVD-R, or may be output to another device by communication. Specifically, the output unit 134 outputs, for example, to an information processing device (for example, a server device that performs data mining) that performs processing for estimating user preferences from frequently accessed URL information. You may make it do.

(A-2) Operation | movement of embodiment Next, operation | movement of the content estimation apparatus 10 which has the above structures is demonstrated.

  Here, first, the operation of the information generation unit 11 will be described with reference to the flowchart of FIG. Specifically, below, the information generation unit 11 creates index information based on the flow information captured by accessing the content C1 (FIG. 2 described above) of the WEB server 20, and the content information storage unit 12 (index management) The operation until registration in the table 121) will be described.

  Further, as described above, it is assumed that the URL of the content C1 shown in FIG. 2 is U1, and is stored in the WEB server 20 of the address S1.

  First, the crawling processing unit 112 sequentially accesses the element contents EC11 to EC13 constituting the content C1, and at that time, the data (traffic) sent from the WEB server 20 to the crawling processing unit 112 by the flow information acquisition unit 111 is obtained. Based flow information is acquired. Then, the flow information acquisition unit 111 supplies the flow information of the element contents C11 to C13 constituting the content C1 to the index creation unit 113. Further, the crawling processing unit 112 supplies information including a URL as the content information of the content C1 to the index creating unit 113 (S101).

  Specifically, the crawling processing unit 112 first performs a process of specifying the set URL (U1) and accessing the WEB server 20, and reads the element content EC11 of the body. The data (traffic) is read by the crawling processing unit 112 via the flow information acquisition unit 111. At this time, the flow information acquisition unit 111 acquires flow information including information on the address S1 and the number of packets P1.

  Then, the crawling processing unit 112 specifies the URLs of the other element contents EC12 and EC13 constituting the content C1 based on the description of the acquired body (element content EC11) so that those flows are not combined into one. Access at a sufficient interval. For example, the crawling processing unit 112 transmits an access request for one element content to the WEB server 20, and no data (packet) arrives for a predetermined time or longer after the data (packet) related to the access request starts to arrive. In this case, it may be assumed that the download related to the element content has been completed and the download of the next element content is started.

  By repeating the above processing, the flow information acquisition unit 111 obtains the packet numbers P1, P2, and P3 for the element contents EC11, EC12, and EC13, respectively. Then, the flow information acquisition unit 111 supplies flow information corresponding to each element content to the index creation unit 113. For example, the flow information corresponding to the element content EC11 includes information on the address S1 and the number of packets P1.

  Next, the index creation unit 113 generates index information based on the flow information of the element contents C11 to C13 supplied from the flow information acquisition unit 111 (S102), and updates the index management table 121 (index) (Additional registration of management information) is performed (S103).

  The index management information additionally registered by the index creation unit 113 in step S103 described above is the seven index management information K10 to K16 shown in FIG.

  Specifically, the index creating unit 113 first generates reference index information related to the content C1 based on the flow information of the element contents EC11, EC12, and EC13. In FIG. 3, the index information of the index management information K10 corresponds to the reference index. The index creation unit 113 creates a key constituting the reference index information from the flow information (address S1 and the number of packets P11, P12, P13) obtained by the flow information acquisition unit 111 for the element contents EC11, EC12, and EC13. Then, the index creation unit 113 acquires the content information (URL: U1) of the content C1, and registers the index management information R10 shown in FIG.

  Then, the index creation unit 113 creates derived index information assuming that two flows among the three flows indicated in the reference index information overlap and are observed as a single flow, and the created index information Is registered in the index management table 121 (index management information R11 to R14 shown in FIG. 3).

  Then, the index creation unit 113 creates derived index information assuming that all three flows indicated in the reference index information are overlapped and observed as one flow, and an index based on the created index information Management information (index management information R15 shown in FIG. 3) is registered in the index management table 121.

  As described above, when there are a plurality of flows indicated in the reference index information, the index creation unit 113 obtains all combinations to be combined of those flows, creates the derived index information for each combination, and creates the created derived index. Index management information based on the information is registered in the index management table 121.

  In addition, the index creation unit 113 creates the derived index information assuming that the flow of the element content EC13 does not occur in order to recognize that the element content EC13 is a style sheet as described above, and the created index Based on the information, the index management information R15 and R16 shown in FIG. 3 are registered. The index management information R16 is based on derived index information that assumes a case where the flow of the element content EC11 and the flow of the element content EC12 are combined.

  As described above, the index creating unit 113 creates the derived index information that is omitted for the flow related to the element content that may be downloaded as described above, among the flows indicated in the reference index information. In addition, when there are a plurality of element contents that may be omitted, the index creating unit 113 creates derived index information for all the omitted combinations. Further, the index creating unit 113 creates derived index information when some or all flows are combined with respect to the derived index information from which some flows are omitted.

  Next, operation | movement of the estimation process part 13 is demonstrated using the flowchart of FIG. Specifically, in the following, when the user terminal 30 accesses the content C1 (FIG. 2 described above) of the WEB server 20, the estimation processing unit 13 performs an index based on the flow information based on the result of observing the traffic. The operation of creating information and accessing the user terminal 30 to specify content will be described.

  First, the user terminal 30 accesses the content C 1 of the WEB server 20, and data (packets) from the WEB server 20 to the user terminal 30 reaches the user terminal 30 via the flow information acquisition unit 131. At this time, the user terminal 30 acquires the element contents EC11 to EC13 constituting the content C1 (S201). Thereby, the flow information acquisition unit 131 acquires the flow information related to the element contents EC11 to EC13 sent to the user terminal 30.

  Here, the flow information acquired by the flow information acquisition unit 131 is often the same as that obtained by the flow information acquisition unit 111 during crawling by the crawling processing unit 112 (that is, the same content as the reference index information). it is conceivable that. Here, the flow information acquisition unit 131 uses the flow with the address “S1” and the number of packets “Px1”, the flow with the address “S1” and the number of packets “Px2”, and the number of packets with the address “S1”. It is assumed that flow information related to three flows called “Px3” is acquired. It is assumed that flow information (information including an address and the number of packets) related to each flow is supplied from the flow information acquisition unit 131 to the content information search unit 132.

  Next, the content information search unit 132 generates search target index information based on the flow information for each flow supplied from the flow information acquisition unit 131 (S202).

  Specifically, the content information search unit 132 generates search target index information with the address “S1”, the number of flows “3”, and the number of packets for each flow “Px1”, “Px2”, and “Px3”, respectively. Is done.

  Next, the content information search unit 132 collates the generated search target index information with each index information in the index management table 121, and detects index information (content information) that is recognized to match (S203).

  Specifically, the content information search unit 132 first detects index information in which “address” and “number of flows” of the search target index information match. And the content information search part 132 performs the process which collates the packet number for every flow of the detected index information, and the packet number of search object index information. When the number of flows is plural, the content information search unit 132 needs to determine a combination for checking the number of packets of the search target index information and the number of packets of the detected index information. In the content information search unit 132, the method for determining the combination for the above-described collation is not limited. For example, the number of packets having the closest values may be combined and compared, or the difference of the most difference may be determined. A combination that reduces the total may be obtained and compared.

  Here, as an example, it is assumed that the content information search unit 132 detects the index information of the index management information K10 shown in FIG. 3 as the search target index information having the same address and the number of flows. Then, it is assumed that the content information search unit 132 determines to compare the number of packets for three combinations of “Px1 and P11”, “Px2 and P12”, and “Px3 and P13”. Then, the content information search unit 132 compares the number of packets for each combination and determines whether or not the difference is within a predetermined range. For example, the content information search unit 132 matches the combination if the number of packets of the search target index information is within ± 3% (the allowable range may be a constant) with respect to the number of packets of the index information. You may make it admit that it does. Then, when it is recognized that the number of packets matches for all combinations (flows), the content information search unit 132 detects that the index information matches the search target index information. Note that the content information search unit 132 does not recognize that the number of packets matches for all combinations (flows), but if the number of combinations that do not match the number of packets is equal to or less than a predetermined number, the index information is You may make it detect as what corresponds with search object index information.

  In this example, the index creation unit 113 represents the index information of the index management information K10 shown in FIG. 3 and the index management information related to the content C2 (hereinafter referred to as “K20”) as index information that matches the search target index information. ) Index information (not shown) is detected. It is assumed that the index information related to the index management information K20 has an address “S1”, a flow number “3”, and an original flow number “3”.

  Then, the content information search unit 132 supplies the detected index information and content information (URL) corresponding to the index information to the reliability calculation unit 133.

  Then, the reliability calculation unit 133 calculates the reliability for each of the index information supplied from the content information search unit 132 (S204).

  Here, the reliability calculation unit 133 calculates the reliability for each of the index information of the index management information K10 and the index information of the index management information K20 using the above equation (1).

  For the index management information K10, since the original number of flows f0 = 3, the number of flows f1 = 3, and the total number of searched contents df = 2, the reliability A is 3 / (3 × √2) ≈0.71. Become. For the index management information K20, since f0 = 4, f1 = 3, and df = 2, the reliability A is 3/4 × √2) ≈0.53.

  Then, the reliability calculation unit 133 supplies content information (URL) corresponding to the index information supplied from the content information search unit 132 and corresponding reliability information to the output unit 134.

  Then, the output unit 134 performs processing for outputting the content information and the reliability information supplied from the reliability calculation unit 133 by a predetermined method (S205).

  The output unit 134 may output all content information when a plurality of pieces of content information are supplied from the reliability calculation unit 133, or some content information (for example, the reliability is the highest). Only high content information) may be output. Further, the output unit 134 may output only content information by omitting reliability information. The output unit 134 may output other information in association with the content information. For example, the output unit 134 may hold the current date and time, an identifier (for example, an IP address or a host name) of the user terminal 30, and the like, and output it together with the content information.

(A-3) Effects of Embodiment According to this embodiment, the following effects can be achieved.

  The estimation processing unit 13 of the content estimation apparatus 10 can estimate the content (content) of the traffic only from the flow information of the traffic without referring to the data (payload) of the packet transmitted from the WEB server 20. As a result, the content estimation apparatus 10 analyzes the contents of the packet when the information specifying the URL in the packet is encrypted and cannot be read or when there is too much traffic (number of packets) passing through the observation point. Even if the URL cannot be obtained or the traffic can be observed only by a device that can only see the flow information and can only acquire the flow information, the content accessed by the user (user terminal 30) can be specified. it can.

  Further, by using the content estimation device 10, the URL in the packet is directly acquired by a device that analyzes the accessed URL information such as user preference information analysis (for example, a server device that performs data mining processing). Even if the URL cannot be acquired due to problems such as processing load or the efficiency is low, information necessary for analysis processing (for example, content information including the URL accessed by the user terminal 30) is retained. Can do.

(B) Other Embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments as exemplified below.

(B-1) In the above embodiment, the information generation unit 11 has been described only for the process of adding index management information to the index management table 121. However, the crawling processing unit 112 performs the same content multiple times. When accessed, the contents of the index management table 121 may be updated based on the latest flow information acquired by the flow information acquisition unit 111 at that time. For example, if the content includes dynamically changing element content such as an advertisement banner, the content of flow information (particularly the number of packets) may change.

  When the URL of the same content is registered in the content information storage unit 12, the index creation unit 113 uses the flow information obtained this time and the reference index information registered in the index management table 121 for each flow ( The number of packets of each element content) is compared. Then, if the data size such as the number of packets is different as a result of the comparison, the index creating unit 113 determines the number of packets of the reference index information and the derived index information derived from the reference index information as the latest obtained packet. The number is updated to information indicating a range including the number and the number of packets obtained in the past.

  For example, with the index management information K10 to K16 relating to the content C1 already registered in the index management table 121 as shown in FIG. 3, the crawling processing unit 112 accesses the content C1 again, and the flow information is acquired. In this case, the index creation unit 113 may update the contents of the index management table 121 based on the latest flow information. Here, it is assumed that the element content EC12 is an image of a banner advertisement, and the number of packets related to the element content EC12 is larger than the previous P12 (hereinafter referred to as “P12 ′”).

  In this case, the index creation unit 113 sets P12 to P12 ′ as P12 in the item of the number of packets constituting the index information of the index management information (index management information K corresponding to the content C1) whose URL is U1. In the case of (P12> P12 ′), a process of rewriting “P12 to P12 ′” is performed so that the range of P12 ′ to P12 can be taken.

  As a result, the index management information K10 to K16 has contents as shown in FIG.

  When the range designation expression such as “P12 to P12 ′” described above is added to the item of the number of packets in the index management information, the content information search unit 132 compares the number of packets of the search target index information. Also, it is necessary to make a comparison in consideration of the above-mentioned range specification.

  Here, as an example, the content information search unit 132 compares the number of packets P2x of the search target index information with the number of packets of index information “P12 to P12 ′” constituting the index management information K10 shown in FIG. Will be described.

  For example, when “P2x” is a value within the range of “P12 to P12 ′”, the content information search unit 132 determines that “P2x” and “P12 to P12 ′” are the number of matching packets. May be. In addition, even if the content information search unit 132 deviates from “P12 to P12 ′”, if the error is within a predetermined range (for example, within ± 3%), “P2x” and “P12 to P12 ′”. May be determined as the number of matching packets. Specifically, for example, when “P2x” is within the range of “(P12 × 0.97) to (P12 ′ × 1.03)”, “P2x” and “P12 to P12 ′” match. The number of packets to be determined may be determined.

(B-2) In the above embodiment, URL information is registered as the content information of the index management table 121. However, information indicating a part of the content related to the URL and characteristics related to the content. May also be registered. For example, when the estimation result of the content estimation device 10 is used for a system that analyzes rumors information using a word in the content (for example, a data mining system), a word that appears in the content as content information ( For example, a predetermined number of detected words or keywords specified by SEO (Search Engine Optimization) may be registered. Thereby, in the system that uses the estimation result of the content estimation device 10 described above, it is possible to batch processes such as separately preparing a DB that associates the URL of the content with the word in the content and acquiring the word therefrom, The efficiency of the entire system can be improved.

(B-3) In the above embodiment, the flow information acquisition unit 111 of the information generation unit 11 and the flow information acquisition unit 131 of the estimation processing unit 13 are described as separate components, but depending on the network configuration (For example, when the positions of the flow observation points are the same) may be constructed as one component.

(B-4) The index creation unit 113 of the above embodiment creates not only the reference index information but also the derived index information derived from the reference index information. The generation may be omitted. In the above embodiment, the index creating unit 113 derives the derived index information considering the case where the flows indicated by the reference index information are combined and the case where a part of the flow indicated by the reference index information is omitted. Although an index is created, derived index information may be created in consideration of only a part of the above conditions. For example, the index creation unit 113 creates derived index information that considers only when the flows indicated by the reference index information are combined, and the derived index information when a part of the flow indicated by the reference index information is omitted. Generation may not be performed.

(B-5) In the above embodiment, the number of packets is observed as a value indicating the data amount of each flow, but an accumulated value of the packet data amount (may be an accumulated value of the data amount of only the payload) is observed. You may make it do.

(B-6) In the above embodiment, the estimation processing unit 13 includes the reliability calculation unit 133. However, the reliability calculation unit 133 is omitted (or operation can be stopped according to the setting). Configuration).

(B-7) In the above embodiment, the information generation unit 11 is mounted on the content estimation device 10, but the information generation unit 11 is omitted, and the estimation processing is performed by holding the data of the index management table 121 from the outside. You may make it utilize in the part 13. FIG. Further, the content estimation device 10 may read and hold the index management table 121 from a storage unit constructed as an external storage device without including the content information storage unit 12. Further, the information generation unit 11 or the estimation processing unit 13 alone may be constructed as a single information processing apparatus.

  DESCRIPTION OF SYMBOLS 10 ... Content estimation apparatus, 11 ... Information generation part, 111 ... Flow information acquisition part, 112 ... Crawling process part, 113 ... Index production part, 12 ... Content information storage part, 121 ... Index management table, 13 ... Estimation process part, 131 ... Flow information acquisition unit, 132 ... Content information search unit, 133 ... Reliability calculation unit, 134 ... Output unit, 20 ... WEB server, C1-C4 ... Content, EC11-EC13 ... Element content, 30 ... User terminal, N1 ... Internet, N2 ... Access network.

Claims (4)

A first information holding means for holding first feature information including a feature amount relating to a flow of data when each transmission data is transmitted from a data transmission device holding a plurality of transmission data;
Second information holding means for holding second feature information including feature quantities relating to the flow of data when any transmission data is transmitted from the data transmission device;
Collation processing means for performing processing for collating the second feature information held by the second information holding means and the first feature information held by the first information holding means;
An information processing apparatus comprising: an estimation processing unit that estimates transmission data transmitted by the data transmission device using a collation result of the collation processing unit. Transmission data requesting means for requesting the data transmission device to transmit each transmission data;
When the transmission data requested by the transmission data request means is transmitted from the data transmission device, the flow of the data is observed, and the first characteristic information relating to the transmission data is generated based on the observation result Means and
The information processing apparatus according to claim 1, wherein the first information holding unit holds first information for each transmission data generated by the feature information generation unit. When the transmission data is composed of a plurality of element data, the transmission data request means requests the data transmission device to transmit the element data at different timings,
The feature information generation unit acquires, for each element data, a feature amount related to a data flow when transmitted from the data transmission device, and acquires each element acquired in the first feature information related to the transmission data. The information processing apparatus according to claim 2, further comprising information on a feature amount of data. Computer
A first information holding means for holding first feature information including a feature amount relating to a flow of data when each transmission data is transmitted from a data transmission device holding a plurality of transmission data;
Second information holding means for holding second feature information including feature quantities relating to the flow of data when any transmission data is transmitted from the data transmission device;
Collation processing means for performing processing for collating the second feature information held by the second information holding means and the first feature information held by the first information holding means;
An information processing program that functions as an estimation processing unit that estimates transmission data transmitted by the data transmission device using a verification result of the verification processing unit.

Images