WO2013001571A1 - Unstructured data analysis system - Google Patents

Unstructured data analysis system Download PDF

Info

Publication number
WO2013001571A1
WO2013001571A1 PCT/JP2011/003695 JP2011003695W WO2013001571A1 WO 2013001571 A1 WO2013001571 A1 WO 2013001571A1 JP 2011003695 W JP2011003695 W JP 2011003695W WO 2013001571 A1 WO2013001571 A1 WO 2013001571A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
information
meta information
request
meta
Prior art date
Application number
PCT/JP2011/003695
Other languages
French (fr)
Japanese (ja)
Inventor
井口　慎也
横井　一仁
児玉　昇司
陽介 石井
Original Assignee
株式会社日立製作所
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社日立製作所 filed Critical 株式会社日立製作所
Priority to PCT/JP2011/003695 priority Critical patent/WO2013001571A1/en
Publication of WO2013001571A1 publication Critical patent/WO2013001571A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying

Definitions

  • the present invention relates to a system for extracting semantic information by analyzing unstructured data such as text, video, images, and audio.
  • Non-patent Document 1 In the Internet world, there is a growing trend to build applications that process large amounts of text data to extract and use useful information. Google has published Google Squared, which presents search results in tabular form (Non-patent Document 1). Unlike normal keyword search, Google Squared displays names and attributes belonging to the lower level of the search keyword in a tabular format. For example, for a query “cat”, names such as American Shorthair and Persian are displayed together with images and descriptions. Images and descriptions are links to resources on the Internet. One of the features is that the name and attributes to be displayed have an interface that can be extended by the user. IBM, on the other hand, built a question-and-answer (QA) system called "Watson" and won the highest amount by challenging a popular quiz program in the US (Non-patent Document 2).
  • QA question-and-answer
  • This system instantly seeks answers from a variety of complex queries expressed in natural language.
  • Common to these technologies is the aggregation of information from different resources to acquire and use useful knowledge.
  • the key is “information extraction” technology, which is how to extract useful information from a large amount of documents.
  • Non-patent Document 3 research and development of technology for processing synonyms with different expressions but the same meaning with computers is actively performed in order to process fluctuations in words handled by humans.
  • Non-patent Document 3 research and development such as high-speed similar image search (Non-Patent Document 4), voice search (Non-Patent Document 5), etc. to extract structured information from unstructured data in various formats such as video, images, and voices Has also been implemented.
  • Patent Document 1 a system that automatically recognizes natural input from a user such as a conversation sentence, extracts various information from a database, and presents it to the user.
  • This system is equipped with two different types of artificial intelligence, one with limited ability to recognize user input but fast response speed, and the other with wide and high accuracy recognition of user input.
  • it takes more processing time to improve accuracy.
  • the former is executed. If the latter is not possible, the latter is used to allow flexible control of the recognition rate and the processing time.
  • Patent Document 2 A hierarchical database that can add and delete various data analysis functions has also been proposed so that various unstructured data can be handled in an integrated manner in the existing database.
  • Google Squared [online], [Search June 13, 2011], Internet (URL http://www.google.com/squared) Watson in the world, [online], [search June 13, 2011], Internet (URL http://www-03.ibm.com/innovation/us/watson/ Morimoto, Toga; Supervised synonym extraction using context similarity and notation similarity, The 16th Annual Conference of the Association for Language Processing, 2010 Matsubara, et al; High-speed Similarity-based image retrieval with Data-alignment optimization using Self-organization algorithm, ISM2009 Kanda, Gaien; Open-vocabulary keyword detection from super-large scale speech database, MMSP, 2008
  • Patent Document 2 when the data analysis function to be called exists in an external environment such as the Internet, there is a possibility of data leaking in the process of transmitting / receiving the analysis data and the result to / from the data analysis function.
  • the data analysis function provider is not always reliable, and there is a possibility of data leakage from the data analysis function provider.
  • an application when an application requests data from the data analysis system, it provides a technology for acquiring optimal data within the time required by the application by specifying the data search time limit.
  • the data analysis system is connected to a storage device that stores unstructured data having metadata and a data body, A metadata extraction unit that acquires metadata of unstructured data from the storage device and creates first metadata from the metadata; Obtain metadata and data body of unstructured data from the storage device, extract semantic information representing the contents of the unstructured data from the metadata and the data body, and create second meta information having the semantic information An information extraction unit to A meta information repository that stores the first meta information and the second meta information in association with each other; A meta information management unit that extracts meta information stored in the meta information repository in response to a request from the application; A meta information management unit extracts meta information by a method specified by a request to create output data, and has a data output unit that outputs the output data to the application, When extracting the semantic information of the unstructured data, the information extraction unit calculates a score value indicating the degree that the semantic information represents the contents of the unstructured data, and includes the score value in the second meta information. .
  • the application accesses the data analysis system, specify the accuracy of the information to be acquired, and when searching the meta information repository in the data analysis system, search for information with a low score value is omitted, and the entire search
  • the amount can be reduced and the response speed can be improved.
  • search omissions are also reduced by searching the meta information repository for synonyms related to search keywords from applications. And even if the request from the application is unknown, it is possible to search the meta information repository by extracting the keywords included in the format.
  • the access request identification is an example of determination for distributing requests from applications.
  • FIG. 1 is a diagram illustrating a functional configuration example of a system.
  • reference numeral 101 is a data analysis system
  • reference numeral 102 is an external network
  • reference numeral 103 is an external information extraction unit
  • reference numeral 104 is a data holding system
  • reference numeral 105 is a raw data store
  • reference numeral 106 is an access request identification unit
  • reference numeral 107 is Meta information management unit
  • reference numeral 108 is an application
  • reference numeral 110 is an output suppression pattern determination unit
  • reference numeral 111 is a data shaping unit
  • reference numeral 112 is a meta information repository
  • reference numeral 113 is an output suppression rule
  • reference numeral 114 is an information extraction unit
  • reference numeral 115 is Outsourcing determination unit
  • 116 is an extraction information association unit
  • 117 is a data extraction system
  • 118 is a learning mechanism
  • 119 is a data store
  • 120 is external reference data
  • 121 is an extraction model
  • 122 is an access identification Rule
  • the data analysis system 101 is a system that manages meta information 124 generated by acquiring, analyzing, and analyzing data on the data store 119, and processing and outputting the meta information in response to a request from the application 108.
  • the substance of the data analysis system 101 includes a memory, a processing device (CPU), a network interface connected to the external network 102, an interface for accessing a storage device in which the data store 119 is mounted, a meta information repository 112, and an information extraction unit.
  • a computer having a storage device in which the usage rule 139, the extraction model 121, the learning sample data 129, and the learning restriction rule 130 are mounted and stored, and various processing units and functions constituting the data analysis system 101 described below are as follows.
  • the data analysis system 101 includes an access request identification unit 106, a meta information management unit 107, an output suppression pattern determination unit 110, a data shaping unit 111, a meta information repository 112, an information extraction unit 114, an outsourcing determination unit 115, an extraction information association unit 116, a metadata extraction unit 123, an external data capturing unit 128, learning sample data 129, a learning restriction rule 130, an information extraction unit usage rule 139, a data update monitoring unit 143, and an authentication processing unit 144.
  • the external network 102 is an external network such as the Internet, and a server equipped with an information extraction function that is accessed by the data analysis system during information analysis is arranged.
  • a server equipped with an information extraction function that is accessed by the data analysis system during information analysis is arranged.
  • an external information extraction unit 103, external reference data 120, and an external extraction model 142 are mounted.
  • the external information extraction unit 103 exists in the external network 102 and analyzes the metadata and the data body stored in the data store 119 in the same manner as the information extraction unit 114 of the data analysis system, and generates meta information 124.
  • the entity is realized, for example, by an information extraction processing program stored in a server or the like installed in a data center or the like being executed by a processing device of the server.
  • the data holding system 104 is a system having raw data (unstructured data) that is extracted by the data extraction system.
  • raw data unstructured data
  • an information sharing system in a company, a file server, etc. have a raw data store 105, and are connected to a data extraction system 117.
  • the raw data store 105 is a data storage device that stores the raw data 133 to be extracted by the data extraction system, and is a storage device such as a file storage or a hard disk in the server.
  • the access request identifying unit 106 analyzes the request content from the application according to the access identification rule to determine the processing method, and requests the authentication processing unit, metadata extraction unit, outsourcing determination unit, and meta information management unit to perform processing. It is a function and is connected to the meta information management unit 107 and the application 108.
  • the meta information management unit 107 is a function that searches the meta information repository using data received from the access request identification unit or the extracted information association unit and requests the data shaping unit to process data for output. It also manages the meta information repository.
  • the access request identifying unit 106 and the data shaping unit 111 are connected.
  • the application 108 is a system that uses the data analysis system 101. For example, a system that integrates and displays a plurality of medical information, a content search system that uses a natural language, and the like are conceivable.
  • the entity is realized by a processing device on a computer executing an application program on a memory.
  • the application is connected to the access request identifying unit 106 and the output suppression pattern determining unit 110.
  • the output suppression pattern determination unit 110 checks the content of the shaped data created by the data shaping unit according to the content of the output suppression rule, and determines whether output is possible.
  • the data shaping unit 111 and the application 108 are connected.
  • the data shaping unit 111 has a function of processing data (meta information) acquired from the meta information repository according to the schema definition and outputting the processed data to the application, and is connected to the meta information management unit 107 and the output suppression pattern determination unit 110.
  • the data body 134 may be acquired from the data store 119 and integrated, or the application 108 may execute this processing.
  • the meta information repository 112 provides functions for storing meta information 124 and searching for and outputting necessary meta information in response to requests from various modules.
  • a storage format of the meta information for example, flexible structures such as RDF (Resource Description Framework), graph structure, and tree structure are conceivable.
  • RDF Resource Description Framework
  • graph structure a structure that represents and stores all data in the form of subject-predicate-object.
  • the access request identifying unit 106, the output suppression pattern determining unit 110, and the data shaping unit 111 store information that is referred to when executing processing.
  • the meta information repository 112 includes an output suppression rule 113, an access identification rule 122, meta information 124, and a schema definition 125.
  • the output suppression rule 113 registers a judgment rule used for output suppression, for example, a table that manages a combination of a series of keywords that are not permitted to be output for each access authority. However, the substance of the output control rule 113 is converted into a form corresponding to the management structure of the meta information repository 112 and stored.
  • the information extraction unit 114 is a function that extracts the meaning from the data body acquired via the outsourced determination unit and creates meta information. For example, when a sunflower image is inserted, it is a “sunflower” and becomes a “three” image. This is a function for generating information such as being included and executing processing for conversion to meta information, and is composed of a learning mechanism 118 and an extraction model 121.
  • the outsourcing determination unit 115 extracts the meta information of the data extracted from the data storage system and stored in the data store or the data attached to the access request acquired from the access request identification to the information extraction unit on the external network. It is determined whether it is to be performed or to be performed by an information extraction unit in the data analysis system.
  • the extracted information association unit 116 associates the metadata, the meta information obtained from the outsourcing determination unit, and the information extraction unit, and stores them in the meta information repository access. Note that when associating metadata and meta information, the meta information repository may be referred to.
  • the data extraction system 117 is a system that executes processing for extracting raw data from a single data storage system or a plurality of data storage systems and storing it in a data store, and is connected to the data storage system 104.
  • the learning mechanism 118 is a function for learning based on data provided by the external data capturing unit and creating an extraction model.
  • the data store 119 is a storage device that holds the data extracted by the data extraction system, and includes a file storage or the like. Depending on the specifications of the data analysis system, the metadata and the data body may be stored separately.
  • External reference data 120 is data existing in a storage device on an external network that is referred to by the information extraction unit as necessary.
  • the extraction model 121 is a data group used as a criterion when used for extracting meaning from information input to the information extraction unit, and is created by a learning mechanism. Using the criteria registered in the extraction model, the information extraction unit creates meta information from the input information (data body). It is also conceivable that a plurality of extraction models 121 are prepared and switched for use such as request contents of an application or backup when learning fails.
  • the access identification rule 122 is a group of rules used for determination by the access request identification unit.
  • the metadata extraction unit 123 has a function of extracting metadata from data acquired from the data store or the access request identification unit.
  • Meta information 124 is information (meta information) extracted by the information extraction unit.
  • Schema definition 125 is a group of rules that the data shaping unit refers to when processing data (meta information) into a format according to the request of the application. For example, when meta information is converted into a table format and output, which attribute of the meta information is used as a table column is described.
  • the external data capturing unit 128 has a function of capturing an extraction model and data on an external network as necessary, and performing comparison processing with the sample data for learning, and is connected to the sample data 129 for learning.
  • the learning sample data 129 is sample data used by the information extraction unit for learning.
  • the sample data 129 includes document data, sample images, sample audio data, etc. with parts of speech, and is connected to the external data capturing unit 128. .
  • the learning restriction rule 130 is a rule that controls the learning mechanism of each information extraction unit. Control learning timing using various learning data. For example, whether or not to learn the result of the information extraction process of data from the data store, if the data in the meta information repository is used for learning, automatically learn the result when the meta information repository is updated Whether or not is described.
  • Raw data 133 is a data body to be extracted by the data extraction system, and includes a data body 134 and metadata 135.
  • the data body 134 is an entity such as an image, sound, or document.
  • Metadata 135 exists for each analysis target data, and is information that explains what the analysis target data is. For example, data indicating data storage location information, last update date / time information, last access date / time information, access authority, file owner, and the like.
  • the information extraction unit usage rule 139 is a group of criteria used by the outsourcing determination unit to determine an information extraction outsourcing destination.
  • the external extraction model 142 is model data used for extracting metadata created by the learning mechanism, and is arranged in a data center or the like existing on an external network.
  • the data update monitoring unit 143 has a function of monitoring the update state of data on the data store and notifying the metadata extraction unit and the outsourcing determination unit when an update is detected.
  • the authentication processing unit 144 has a function of receiving authentication information from the application via the access request identifying unit and checking the validity of the application access by collating with the authentication information stored in the meta information repository.
  • FIG. 2 is a diagram showing an example of the structure of the meta information 124 stored in the meta information repository 112.
  • the meta information is expressed by, for example, a graph structure or a tree structure composed of nodes and edges connecting the nodes. This makes it possible to express various things and the relationships between them.
  • an example of a node and an edge is shown.
  • An example of data held by nodes and edges will be described later.
  • reference numeral 201 is node 1 and reference numeral 202 is an edge.
  • the node 1201 has a plurality of attribute information characterizing the node itself and an edge list related to the node.
  • Edge 202 represents the relationship between nodes. It has an attribute list that characterizes the relationship between nodes.
  • Figure 3 shows the metadata structure
  • Metadata 135 is a set of attribute values describing the outline of the data body (file etc.) of raw data. A list of these parameters is shown. Note that not all attributes included in this example are always necessary, and the metadata is configured by combining according to the situation.
  • “Attribute name” 301 is a general name of the attribute value included in the metadata.
  • Line 303 is a description regarding what “attribute name” is “date and time”
  • line 304 is a description regarding what “attribute name” is “size”
  • line 305 indicates that “attribute name” is “”
  • Line 306 is a description about what “attribute name” is “type”
  • line 307 is a description about what “attribute name” is “access authority”. is there.
  • Example of attribute value 302 is an example of an attribute value.
  • attribute name is “date and time”
  • example of attribute value” is “2011/7/7”, for example, indicating the creation date or update date and time of the data body and metadata.
  • attribute name is “size”
  • example of attribute value is, for example, “115 MByte”, indicating the size of raw data.
  • attribute name is “location of acquisition source”
  • example of attribute value is “NAS1 / doc”, for example, and indicates information for specifying the acquisition source (storage location) of raw data.
  • attribute name is “type”
  • example of attribute value is, for example, “sentence, video, image, etc.” and indicates the type of raw data.
  • attribute name is “access authority”
  • example of attribute value indicates a list of accessible user IDs.
  • Fig. 4 is a diagram showing nodes of meta information.
  • “Attribute name” 401 is a general name of the attribute value included in the node.
  • Line 403 is a description related to what “attribute name” is “date and time”
  • line 404 is a description related to what “attribute name” is “node type”
  • line 405 is a description related to “attribute name”.
  • Line 406 is a description related to “connection edge list”, line 406 is a description related to “attribute name” being “metadata”, and line 407 is a description related to “attribute name” being “access authority”
  • Line 408 is a description regarding an item whose “attribute name” is “extraction information”
  • line 409 is a description regarding an item whose “attribute name” is “score information”
  • line 410 is an attribute name Is a description of what is “number of uses”.
  • Example of attribute value 402 is an example of an attribute value.
  • the “attribute name” is “date and time”
  • the “example of attribute value” is, for example, “2011/7/7” and indicates the creation date or update date of the node of the meta information.
  • “attribute name” is “node type”
  • “example of attribute value” is, for example, a proper noun, a keyword, file information, etc., and indicates the type of raw data.
  • attribute name is “connection edge list”
  • “example of attribute value” is “edge 1, edge 5,...”, For example, and indicates identification information of an edge connected to the node.
  • attribute name is “metadata”
  • example of attribute value is “AAA metadata”, for example, indicating the metadata identification information of the raw data from which the metadata information is extracted and the metadata itself .
  • attribute name is “access authority”
  • example of attribute value is, for example, “accessible user ID list”, and indicates the ID or user group identification name of a user who can access the node.
  • attribute name is “extraction information”
  • example of attribute value indicates, for example, information extracted from raw data such as extracted keywords, values, and the like.
  • attribute name is “score information”
  • example of attribute value is a numerical value such as “4.3”, for example, and represents the likelihood of information registered as extracted information 408 calculated based on a predetermined standard. Information.
  • the model 121 of the information extraction unit 114 is provided. It is conceivable that the similarity between how much the image pattern data group recorded in the image data and the analysis target data are similar is indicated as a score value. An equivalent method can be considered for analysis of voice, sensor information, and the like.
  • attribute name is “use count”
  • example of attribute value is a numerical value such as “15”, for example, and is information indicating the number of times the information of the node is output to the application.
  • FIG. 5 is a diagram showing a list of attributes held by edges constituting the meta information.
  • the attribute may be added or deleted according to the purpose of use of the edge.
  • “Attribute name” 501 is a general name of the attribute value included in the edge
  • line 503 is a description regarding “attribute name” being “date and time”
  • line 504 is “attribute name” is “edge type”.
  • Line 505 is a description regarding an item whose “attribute name” is “connection source node”
  • line 506 is a description regarding an item whose “attribute name” is “connection destination node”.
  • line 507 is a description regarding the “attribute name” being “access authority”
  • line 508 is a description regarding the “attribute name” being “extraction information”
  • line 509 is the description regarding “attribute name” being “score information” ”
  • a line 510 is a description about the item whose“ attribute name ”is“ number of times of use ”.
  • attribute value 502 describes an example of an attribute value.
  • attribute name is “date and time”
  • example of attribute value is “2011/7/7” and indicates the date and time when the edge was generated and updated.
  • attribute name is “edge type”
  • example of attribute value is, for example, a parent-child relationship, a similarity relationship, a synonym relationship, etc., and a relationship between a connection source node and a connection destination node connected by the edge Indicates.
  • the “attribute name” is “connection source node”
  • the “attribute value example” is, for example, “node 1”, which is identification information of the connection source node of the edge
  • the “attribute name” is “connection destination node”.
  • “example of attribute value” is “node 5”, which is identification information of a connection destination node of the edge.
  • “attribute name” is “access authority”
  • “example of attribute value” is a list of IDs of users who can access the edge.
  • “attribute name” is “extracted information”
  • “example of attribute value” indicates, for example, extracted keywords, values, and the like and extracted from raw data.
  • attribute name is “score information”
  • example of attribute value is a numerical value such as “7.3”, for example, and represents the likelihood of information registered as extracted information 508 calculated based on a predetermined standard. Information. An example of the calculation method is equivalent to the “score value” at the node.
  • attribute name is “usage count”
  • example of attribute value is a numerical value such as “15”, for example, and is information indicating the number of times the information on the edge is output to the application.
  • FIG. 6 is a diagram showing an example of a request issued by the application to the data analysis system.
  • FIG. 6 shows an example of parameters specified by the data output request issued by the application to the data analysis system. Note that it is not necessary to specify all the parameters shown in FIG. 6 depending on the request contents and the type of application.
  • “Parameter name” 601 is the name of the parameter.
  • Line 603 is a description regarding what “parameter name” is “request ID”
  • line 604 is a description regarding what “parameter name” is “information registration permission flag”
  • line 605 indicates “parameter name”.
  • Line 606 is a description regarding what “parameter name” is “request confidentiality level”
  • line 607 is a description regarding “parameter name” is “accuracy level”.
  • Line 608 is a description regarding what “parameter name” is “authentication information”
  • line 609 is a description regarding what “parameter name” is “keyword”
  • line 610 indicates “ The description of the parameter name is “synonym search permission flag”
  • line 611 is the description of the parameter name “meta information”
  • line 612 is the parameter name “required”. Is a description about what is "query”
  • line 613 is a description about what "parameter name” is "output format”
  • line 614 is a description about what "parameter name” is "cache flag”
  • Line 615 is a description regarding the “parameter name” being “prefetch flag”
  • line 616 is a description regarding the “parameter name” being “file”.
  • Example of parameter value 602 is an example of a parameter value.
  • “parameter name” is “request ID”, “example of parameter value” is “111”, for example, and is information for uniquely identifying the request.
  • “parameter name” is “information registration permission flag”, “parameter value example” is “permitted” or “non-permitted”, and in the case of “permitted”, the number of accesses of meta information used as a result of this request Updates and registration of new meta information when it is generated are allowed to be registered in the meta information repository. “Non-permitted” indicates that the above processing is not performed.
  • the “parameter name” is “data search time limit”
  • the “parameter value example” is a time such as “100 ms”, for example, and the time allowed by the application as the data search time is specified.
  • “parameter name” is “request confidentiality level”
  • “example of parameter value” is a numerical value indicating the confidentiality of the request, such as “5”.
  • “parameter name” is “accuracy level”
  • “parameter value example” is a numerical value indicating the accuracy level of meta information allowed by the application, such as “4”. As the accuracy value, for example, the case of using the “score value” of the meta information is considered.
  • parameter name is “authentication information”
  • “parameter value example” is application user authentication information
  • “parameter name” is “keyword”
  • “parameter value example” is, for example, “” Aspirin "" Is a search keyword specified by the application.
  • “Parameter name” is “Synonym search permission flag”
  • “Example of parameter value” is “Permitted” or “Not permitted”. In the case of “not allowed”, it is requested not to search for synonyms of keywords.
  • the “parameter name” is “meta information”
  • the “example of parameter value” is, for example, “meta information structure data representing an aspirin component”, which is information for specifying the type of meta information to be acquired.
  • parameter name is “request query”
  • example of parameter value is “SQL, SPARQL, etc. for acquiring aspirin components and prescription list”
  • parameter name is “output format”
  • example of parameter value indicates an output format specified by the application, such as a table format or a list format.
  • Parameter Name is “Cache Flag”, “Parameter Value Example” is “Cache Required” or “Cache Not Required”. The cached information is cached, and the cached information is returned when an equivalent request is received. In the case of “no cache required”, every time the same request is received from the application, the meta information is accessed and the output data is generated again.
  • Parameter name is "Prefetch flag”
  • “Parameter value example” is “Valid” or “Invalid”. If it is "Valid”, data with a structure similar to the data output content is meta information Search in the repository, create the list data, and respond using this list data when an equivalent request comes from the application.
  • the list data may be stored on the meta information repository, or another storage unit may be created. In the case of “invalid”, the above processing is not performed.
  • “parameter name” is “file”
  • “example of parameter value” is information for specifying a file to be searched such as “aspirin image”.
  • FIG. 7 is a diagram showing an example of an access identification rule.
  • “Attribute name” 701 is a general name of an attribute value included in the metadata.
  • Line 703 is a description relating to “attribute name” being “outsourcing processing time threshold”
  • line 704 is a description relating to “attribute name” being “authentication confidentiality level”
  • line 705 is “ This is a description related to “attribute name” being “corresponding request query type list”.
  • Example of attribute value 702 is an example of an attribute value.
  • the “attribute name” is “outsourcing processing time threshold”
  • the “attribute value example” is a time such as “100 ms”
  • the “attribute name” is “attribute value required for authentication confidentiality level”.
  • “Example” is a numerical value indicating the level of confidentiality, such as "2”.
  • “Attribute name” is "Corresponding request query type list”
  • Example of attribute value is data such as CSV, SQL, SPARQL, ... This is a list of queries that can be processed by the analysis system.
  • FIG. 8 is a diagram showing a process of extracting metadata from raw data, converting it into meta information, and registering it in the meta information repository 112.
  • the raw data acquired by the data extraction system from the data holding system is stored in the data store, the data analysis system extracts the metadata from this, and creates the meta information based on this metadata. It is a process of registering in the meta information repository.
  • Step 801 is a process in which the data extraction system acquires raw data from the data holding system.
  • Step 802 is a process in which the data holding system separates the metadata and the data body from the acquired raw data and stores them separately in the data store.
  • the data holding system may store the acquired raw data in the data store without separating the metadata and the data body.
  • Step 803 is a process in which the metadata extraction unit of the data analysis system acquires metadata from the data store.
  • the metadata extraction unit may receive the notification that the data store has been updated from the data extraction system and acquire the metadata in response to this notification, or the data update monitoring unit may determine whether or not the data store has been updated.
  • the metadata extraction unit may acquire metadata when it is monitored and an update of the data store is detected.
  • Step 804 is processing in which the metadata extraction unit converts the metadata into metadata.
  • a node of meta information having an attribute name and an attribute value including data such as a file name, a file size, and a metadata storage destination that the metadata has is created.
  • a meta information node having the metadata itself as one attribute value may be generated.
  • Step 805 is processing for storing the meta information generated by the metadata extraction unit in step 805 in the meta information repository.
  • Step 806 is processing for examining whether or not the data analysis system continues to extract information from the data body. If information extraction from the data body is executed, the process proceeds to step 807. If the information extraction from the data body is not executed, the process is terminated. For example, when the data body is moving image data, video data and audio data may be included. Information extraction needs to be performed on all of these data.
  • the information extraction execution determination process referred to here indicates a process of determining whether or not the analysis of all the analysis target data included in the data body has been completed.
  • Step 807 is a process in which the outsourcing determination unit reads the data body from the data store.
  • Step 808 is a process for checking whether or not the outsourcing determination unit can cooperate with the external information extraction unit 103 of the external network. If it is possible to cooperate with the information extraction unit of the external network, the process proceeds to step 809. If it is not possible to cooperate with the information extraction unit of the external network, the process ends. For example, the possibility of cooperation is determined based on criteria such as whether the data analysis system is connected to an external network or whether access to the external network is permitted.
  • Step 809 is a process in which the outsourcing determination unit checks the metadata.
  • Step 810 is processing for examining whether or not the outsourcing determination unit needs to be outsourced, that is, whether the external information extraction unit 103 needs to create meta information from the data body. If outsourcing is necessary, go to Step 811. If outsourcing is not necessary, go to Step 813. As this judgment criterion, for example, in the case of a data format that is not supported by the information extraction unit in the data analysis system, it may take a long time to process by the internal information extraction unit, and the processing efficiency may be poor.
  • Step 811 is a process in which the outsourcing determination unit sends the metadata acquired from the data store and the data body to the external information extraction unit and requests information extraction (meta information creation).
  • the external information extraction unit that has received the request extracts semantic information representing the content of the raw data from the metadata and the data body in the same manner as the processing performed by the information extraction unit 114 in step 813 described later. Further, as an index indicating how accurately this semantic information represents the content of the raw data, a score value (score information) is generated by a predetermined method, and meta information having score information and semantic information is created. .
  • encrypted communication may be used to prevent leakage of information to the outside.
  • Step 812 is processing in which the external information extraction unit returns the meta information extracted based on the request of Step 811 to the outsourcing determination unit.
  • encrypted communication may be used to prevent information leakage to the outside.
  • Step 813 is a process in which the data analysis system calls the information extraction unit 114 to create metadata and meta information from the data body.
  • the information extraction unit 114 extracts semantic information representing the content of the raw data from the metadata and the data body. Further, a score value indicating the degree to which the semantic information represents raw data is calculated and used as score information. Then, meta information having semantic information and score information is created.
  • the information extraction unit 114 analyzes the structure of the image and provides a function for generating a keyword meaning it as an attribute value, the image pattern data group recorded in the model 121 of the information extraction unit 114, By determining how similar the analysis target data is, by generating meta information with the keyword corresponding to the image pattern data with the highest similarity and the attribute value with the similarity as “score value” Meta information explaining what the analysis target image means can be generated.
  • An equivalent method can be considered for analysis of voice, sensor information, and the like.
  • Step 814 is a process of associating the meta information converted by the extraction information association unit with the meta information extracted by the information extraction unit and storing it in the meta information repository.
  • Fig. 9 is a diagram showing the cooperative operation between the application and the data analysis system.
  • Step 901 is a process in which the application issues a processing request to the data analysis system.
  • An example of the processing request issued by the application is as shown in FIG.
  • various formats such as a unique format, a Web service call, and an SQL can be considered as the format of the processing request.
  • Step 902 is a process in which the access request identification unit determines the request content according to the access request identification rule. Details will be described in detail with reference to FIG.
  • Step 903 is a process for checking whether the processing request cannot be processed (accessed) by the meta information management unit based on the determination in step 902. If the meta information management unit cannot process, the process proceeds to step 904. If it can be processed by the meta information management unit, the process proceeds to step 905.
  • Step 904 is processing in which the external information extraction unit 103 or the information extraction unit 114 generates access meta information (that is, meta information used as a search condition). Details will be described later with reference to FIG.
  • Step 905 is a process in which the meta information management unit acquires meta information corresponding to the application request from the meta information repository. Specifically, based on keywords specified in the application request, request query type, meta information, file, etc., search meta information in the meta information repository for meta information that matches this condition. To get it. If the meta information has been generated in step 904, the meta information management unit searches the meta information repository using the generated meta information as a search condition. When the accuracy level 607 is specified in the application request, the meta information management unit searches the meta information only for the meta information whose score information 409 or 509 is the accuracy level 607 or higher. I do.
  • the meta information management unit sets the synonym dictionary (each of the plurality of keywords included in the meta information repository).
  • the synonym of the keyword is first extracted, and the meta information is searched using each of the extracted one or more synonyms as a search condition. Get meta information.
  • Step 906 is processing in which the data shaping unit and the output suppression pattern determination unit reshape the acquired meta information data and determine output suppression. Details will be described later with reference to FIG.
  • Step 907 is a process for outputting the formatted data to the application.
  • Step 908 is a process for checking whether the prefetch flag 615 of the processing request from the application is valid. If the prefetch flag of the application request is valid, the process proceeds to step 909. If the prefetch flag of the application request is not valid, the process ends.
  • Step 909 searches the meta information repository for meta information having a structure similar to the meta information output in step 907, creates a list of identification information of the meta information obtained by searching, and stores it in the meta information repository. Process.
  • FIG. 10 is a diagram showing an access request identification determination.
  • This processing flow corresponds to the details of step 902 in FIG. 9 and is a process in which the access request identifying unit distributes requests from applications.
  • Step 1001 is a process for acquiring an access request (processing request) from the application.
  • Step 1002 is a process of checking whether the request confidentiality level 606 set in the processing request is equal to or higher than the authentication confidentiality level 704 set as the access identification rule. If the requested confidentiality level 606 is the authentication required confidentiality level 704 or higher, the process proceeds to step 1003. If the requested confidentiality level 606 is not higher than the authentication confidentiality level 704, the process proceeds to step 1008.
  • Step 1003 is a process for checking whether there is authentication information 608 in the access request. If there is authentication information, the process proceeds to step 1004. If there is no authentication information, the process proceeds to step 1007.
  • Step 1004 is a process of requesting execution of authentication processing by passing authentication information 608 to the authentication processing unit.
  • Step 1005 is a process for checking whether an authentication error has occurred based on the authentication result received from the authentication processing unit. If an authentication error has occurred, go to Step 1006. If no authentication error has occurred, the process proceeds to step 1008.
  • Step 1006 is a process for returning an error to the application.
  • Step 1007 is a process for returning an error to the application.
  • Step 1008 is a process for checking whether or not the search meta information 611 is specified in the access request. If the search meta information is designated, the process proceeds to step 1009. If no search meta information is specified, the process proceeds to step 1010.
  • Step 1009 is a process in which the meta information management unit determines that the meta information repository can be searched using the search meta information 611 as a search condition. In this case, it is determined in step 903 of FIG. 9 that the meta information management unit can process.
  • Step 1010 is a process for checking whether or not the keyword 609 is specified in the access request. If a keyword is specified, the process proceeds to step 1011. If no keyword is specified, the process proceeds to step 1014.
  • Step 1011 is a process for checking whether the synonym search is permitted in the access request, that is, whether the synonym search permission flag 610 is permitted. If the synonym search is permitted, the process proceeds to step 1013. If the synonym search is not permitted, the process proceeds to step 1012.
  • Step 1012 is processing in which the meta information management unit determines that the meta information can be acquired (accessed) from the meta information repository using the keyword. In this case, it is determined in step 903 of FIG. 9 that the meta information management unit can process.
  • Step 1013 is processing for determining that it is necessary to execute access synonym extraction processing of the meta information repository.
  • the meta information management unit determines that the synonym extraction process is necessary.
  • Step 1014 is a process for checking whether the request query 612 is specified in the access request. If a request query is specified, the process proceeds to step 1015. If no request query is specified, the process proceeds to step 1018.
  • Step 1015 is a process for checking whether or not the request query confirmed in Step 1014 is registered in the corresponding request query type list 705 of the access identification rule shown in FIG. If the request query is registered in the corresponding request query type list of the access identification rule, the process proceeds to step 1017. If the request query is not registered in the corresponding request query type list of the access identification rule, the process proceeds to step 1016.
  • Step 1016 is processing in which the information extraction unit determines that query analysis is necessary.
  • the process proceeds to step 904, where the specified query is executed by the external information extraction unit, the information extraction unit, etc. Information will be extracted.
  • the meta information obtained by this query processing will be used later as a search condition by the meta information management unit in step 905 of FIG.
  • Step 1017 is processing in which the meta information management unit executes query processing to determine that search meta information can be acquired (accessed) from the meta information repository.
  • Step 1018 is a process for checking whether the file 616 is specified in the access request. If a file is specified, the process proceeds to step 1019. If no file is specified, the process proceeds to step 1020.
  • Step 1019 is a process of extracting metadata from the file, converting it into meta information, and storing it in the meta information repository.
  • the calculation of the meta information and the score value is as described above. In this step, if metadata extraction fails, an error is returned.
  • Step 1020 is a process in which the meta information management unit determines that the meta information can be acquired (accessed) from the meta information repository using the meta information created in step 1019 as the search meta information. In this case, it is determined in step 903 of FIG. 9 that the meta information management unit can process.
  • FIG. 11 is a diagram showing access meta information generation. This process is a detailed flow of step 904 in FIG. 9 and shows a flow of a process of extracting meta information for access to the meta information repository based on the analysis target data included in the request from the application.
  • Step 1101 is a process in which the metadata extraction unit transmits the analysis target data to the outsourcing determination unit.
  • the analysis target data includes, for example, a request query 612 and the like, and attached data such as voice data and image data attached to a request from the application.
  • Step 1102 is a process in which the meta information extraction unit extracts metadata of analysis target data.
  • Step 1103 is a process in which the outsourcing determination unit checks whether the data to be analyzed is an outsourcing object.
  • the content of the metadata of the analysis target data is compared with the information extraction unit usage rule, and it is determined whether or not it is an outsourcing target.
  • the type of query that can be processed by the external information extraction unit 103 is registered in the information extraction unit usage rule. Therefore, referring to this, it is determined whether the query processing specified by the processing request from the application can be executed by the external information extraction unit 103, and if it can be executed, it may be set as an outsourcing target.
  • Step 1104 is processing in which the outsourcing determination unit sends analysis target data to the external information extraction unit.
  • Step 1105 is a process in which the outsourcing determination unit receives the analysis result (that is, the extracted meta information) of the analysis target data from the external information extraction unit.
  • Step 1106 is a process in which the outsourcing determination unit inputs analysis target data to the information extraction unit, and the information extraction unit that receives the data extracts information by executing information extraction.
  • the extraction method is as exemplified above.
  • Step 1107 is processing for examining whether or not unanalyzed data exists in the analysis target data. In this step, when the analysis target data is composed of a plurality of data such as moving images and voices, it is confirmed whether or not the processing has been executed for all of them. If unanalyzed data exists in the analysis target data, the process returns to step 1103. On the other hand, if unanalyzed data does not exist in the analysis target data, the process proceeds to step 1108.
  • Step 1108 is a process of integrating all extracted meta information in the extracted information association unit.
  • the integrated information is stored in the meta information repository.
  • Fig. 12 is a diagram showing an overview of data shaping and output suppression processing. This process corresponds to the details of step 906 in FIG. 9, and the data shaping unit 111 and the output suppression pattern determination unit 110 process the meta information into an output format according to the application request, and the processing result is satisfactory from the viewpoint of security. This is the process of controlling whether the output is determined.
  • Step 1201 is processing in which the data shaping unit acquires the output target meta information acquired from the meta information repository by the meta information management unit in step 905.
  • Step 1202 is a process in which the data shaping unit acquires the schema definition corresponding to the access request from the application from the meta information repository.
  • the schema definition describes the rules for converting meta information into table format, list format, XML format, etc., according to the output format specified as output format 613 in the application access request Get things from the meta information repository.
  • Step 1203 is a process in which the data shaping unit checks whether the data body reference is specified in the schema definition or the output target meta information. If the data body reference is specified in the schema definition or the output target meta information, the process proceeds to step 1204. If the data body reference is not specified in the schema definition or the output target meta information, the process proceeds to step 1205.
  • Step 1204 is a process in which the data shaping unit acquires the data body whose reference is specified from the data store.
  • Step 1205 is a process in which the data shaping unit generates output data based on the schema definition.
  • Step 1206 is a process in which the output suppression pattern determination unit checks the output data format.
  • the output suppression pattern determination unit checks the format of the output data based on the rules described in the output suppression pattern determination rules in the meta information repository.
  • Step 1207 is a process in which the output suppression pattern determination unit checks whether the output data includes an output prohibition structure based on the output suppression pattern determination rule. If the output prohibition structure is included in the output data, the process proceeds to step 1208. If the output prohibition structure is not included in the output data, the process proceeds to step 1210.
  • Step 1208 is a process in which the output suppression pattern determination unit converts a portion corresponding to the output prohibition structure in the output data into dummy data.
  • the dummy data for example, a method of assigning a predetermined value according to the type such as a character string or a value can be considered.
  • Step 1209 is a process in which the output suppression pattern determination unit outputs output data with a warning to the application.
  • Step 1210 is a process in which the output suppression pattern determination unit outputs output data to the application.
  • FIG. 13 is a diagram showing a flow of processing for registering meta information input by the application in the meta information repository.
  • Step 1301 is a process in which the application issues a meta information registration request to the data analysis system.
  • This meta information registration request includes meta information to be registered.
  • Step 1302 is a process in which the access request identifying unit searches the meta information repository using the meta information received from the application as a search key.
  • Step 1303 is a process for checking whether a structure partially matching the received meta information is found in the meta information repository. If a partially matching structure is found, go to step 1304. If a partially matching structure is not found, go to step 1305.
  • Step 1304 is processing for adding meta information from the application to the top node of the matching structure.
  • Step 1305 is a process of registering as a new data structure.
  • FIG. 14 is a diagram showing a flow of processing for detecting that data in the data store has been updated and performing information extraction.
  • Step 1401 is a process in which the data update monitoring unit detects a data store update.
  • Step 1402 is processing for executing metadata extraction and information extraction processing (creation / update of meta information for update data) for the update data.
  • Step 1403 is a process of updating (reflecting) the meta information corresponding to the corresponding data in the meta information repository.
  • FIG. 15 is a diagram showing the flow of the learning process of the information extraction unit using the learning sample data.
  • Step 1501 is a process in which the information extraction unit starts learning.
  • Step 1502 is a process for checking whether external data can be imported. If external data can be imported, the process proceeds to step 1503. If external data cannot be imported, the process proceeds to step 1509. As a criterion for external data capture, for example, information that learning data that can be accessed by the data analysis system in the external network is registered in the data analysis system and can be connected to the external network. Conceivable.
  • Step 1503 is a process in which the external data capturing unit captures external reference data.
  • Step 1504 is a process for collating the learning sample data with the captured data captured by the external data capturing unit.
  • Step 1505 is a process for checking whether there is a contradictory portion between the fetched data fetched by the external data fetching unit and the learning sample data. If there is a contradiction, the process proceeds to step 1506. If there is no conflict, go to Step 1509.
  • This contradiction means, for example, that the internal learning data is defined as “dogs are animals” and “animals and plants are different”, and external data is defined as “dogs are plants”. In some cases, simple values such as “Mt. Fuji is 3776m high” and “Mt. Fuji is 3022m high” can be considered.
  • Step 1506 is a process for replacing the external data with the learning sample data for the inconsistent portion.
  • Step 1507 is a process for checking whether or not replacement is possible. If it cannot be replaced, the process proceeds to step 1508. If not replaceable, go to step 1509.
  • Step 1508 is a process for canceling the use of the external reference data.
  • Step 1509 is a process in which the learning mechanism executes a learning process using the available learning sample data in the information extraction unit to correct the extraction model.

Abstract

[Problem] When a data management system stores a large volume of unstructured data and an application specifies a short response time, there is insufficient time to extract and process information from the stored unstructured data, and sending back an on-time response is impossible. Also, even if an application makes a demand with a high level of secrecy, a data management system will request data analysis from an external network, and information leakage will occur. [Solution] When an application demands data from a data management system, a response time, information secrecy level and response precision level are specified. A data management system provides a means for requesting data analysis from an external network only when the secrecy level is low. Information extracted from unstructured data is managed by appending a precision-based score thereto, and is mapped to a precision level. When the response time is short, only locations with large scores are searched from managed information.

Description

非構造データ解析システムUnstructured data analysis system
　本発明は、テキスト、映像、画像、音声などの非構造データを解析して意味情報を抽出するシステムに関する。 The present invention relates to a system for extracting semantic information by analyzing unstructured data such as text, video, images, and audio.
　近年、医療や金融、企業情報、政府機関、ビデオ監視分野など様々な分野において、従来は蓄積・保管・参照するだけであった非構造データを分析し、学術研究やマーケティングなどビジネスに再活用したいというニーズが高まっている。しかし非構造データはデータ・サイズやデータ量が大きいため処理に時間がかかり、また数値データとは異なり、そのままの形式では機械処理できないため、再活用が困難という問題がある。 In recent years, in various fields such as medical care, finance, corporate information, government agencies, video surveillance, etc., we want to analyze unstructured data that has traditionally only been stored, stored, and referenced and reuse it in business such as academic research and marketing Needs are growing. However, unstructured data takes a long time to process because of its large data size and data volume, and unlike numerical data, it cannot be processed mechanically in its original form, and therefore has a problem that it is difficult to reuse.
　例えば大量の画像データから類似した画像を高速に検索するには、複数台のコンピュータによる類似度計算処理の並列化やHDD上でのデータ配置の工夫など大量データ処理に向いた基盤技術が必要となる。また紙媒体の医療カルテをスキャンした画像を対象に類似症例検索や統計処理など分析処理を行うためには、スキャン画像から検査結果などの数値データとその意味、固有表現とその属性値などコンピュータが処理可能な情報を抽出し、構造データ化する技術が必要となる。 For example, in order to search for similar images from a large amount of image data at high speed, basic technology suitable for large-scale data processing, such as parallel processing of similarity calculation processing by multiple computers and data arrangement on the HDD, is required. Become. Also, in order to perform analysis processing such as similar case search and statistical processing on images scanned from medical records on paper media, computers such as numerical data such as examination results and their meanings, specific expressions and their attribute values from scanned images are used. A technique for extracting information that can be processed and converting it into structural data is required.
　インターネットの世界では、大量のテキストデータを処理して有用な情報を抽出して利用するためのアプリケーションを構築する動きが広がっている。Googleは、検索結果を表形式で提示するGoogle Squaredを実験公開している(非特許文献1)。Google Squaredでは、通常のキーワード検索とは異なり、検索キーワードの下位に属する名称と属性を表形式で表示する。例えば、catというクエリに対して、American Shorthair、Persianなどの名称が、画像や説明と共に表示される。画像や説明は、インターネット上のリソースへのリンクとなっている。表示する名称や属性はユーザ自身で拡張できるインタフェースを備える点も特徴の一つとなっている。一方、IBMは「Watson」という質問応答(QA)システムを構築し、米国の人気クイズ番組に挑戦して最高金額を獲得した(非特許文献2)。このシステムは自然言語で表現されるバラエティーに富んだ複雑なクエリから回答を瞬時に求める
　これらの技術に共通するのは、異なるリソースからの情報を集約し、有用な知識を獲得して活用している点であり、大量の文書から如何に有用な情報を抜き出すかという「情報抽出」の技術が鍵である。特にテキスト解析の分野では、人が扱う言葉の揺らぎを処理するため、表現は異なるが同じ意味を持つ同義語をコンピュータで処理する技術の研究開発が活発である(非特許文献3)。他方、映像、画像、音声等の多様な形式の非構造データから構造化された情報を抽出するため、高速類似画像検索(非特許文献4)、音声検索(非特許文献5)などの研究開発も実施されている。 In the Internet world, there is a growing trend to build applications that process large amounts of text data to extract and use useful information. Google has published Google Squared, which presents search results in tabular form (Non-patent Document 1). Unlike normal keyword search, Google Squared displays names and attributes belonging to the lower level of the search keyword in a tabular format. For example, for a query “cat”, names such as American Shorthair and Persian are displayed together with images and descriptions. Images and descriptions are links to resources on the Internet. One of the features is that the name and attributes to be displayed have an interface that can be extended by the user. IBM, on the other hand, built a question-and-answer (QA) system called "Watson" and won the highest amount by challenging a popular quiz program in the US (Non-patent Document 2). This system instantly seeks answers from a variety of complex queries expressed in natural language. Common to these technologies is the aggregation of information from different resources to acquire and use useful knowledge. The key is “information extraction” technology, which is how to extract useful information from a large amount of documents. In particular, in the field of text analysis, research and development of technology for processing synonyms with different expressions but the same meaning with computers is actively performed in order to process fluctuations in words handled by humans (Non-patent Document 3). On the other hand, research and development such as high-speed similar image search (Non-Patent Document 4), voice search (Non-Patent Document 5), etc. to extract structured information from unstructured data in various formats such as video, images, and voices Has also been implemented.
　しかし、非構造データ解析においては、一種類のデータのみを対象に、一種類の処理方式で処理できるとは限らず、複数種類のデータの解析結果を互いに連携させて意味を捉えるマルチモーダルと呼ばれる処理の実現が必要である。例えば、人間が会話をしているときに、相手の言葉だけでなく、相手の表情やしぐさを見てそれらを合わせて脳で処理することで、相手が伝えたいことを理解するのと類似している。また、抽出したい情報に応じて、処理方式も異なり、また演算時間も異なる。したがって様々な非構造データを扱うデータ管理システムを実現するには、データ管理システムを利用するアプリケーションの要求と、対象とするデータに応じた処理方式の選択と実行、そしてデータから抽出した多様な意味を柔軟に管理する仕組みが必要となる。 However, in unstructured data analysis, it is not always possible to process only one type of data with one type of processing method, but it is called multimodal that captures meaning by linking the analysis results of multiple types of data to each other Realization of processing is necessary. For example, when a person is talking, it is similar to understanding what the other person wants to convey by looking at the other person's facial expressions and gestures and processing them in the brain. ing. Further, the processing method is different and the calculation time is different depending on the information to be extracted. Therefore, in order to realize a data management system that handles various unstructured data, the requirements of applications that use the data management system, selection and execution of processing methods according to the target data, and various meanings extracted from the data A mechanism to manage the system flexibly is necessary.
　例えば、会話文などユーザからの自然な入力を自動認識してデータベースから各種情報を取り出し、ユーザに提示するシステムが提案されている(特許文献１）。このシステムでは、２種類の異なる人工知能を搭載し、一方はユーザの入力を認識できる能力は限定的だが応答速度が速く、他方はユーザの入力を幅広くかつ高精度で認識可能であり、認識結果から学習しさらに精度を向上するが処理時間がかかる。そしてユーザから入力があった時、前者で処理を実行し、処理できなかった場合、後者を利用することで、認識率と処理時間の柔軟な制御を可能としている。 For example, a system that automatically recognizes natural input from a user such as a conversation sentence, extracts various information from a database, and presents it to the user has been proposed (Patent Document 1). This system is equipped with two different types of artificial intelligence, one with limited ability to recognize user input but fast response speed, and the other with wide and high accuracy recognition of user input. However, it takes more processing time to improve accuracy. When the input is made by the user, the former is executed. If the latter is not possible, the latter is used to allow flexible control of the recognition rate and the processing time.
　そして、既存データベースにおいて、様々な非構造データも統合的に扱えるように、様々なデータ解析機能を追加削除できる階層型データベースも提案されている(特許文献２)。 A hierarchical database that can add and delete various data analysis functions has also been proposed so that various unstructured data can be handled in an integrated manner in the existing database (Patent Document 2).
　また、データベースが複雑化するとアプリケーションからの要求時間内に応答することが困難になる可能性がある。これに対して、アプリケーションから要求時間を受付、データベース内で処理時間の見積もりを実行し、アプリケーションにとって意味のあるデータを時間内に返信できない場合はエラーを返すシステムの提案されている(特許文献３)。 Also, if the database becomes complicated, it may be difficult to respond within the time required by the application. On the other hand, a system has been proposed in which a request time is received from an application, a processing time is estimated in a database, and an error is returned when data meaningful to the application cannot be returned within the time (Patent Document 3). ).
特開2005-339237JP2005-339237 特開2003-99320JP2003-99320 特願平１１－２７８５９８Japanese Patent Application No.11-278598
　上記特許文献２では、呼び出すデータ解析機能が、インターネットなどの外部環境に存在した場合、解析データ及び結果をデータ解析機能と送受信する過程でデータが漏えいする可能性がある。また、データ解析機能の提供元が必ず信頼できるとも限らず、データ解析機能の提供元からデータ漏えいが発生する可能性も考えられる。 In Patent Document 2, when the data analysis function to be called exists in an external environment such as the Internet, there is a possibility of data leaking in the process of transmitting / receiving the analysis data and the result to / from the data analysis function. In addition, the data analysis function provider is not always reliable, and there is a possibility of data leakage from the data analysis function provider.
　上記特許文献３では、アプリケーションからの返信要求時間内に必要なデータ返信できない場合エラーを返すだけである。しかし、膨大なデータを管理する場合、アプリケーションからの返信要求時間が短いと、すべてのデータをそろえることが難しい状況が増え、エラーが頻発し、システムとして機能しなくなる可能性がある。 In the above-mentioned Patent Document 3, an error is only returned when necessary data cannot be returned within the response request time from the application. However, when managing enormous amounts of data, if the response request time from the application is short, the situation in which it is difficult to gather all the data increases, errors frequently occur, and the system may not function.
　上記特許文献２，３では、アプリケーションから未知のアクセス要求形式が来た場合に全く対応することができない。 In the above Patent Documents 2 and 3, when an unknown access request format comes from an application, it cannot be handled at all.
　そこで、アプリケーションがデータ解析システムへデータを要求するとき、要求秘匿性のレベルを通知することで、要求秘匿性が高い場合は、外部委託を実施しないことで情報漏えいを防ぐ技術を提供する。 Therefore, when the application requests data to the data analysis system, a technology for preventing information leakage by not performing outsourcing is provided by notifying the level of request confidentiality, and when the required confidentiality is high.
　また、アプリケーションがデータ解析システムへデータを要求するとき、データ探索制限時間を指定することで、アプリケーションが要求する時間内で最適なデータを取得する技術を提供する。 In addition, when an application requests data from the data analysis system, it provides a technology for acquiring optimal data within the time required by the application by specifying the data search time limit.
　また、アプリケーションがデータ解析システムへデータを要求するとき、得られる結果の正確性レベルを指定することで、データ管理システムが保持するデータで正確性が高いものを抽出してアプリケーションへ返信する技術を提供する。 In addition, when an application requests data from the data analysis system, a technology that extracts the highly accurate data held by the data management system and sends it back to the application by specifying the accuracy level of the results obtained. provide.
　また、アプリケーションがデータ解析システムへデータを要求するとき、キーワードを指定した場合に、キーワードの同義語も利用して関連するデータを抽出し返信する技術を提供する。 In addition, when an application requests data from the data analysis system, when a keyword is specified, a technique is provided that extracts and returns related data using synonyms of the keyword.
　また、アプリケーションがデータ解析システムへデータを要求するとき、要求パラメータが未知の値や構造を持つ場合、非構造データ解析処理機能を用いて情報を抽出し、それをデータ検索用の鍵として利用しデータ改正システム内のデータを検索し返信する技術手段を提供する。 In addition, when an application requests data from the data analysis system, if the requested parameter has an unknown value or structure, information is extracted using the unstructured data analysis processing function and used as a key for data retrieval. Provide technical means to search and return data in the data revision system.
　解決手段は特許請求の範囲に記載の通りであり、一例としてデータ解析システムは以下の構成を有する。 The solution is as described in the scope of claims, and as an example, the data analysis system has the following configuration.
　データ解析システムは、メタデータとデータ本体とを有する非構造データを格納する記憶装置と接続され、
　記憶装置から非構造データのメタデータを取得して当該メタデータから第一のメタ情報を作成するメタデータ抽出部と、
　記憶装置から非構造データのメタデータとデータ本体を取得して、当該メタデータと当該データ本体から非構造データの内容を表す意味情報を抽出して当該意味情報を有する第二のメタ情報を作成する情報抽出部と、
　第一のメタ情報と第二のメタ情報とを関連付けて格納するメタ情報リポジトリと、
　アプリケーションからの要求に応じて、メタ情報リポジトリに格納されているメタ情報を抽出するメタ情報管理部と、
　メタ情報管理部が抽出したメタ情報を要求によって指定される方法で加工して出力データを作成し、当該出力データを前記アプリケーションへ出力するデータ出力部とを有し、
　情報抽出部は、非構造データの意味情報を抽出した際に、当該意味情報が非構造データの内容を表す度合いを示すスコア値を計算し、このスコア値を前記第二のメタ情報に含ませる。 The data analysis system is connected to a storage device that stores unstructured data having metadata and a data body,
A metadata extraction unit that acquires metadata of unstructured data from the storage device and creates first metadata from the metadata;
Obtain metadata and data body of unstructured data from the storage device, extract semantic information representing the contents of the unstructured data from the metadata and the data body, and create second meta information having the semantic information An information extraction unit to
A meta information repository that stores the first meta information and the second meta information in association with each other;
A meta information management unit that extracts meta information stored in the meta information repository in response to a request from the application;
A meta information management unit extracts meta information by a method specified by a request to create output data, and has a data output unit that outputs the output data to the application,
When extracting the semantic information of the unstructured data, the information extraction unit calculates a score value indicating the degree that the semantic information represents the contents of the unstructured data, and includes the score value in the second meta information. .
　アプリケーションがデータ解析システムにアクセスするとき、取得する情報の正確度を指定することで、データ解析システム内のメタ情報リポジトリを探索するときスコア値が低い情報の探索を省略することで、全体の検索量を削減し、応答速度を向上できる。また、アプリケーションが要求の秘匿性レベルを指定することで、不用意な外部ネットワークへの情報解析を移譲を禁止することができ、情報漏えいを防ぐことができる。さらに、アプリケーションからの検索キーワードに関する同義語についてもメタ情報リポジトリから検索することで検索漏れを低減する。そして、アプリケーションからの要求が未知の形式でもそれに含まれるキーワードを抽出してメタ情報リポジトリを検索することが可能となる。 When the application accesses the data analysis system, specify the accuracy of the information to be acquired, and when searching the meta information repository in the data analysis system, search for information with a low score value is omitted, and the entire search The amount can be reduced and the response speed can be improved. Also, by specifying the confidentiality level of the request by the application, transfer of information analysis to an inadvertent external network can be prohibited, and information leakage can be prevented. Furthermore, search omissions are also reduced by searching the meta information repository for synonyms related to search keywords from applications. And even if the request from the application is unknown, it is possible to search the meta information repository by extracting the keywords included in the format.
システム構成の一例である。It is an example of a system configuration. メタ情報リポジトリに格納されるメタ情報の構造の一例である。It is an example of the structure of the meta information stored in a meta information repository. メタデータの構造の一例である。It is an example of the structure of metadata. メタ情報を構成するノードが持つ属性の一例である。It is an example of the attribute which the node which comprises meta information has. メタ情報を構成するエッジが持つ属性の一例である。It is an example of the attribute which the edge which comprises meta information has. アプリケーションがデータ解析システムへ発行するデータ出力要求で指定するパラメータの一例である。It is an example of a parameter specified by a data output request issued by an application to a data analysis system. アプリケーションからのアクセス要求を識別ルールの一例である。It is an example of an identification rule for an access request from an application. 生データを解析しメタ情報リポジトリへ登録するまでの流れの一例である。It is an example of a flow from analyzing raw data to registering it in a meta information repository. アプリケーションとデータ解析システムの連携動作の一例である。It is an example of the cooperation operation | movement of an application and a data analysis system. アクセス要求識別がアプリケーションからの要求を振り分ける判断の一例である。The access request identification is an example of determination for distributing requests from applications. アプリケーションのアクセス要求からメタ情報リポジトリアクセス情報を生成する流れの一例である。It is an example of the flow which produces | generates meta information repository access information from the access request of an application. アプリケーションの要求に応じてメタ情報を出力形式に加工する流れの一例である。It is an example of the flow which processes meta information into an output format according to the request | requirement of an application. アプリケーションが入力したメタ情報をメタ情報リポジトリへ登録する処理の流れの一例である。It is an example of the flow of a process which registers the meta information which the application input into a meta information repository. データストア内のメタデータが更新されたことを検出して、情報抽出を実行する処理の流れの一例である。It is an example of the flow of a process which detects that the metadata in a data store was updated and performs information extraction. 学習用サンプルデータを用いた情報抽出部の学習処理の流れの一例である。It is an example of the flow of the learning process of the information extraction part using the sample data for learning.
　図1はシステムの機能構成例を示す図である。
この図において、符号101はデータ解析システム、符号102は外部ネットワーク、符号103は外部情報抽出部、符号104はデータ保有システム、符号105は生データストア、符号106はアクセス要求識別部、符号107はメタ情報管理部、符号108はアプリケーション、符号110は出力抑制パターン判定部、符号111はデータ整形部、符号112はメタ情報リポジトリ、符号113は出力抑制ルール、符号114は情報抽出部、符号115は外部委託判定部、符号116は抽出情報関連付け部、符号117はデータ抽出システム、符号118は学習機構、符号119はデータストア、符号120は外部参照データ、符号121は抽出モデル、符号122はアクセス識別ルール、符号123はメタデータ抽出部、符号124はメタ情報、符号125はスキーマ定義、符号128は外部データ取り込み部、符号129は学習用サンプルデータ、符号130は学習制限ルール、符号133は生データ、符号134はデータ本体、符号135はメタデータ、符号139は情報抽出部利用ルール、符号142は外部抽出モデル、符号143はデータ更新監視部、符号144は認証処理部である。 FIG. 1 is a diagram illustrating a functional configuration example of a system.
In this figure, reference numeral 101 is a data analysis system, reference numeral 102 is an external network, reference numeral 103 is an external information extraction unit, reference numeral 104 is a data holding system, reference numeral 105 is a raw data store, reference numeral 106 is an access request identification unit, reference numeral 107 is Meta information management unit, reference numeral 108 is an application, reference numeral 110 is an output suppression pattern determination unit, reference numeral 111 is a data shaping unit, reference numeral 112 is a meta information repository, reference numeral 113 is an output suppression rule, reference numeral 114 is an information extraction unit, reference numeral 115 is Outsourcing determination unit, 116 is an extraction information association unit, 117 is a data extraction system, 118 is a learning mechanism, 119 is a data store, 120 is external reference data, 121 is an extraction model, 122 is an access identification Rule, reference numeral 123 is a metadata extraction unit, reference numeral 124 is meta information, reference numeral 125 is a schema definition, reference numeral 128 is an external data capturing unit, reference numeral 129 is sample data for learning Reference numeral 130 is a learning restriction rule, reference numeral 133 is raw data, reference numeral 134 is a data body, reference numeral 135 is metadata, reference numeral 139 is an information extraction unit usage rule, reference numeral 142 is an external extraction model, reference numeral 143 is a data update monitoring unit, reference numeral 144 is an authentication processing unit.
　データ解析システム101はデータストア119上のデータを取得、解析して生成したメタ情報124を管理し、アプリケーション108の要求に応じてこのメタ情報を加工して出力するシステムである。データ解析システム１０１の実体はメモリ、処理装置（ＣＰＵ）、外部ネットワーク１０２に接続するネットワークインタフェース、データストア１１９が実装されている記憶装置にアクセスするためのインタフェース、およびメタ情報リポジトリ１１２や情報抽出部利用ルール１３９、抽出モデル１２１、学習用サンプルデータ１２９、学習制限ルール１３０が実装・格納される記憶装置を有する計算機であり、以下に説明するデータ解析システム１０１を構成する各種処理部や機能は、メモリ上のプログラムがＣＰＵによって実行されることにより実現される。データ解析システム１０１は、アクセス要求識別部106、メタ情報管理部107、出力抑制パターン判定部110、データ整形部111、メタ情報リポジトリ112、情報抽出部114、外部委託判定部115、抽出情報関連付け部116、メタデータ抽出部123、外部データ取り込み部128、学習用サンプルデータ129、学習制限ルール130、情報抽出部利用ルール139、データ更新監視部143、認証処理部144を有している。 The data analysis system 101 is a system that manages meta information 124 generated by acquiring, analyzing, and analyzing data on the data store 119, and processing and outputting the meta information in response to a request from the application 108. The substance of the data analysis system 101 includes a memory, a processing device (CPU), a network interface connected to the external network 102, an interface for accessing a storage device in which the data store 119 is mounted, a meta information repository 112, and an information extraction unit. A computer having a storage device in which the usage rule 139, the extraction model 121, the learning sample data 129, and the learning restriction rule 130 are mounted and stored, and various processing units and functions constituting the data analysis system 101 described below are as follows. This is realized by a program on the memory being executed by the CPU. The data analysis system 101 includes an access request identification unit 106, a meta information management unit 107, an output suppression pattern determination unit 110, a data shaping unit 111, a meta information repository 112, an information extraction unit 114, an outsourcing determination unit 115, an extraction information association unit 116, a metadata extraction unit 123, an external data capturing unit 128, learning sample data 129, a learning restriction rule 130, an information extraction unit usage rule 139, a data update monitoring unit 143, and an authentication processing unit 144.
　外部ネットワーク102はインターネット等の外部ネットワークであり、データ解析システムが情報解析時にアクセスする情報抽出機能を搭載したサーバなどが配置されている。このサーバには、外部情報抽出部103、外部参照データ120、外部抽出モデル142が実装されている。 The external network 102 is an external network such as the Internet, and a server equipped with an information extraction function that is accessed by the data analysis system during information analysis is arranged. In this server, an external information extraction unit 103, external reference data 120, and an external extraction model 142 are mounted.
　外部情報抽出部103は外部ネットワーク102に存在し、データ解析システムが有する情報抽出部114と同様にデータストア１１９に格納されたメタデータやデータ本体を解析してメタ情報１２４を生成する。実体は例えば、データセンタ等に設置されたサーバ等に格納された情報抽出処理プログラムがサーバの処理装置によって実行されて実現されたものである。 The external information extraction unit 103 exists in the external network 102 and analyzes the metadata and the data body stored in the data store 119 in the same manner as the information extraction unit 114 of the data analysis system, and generates meta information 124. The entity is realized, for example, by an information extraction processing program stored in a server or the like installed in a data center or the like being executed by a processing device of the server.
　データ保有システム104はデータ抽出システムがデータの抽出対象とする生データ（非構造データ）を持つシステムである。例えば、企業内の情報共有システム、ファイルサーバ等であり、生データストア105を有しており、データ抽出システム117と接続されている。 The data holding system 104 is a system having raw data (unstructured data) that is extracted by the data extraction system. For example, an information sharing system in a company, a file server, etc., have a raw data store 105, and are connected to a data extraction system 117.
　生データストア105はデータ抽出システムがデータの抽出対象とする生データ１３３を格納するデータ保存装置であり、例えばファイルストレージ、サーバ内のハードディスク等の記憶装置である。 The raw data store 105 is a data storage device that stores the raw data 133 to be extracted by the data extraction system, and is a storage device such as a file storage or a hard disk in the server.
　アクセス要求識別部106はアプリケーションからの要求内容を、アクセス識別ルールに従い解析して処理方法を判断して、認証処理部、メタデータ抽出部、外部委託判定部、メタ情報管理部へ処理を依頼する機能であり、メタ情報管理部107、アプリケーション108と接続されている。 The access request identifying unit 106 analyzes the request content from the application according to the access identification rule to determine the processing method, and requests the authentication processing unit, metadata extraction unit, outsourcing determination unit, and meta information management unit to perform processing. It is a function and is connected to the meta information management unit 107 and the application 108.
　メタ情報管理部107はアクセス要求識別部あるいは、抽出情報関連付け部から受け取ったデータを用いて、メタ情報リポジトリを検索して、データ整形部へ出力用データ加工を依頼する機能である。メタ情報リポジトリの管理も実行する。そして、アクセス要求識別部106、データ整形部111と接続されている。 The meta information management unit 107 is a function that searches the meta information repository using data received from the access request identification unit or the extracted information association unit and requests the data shaping unit to process data for output. It also manages the meta information repository. The access request identifying unit 106 and the data shaping unit 111 are connected.
　アプリケーション108はデータ解析システム101を使用するシステムであり、例えば複数の医療情報を統合して表示するシステム、自然言語を用いたコンテンツ検索システム、等が考えられる。実体は、計算機上の処理装置がメモリ上のアプリケーションプログラムを実行することによって実現される。アプリケーションは、アクセス要求識別部106、出力抑制パターン判定部110と接続されている。 The application 108 is a system that uses the data analysis system 101. For example, a system that integrates and displays a plurality of medical information, a content search system that uses a natural language, and the like are conceivable. The entity is realized by a processing device on a computer executing an application program on a memory. The application is connected to the access request identifying unit 106 and the output suppression pattern determining unit 110.
　出力抑制パターン判定部110は出力抑制ルールの内容に従って、データ整形部が作成した整形済みデータの内容をチェックし、出力可能か判断する。そして、データ整形部111、アプリケーション108と接続されている。 The output suppression pattern determination unit 110 checks the content of the shaped data created by the data shaping unit according to the content of the output suppression rule, and determines whether output is possible. The data shaping unit 111 and the application 108 are connected.
　データ整形部111はメタ情報リポジトリから取得したデータ（メタ情報）をスキーマ定義にしたがって加工してアプリケーションへ出力する機能であり、メタ情報管理部107、出力抑制パターン判定部110と接続されている。尚、出力形式に応じて、データストア119からデータ本体134を取得して統合する場合もあるし、この処理をアプリケーション108が実施する場合も考えられる。 The data shaping unit 111 has a function of processing data (meta information) acquired from the meta information repository according to the schema definition and outputting the processed data to the application, and is connected to the meta information management unit 107 and the output suppression pattern determination unit 110. Depending on the output format, the data body 134 may be acquired from the data store 119 and integrated, or the application 108 may execute this processing.
　メタ情報リポジトリ112はメタ情報124の保存と、各種モジュールからの要求に応じて必要なメタ情報を検索して出力する機能を提供する。尚、メタ情報の保存形式として、例えば、RDF（Resource Description Framework;リソース記述フレームワーク）、グラフ構造、木構造等の柔軟性がある構造が考えられる。ここで、ＲＤＦとは、主語-述語-目的語の形式であらゆるデータを表現し保存するデータ形式である。また、多様なデータ形式に対応可能なため、アクセス要求識別部106、出力抑制パターン判定部110、データ整形部111が、処理を実行する際に参照する情報も格納する方法も考えられる。一方で、処理性能を考慮し、これら３機能別々に個別のデータ保存部を設ける構成も考えられる。そして、メタ情報リポジトリ112には、出力抑制ルール113、アクセス識別ルール122、メタ情報124、スキーマ定義125が含まれる。 The meta information repository 112 provides functions for storing meta information 124 and searching for and outputting necessary meta information in response to requests from various modules. As a storage format of the meta information, for example, flexible structures such as RDF (Resource Description Framework), graph structure, and tree structure are conceivable. Here, RDF is a data format that represents and stores all data in the form of subject-predicate-object. Further, since it can support various data formats, a method is also conceivable in which the access request identifying unit 106, the output suppression pattern determining unit 110, and the data shaping unit 111 store information that is referred to when executing processing. On the other hand, in consideration of processing performance, a configuration in which individual data storage units are provided separately for these three functions is also conceivable. The meta information repository 112 includes an output suppression rule 113, an access identification rule 122, meta information 124, and a schema definition 125.
　出力抑制ルール113は出力抑制に利用される判断ルール、たとえばアクセス権限毎に、出力が許可されない一連のキーワードの組み合わせを管理した表などが登録されている。ただし、出力制御ルール113の実体はメタ情報リポジトリ112の管理構造に応じた形に変換されて格納される。 The output suppression rule 113 registers a judgment rule used for output suppression, for example, a table that manages a combination of a series of keywords that are not permitted to be output for each access authority. However, the substance of the output control rule 113 is converted into a form corresponding to the management structure of the meta information repository 112 and stored.
　情報抽出部114は外部委託判定部経由で取得したデータ本体から、意味を抽出しメタ情報を作り出す機能で、例えば、ひまわりの画像を入れると、それが"ひまわり"であり"3本"画像に含まれるといった情報を作り出し、メタ情報に変換する処理を実行する機能であり、学習機構118、抽出モデル121から構成されている。 The information extraction unit 114 is a function that extracts the meaning from the data body acquired via the outsourced determination unit and creates meta information. For example, when a sunflower image is inserted, it is a “sunflower” and becomes a “three” image. This is a function for generating information such as being included and executing processing for conversion to meta information, and is composed of a learning mechanism 118 and an extraction model 121.
　外部委託判定部115はデータ保存システムから抽出されてデータストアに格納されたデータや、アクセス要求識別から取得したアクセス要求に添付されたデータの、メタ情報の抽出を外部ネットワーク上の情報抽出部に行わせるか、データ解析システム内部の情報抽出部に行わせるか判定する。 The outsourcing determination unit 115 extracts the meta information of the data extracted from the data storage system and stored in the data store or the data attached to the access request acquired from the access request identification to the information extraction unit on the external network. It is determined whether it is to be performed or to be performed by an information extraction unit in the data analysis system.
　抽出情報関連付け部116はメタデータ、外部委託判定部、情報抽出部から得たメタ情報を関係付けて、メタ情報リポジトリアクセスへ格納する。尚、メタデータとメタ情報を関係付けるときに、メタ情報リポジトリを参照する場合もある。 The extracted information association unit 116 associates the metadata, the meta information obtained from the outsourcing determination unit, and the information extraction unit, and stores them in the meta information repository access. Note that when associating metadata and meta information, the meta information repository may be referred to.
　データ抽出システム117は単一のあるいは複数のデータ保有システムから生データを抽出しデータストアへ保存する処理を実行するシステムであり、データ保有システム104と接続されている。 The data extraction system 117 is a system that executes processing for extracting raw data from a single data storage system or a plurality of data storage systems and storing it in a data store, and is connected to the data storage system 104.
　学習機構118は外部データ取り込み部が提供するデータを基に学習し、抽出モデルを作成する機能である。 The learning mechanism 118 is a function for learning based on data provided by the external data capturing unit and creating an extraction model.
　データストア119はデータ抽出システムが抽出したデータを、保持する記憶装置であり、ファイルストレージ等で構成される。尚、データ解析システムの仕様に応じて、メタデータとデータ本体に分離して格納する場合もあり得る。 The data store 119 is a storage device that holds the data extracted by the data extraction system, and includes a file storage or the like. Depending on the specifications of the data analysis system, the metadata and the data body may be stored separately.
　外部参照データ120は情報抽出部が必要に応じて参照する外部ネットワーク上の記憶装置に存在するデータである。 External reference data 120 is data existing in a storage device on an external network that is referred to by the information extraction unit as necessary.
　抽出モデル121は情報抽出部に入力された情報から意味の抽出に用いるとき判断基準として利用されるデータ群であり、学習機構により作成されるである。抽出モデルに登録されている判断基準を用いて、情報抽出部は入力された情報（データ本体）からメタ情報を作成する。尚、複数の抽出モデル121を準備しておき、アプリケーションの要求内容や、学習失敗時のバックアップなどの用途で切り替えて利用することも考えられる。 The extraction model 121 is a data group used as a criterion when used for extracting meaning from information input to the information extraction unit, and is created by a learning mechanism. Using the criteria registered in the extraction model, the information extraction unit creates meta information from the input information (data body). It is also conceivable that a plurality of extraction models 121 are prepared and switched for use such as request contents of an application or backup when learning fails.
　アクセス識別ルール122はアクセス要求識別部が判断に利用するルール群である。 The access identification rule 122 is a group of rules used for determination by the access request identification unit.
　メタデータ抽出部123はデータストア、あるいはアクセス要求識別部から取得したデータからメタデータを抽出する機能である。 The metadata extraction unit 123 has a function of extracting metadata from data acquired from the data store or the access request identification unit.
　メタ情報124は情報抽出部により抽出された情報（メタ情報）である。 Meta information 124 is information (meta information) extracted by the information extraction unit.
　スキーマ定義125はデータ整形部がアプリケーションの要求に応じた形式へデータ（メタ情報）を加工する時に参照するルール群である。例えば、メタ情報を表形式へ変換して出力する場合、メタ情報のどの属性を表の列にするか等が記載されている。 Schema definition 125 is a group of rules that the data shaping unit refers to when processing data (meta information) into a format according to the request of the application. For example, when meta information is converted into a table format and output, which attribute of the meta information is used as a table column is described.
　外部データ取り込み部128は外部ネットワーク上の抽出モデルとデータを必要に応じて取り込み、学習用サンプルデータとの比較処理を実施する機能であり、学習用サンプルデータ129と接続されている。学習用サンプルデータ129は情報抽出部が学習に利用するサンプルデータであり、例えば、品詞の解説が付いた文書データ、サンプル画像、サンプル音声データ等であり、外部データ取り込み部128と接続されている。 The external data capturing unit 128 has a function of capturing an extraction model and data on an external network as necessary, and performing comparison processing with the sample data for learning, and is connected to the sample data 129 for learning. The learning sample data 129 is sample data used by the information extraction unit for learning. For example, the sample data 129 includes document data, sample images, sample audio data, etc. with parts of speech, and is connected to the external data capturing unit 128. .
　学習制限ルール130は各情報抽出部の学習機構を制御するルールである。各種学習用データを用いた学習タイミングを制御する。例えば、データストアからのデータの情報抽出処理による結果を学習するか否か、メタ情報リポジトリ内のデータを学習に利用していた場合、メタ情報リポジトリが更新されたらその結果を自動的に学習するか否か、等のルールが記載される。 The learning restriction rule 130 is a rule that controls the learning mechanism of each information extraction unit. Control learning timing using various learning data. For example, whether or not to learn the result of the information extraction process of data from the data store, if the data in the meta information repository is used for learning, automatically learn the result when the meta information repository is updated Whether or not is described.
　生データ133はデータ抽出システムが抽出対象とするデータ本体であり、データ本体134、メタデータ135から構成されている。 Raw data 133 is a data body to be extracted by the data extraction system, and includes a data body 134 and metadata 135.
　データ本体134は画像、音声、文書等の実体である。 The data body 134 is an entity such as an image, sound, or document.
　メタデータ135は解析対象データ毎に存在し、解析対象データが何か説明する情報である。例えば、データの格納先情報、最終更新日時情報、最終アクセス日時情報、アクセス権限、ファイル所有者等をしめすデータである。 Metadata 135 exists for each analysis target data, and is information that explains what the analysis target data is. For example, data indicating data storage location information, last update date / time information, last access date / time information, access authority, file owner, and the like.
　情報抽出部利用ルール139は外部委託判定部が情報抽出委託先を決定するために利用する判断基準群である。 The information extraction unit usage rule 139 is a group of criteria used by the outsourcing determination unit to determine an information extraction outsourcing destination.
　外部抽出モデル142は学習機構により作成されるメタデータの抽出に用いるモデルデータであり、外部ネットワーク上に存在するデータセンタ等に配置されたものである。 The external extraction model 142 is model data used for extracting metadata created by the learning mechanism, and is arranged in a data center or the like existing on an external network.
　データ更新監視部143はデータストア上のデータの更新状態を監視し、更新を検出すると、メタデータ抽出部と外部委託判定部へ通知する機能である。 The data update monitoring unit 143 has a function of monitoring the update state of data on the data store and notifying the metadata extraction unit and the outsourcing determination unit when an update is detected.
　認証処理部144はアプリケーションからの認証情報をアクセス要求識別部経由で受け取り、メタ情報リポジトリ内に格納した認証情報と照合して、アプリケーションのアクセスの正当性を確認する機能である。 The authentication processing unit 144 has a function of receiving authentication information from the application via the access request identifying unit and checking the validity of the application access by collating with the authentication information stored in the meta information repository.
　図2はメタ情報リポジトリ112に格納されるメタ情報124の構造の一例を示す図である。
メタ情報は、例えば、ノードとそれを結ぶエッジで構成されたグラフ構造、木構造等で表現される。これにより、様々な事物とその間の関係を表現することが可能である。本例では、ノードとエッジの一例を示す。尚、ノードとエッジが持つデータの一例は後述する。
この図において、符号201はノード１、符号202はエッジである。 FIG. 2 is a diagram showing an example of the structure of the meta information 124 stored in the meta information repository 112. As shown in FIG.
The meta information is expressed by, for example, a graph structure or a tree structure composed of nodes and edges connecting the nodes. This makes it possible to express various things and the relationships between them. In this example, an example of a node and an edge is shown. An example of data held by nodes and edges will be described later.
In this figure, reference numeral 201 is node 1 and reference numeral 202 is an edge.
　ノード１201はノード自体を特徴づける複数の属性情報と、このノードと関係があるエッジ一覧を持つ。エッジ202はノード間の関係を表す。ノード間がどのような関係を持つかを特徴づける属性一覧を持つ。 The node 1201 has a plurality of attribute information characterizing the node itself and an edge list related to the node. Edge 202 represents the relationship between nodes. It has an attribute list that characterizes the relationship between nodes.
　図3はメタデータの構造を示す図である。 Figure 3 shows the metadata structure.
　メタデータ135は生データのデータ本体（ファイル等）の概要を記述した属性値の集合である。このパラメータの一覧を示す。尚、本例に含まれるすべての属性が常に必要ではなく、状況に応じて組み合わせてメタデータを構成する。 Metadata 135 is a set of attribute values describing the outline of the data body (file etc.) of raw data. A list of these parameters is shown. Note that not all attributes included in this example are always necessary, and the metadata is configured by combining according to the situation.
　「属性名」301はメタデータに含まれる属性値の一般名称である。であり、行303は「属性名」が「日時」であるものに関する記述であり、行304は「属性名」が「サイズ」であるものに関する記述であり、行305は「属性名」が「取得元の場所」であるものに関する記述であり、行306は「属性名」が「種別」であるものに関する記述であり、行307は「属性名」が「アクセス権限」であるものに関する記述である。 “Attribute name” 301 is a general name of the attribute value included in the metadata. , Line 303 is a description regarding what “attribute name” is “date and time”, line 304 is a description regarding what “attribute name” is “size”, and line 305 indicates that “attribute name” is “ Line 306 is a description about what “attribute name” is “type”, and line 307 is a description about what “attribute name” is “access authority”. is there.
　「属性値の例」302は属性値の一例である。例えば、「属性名」が「日時」であるものの「属性値の例」は例えば「2011/7/7」でありデータ本体とメタデータの作成日時、または更新日時を示している。「属性名」が「サイズ」であるものの「属性値の例」は例えば「115MByte」であり生データのサイズを示す。「属性名」が「取得元の場所」であるものの「属性値の例」は例えば「NAS1/doc」であり生データの取得元（格納場所）を特定するための情報を示す。「属性名」が「種別」であるものの「属性値の例」は例えば「文章、映像、画像、等」であり生データのタイプを示す。「属性名」が「アクセス権限」であるものの「属性値の例」はアクセス可能なユーザID一覧を示す。 “Example of attribute value” 302 is an example of an attribute value. For example, although “attribute name” is “date and time”, “example of attribute value” is “2011/7/7”, for example, indicating the creation date or update date and time of the data body and metadata. Although “attribute name” is “size”, “example of attribute value” is, for example, “115 MByte”, indicating the size of raw data. Although “attribute name” is “location of acquisition source”, “example of attribute value” is “NAS1 / doc”, for example, and indicates information for specifying the acquisition source (storage location) of raw data. Although “attribute name” is “type”, “example of attribute value” is, for example, “sentence, video, image, etc.” and indicates the type of raw data. Although “attribute name” is “access authority”, “example of attribute value” indicates a list of accessible user IDs.
　図4はメタ情報のノードを示す図である。 Fig. 4 is a diagram showing nodes of meta information.
　これは、メタ情報を構成するノードが持つ属性の一例を示す。尚、ノードの利用目的に応じて属性を追加削除してもよい。 This is an example of the attributes of the nodes that make up the meta information. Note that attributes may be added and deleted according to the purpose of use of the node.
　「属性名」401はノードに含まれる属性値の一般名称である。であり、行403は「属性名」が「日時」であるものに関する記述であり、行404は「属性名」が「ノード種別」であるものに関する記述であり、行405は「属性名」が「接続エッジ一覧」であるものに関する記述であり、行406は「属性名」が「メタデータ」であるものに関する記述であり、行407は「属性名」が「アクセス権限」であるものに関する記述であり、行408は「属性名」が「抽出情報」であるものに関する記述であり、行409は「属性名」が「スコア情報」であるものに関する記述であり、行410は「属性名」が「利用回数」であるものに関する記述である。 “Attribute name” 401 is a general name of the attribute value included in the node. Line 403 is a description related to what “attribute name” is “date and time”, line 404 is a description related to what “attribute name” is “node type”, and line 405 is a description related to “attribute name”. Line 406 is a description related to “connection edge list”, line 406 is a description related to “attribute name” being “metadata”, and line 407 is a description related to “attribute name” being “access authority” Line 408 is a description regarding an item whose “attribute name” is “extraction information”, line 409 is a description regarding an item whose “attribute name” is “score information”, and line 410 is an attribute name Is a description of what is “number of uses”.
　「属性値の例」402は属性値の一例である。「属性名」が「日時」であるものの「属性値の例」は例えば「2011/7/7」でありメタ情報のノードの作成日時または更新日時を示す。「属性名」が「ノード種別」であるものの「属性値の例」は例えば固有名詞、キーワード、ファイル情報等であり生データのタイプを示す。「属性名」が「接続エッジ一覧」であるものの「属性値の例」は例えば「エッジ1、エッジ５、・・・」であり当該ノードに接続されるエッジの識別情報を示す。「属性名」が「メタデータ」であるものの「属性値の例」は例えば「AAAメタデータ」であり当該メタ情報の抽出元となった生データのメタデータの識別情報とメタデータ自体を示す。「属性名」が「アクセス権限」であるものの「属性値の例」は例えば「アクセス可能なユーザID一覧」であり当該ノードにアクセス可能なユーザのＩＤまたはユーザグループ識別名を示す。「属性名」が「抽出情報」であるものの「属性値の例」は例えば抽出したキーワード、値、等生データから抽出された情報を示す。「属性名」が「スコア情報」であるものの「属性値の例」は例えば「4.3」などの数値であり所定の基準に基づき算出された抽出情報４０８として登録されている情報の確からしさを表す情報である。この「スコア値」の算出方法として、例えば情報抽出部１１４が画像の構造を解析して、それを意味するキーワードを属性値として生成する機能を提供するとした場合、この情報抽出部１１４のモデル１２１に記録された画像パターンデータ群と、解析対象データがどれだけ類似しているかの類似性をスコア値として示す場合が考えられる。音声、センサ情報などの解析についても同等な方法が考えられる。「属性名」が「利用回数」であるものの「属性値の例」は例えば「15」などの数値であり、当該ノードの情報がアプリケーションに出力された回数を示す情報である。 “Example of attribute value” 402 is an example of an attribute value. Although the “attribute name” is “date and time”, the “example of attribute value” is, for example, “2011/7/7” and indicates the creation date or update date of the node of the meta information. Although “attribute name” is “node type”, “example of attribute value” is, for example, a proper noun, a keyword, file information, etc., and indicates the type of raw data. Although “attribute name” is “connection edge list”, “example of attribute value” is “edge 1, edge 5,...”, For example, and indicates identification information of an edge connected to the node. Although the “attribute name” is “metadata”, the “example of attribute value” is “AAA metadata”, for example, indicating the metadata identification information of the raw data from which the metadata information is extracted and the metadata itself . Although “attribute name” is “access authority”, “example of attribute value” is, for example, “accessible user ID list”, and indicates the ID or user group identification name of a user who can access the node. Although “attribute name” is “extraction information”, “example of attribute value” indicates, for example, information extracted from raw data such as extracted keywords, values, and the like. Although “attribute name” is “score information”, “example of attribute value” is a numerical value such as “4.3”, for example, and represents the likelihood of information registered as extracted information 408 calculated based on a predetermined standard. Information. As a method of calculating the “score value”, for example, when the information extraction unit 114 provides a function of analyzing the structure of an image and generating a keyword that represents the attribute value as an attribute value, the model 121 of the information extraction unit 114 is provided. It is conceivable that the similarity between how much the image pattern data group recorded in the image data and the analysis target data are similar is indicated as a score value. An equivalent method can be considered for analysis of voice, sensor information, and the like. Although “attribute name” is “use count”, “example of attribute value” is a numerical value such as “15”, for example, and is information indicating the number of times the information of the node is output to the application.
　図5はメタ情報を構成するエッジが持つ属性一覧を示す図である。尚、エッジの利用目的に応じて属性を追加削除してもよい。
「属性名」501はエッジに含まれる属性値の一般名称であり、行503は「属性名」が「日時」であるものに関する記述であり、行504は「属性名」が「エッジ種別」であるものに関する記述であり、行505は「属性名」が「接続元ノード」であるものに関する記述であり、行506は「属性名」が「接続先ノード」であるものに関する記述であり、行507は「属性名」が「アクセス権限」であるものに関する記述であり、行508は「属性名」が「抽出情報」であるものに関する記述であり、行509は「属性名」が「スコア情報」であるものに関する記述であり、行510は「属性名」が「利用回数」であるものに関する記述である。 FIG. 5 is a diagram showing a list of attributes held by edges constituting the meta information. The attribute may be added or deleted according to the purpose of use of the edge.
“Attribute name” 501 is a general name of the attribute value included in the edge, line 503 is a description regarding “attribute name” being “date and time”, and line 504 is “attribute name” is “edge type”. Line 505 is a description regarding an item whose “attribute name” is “connection source node”, and line 506 is a description regarding an item whose “attribute name” is “connection destination node”. 507 is a description regarding the “attribute name” being “access authority”, line 508 is a description regarding the “attribute name” being “extraction information”, and line 509 is the description regarding “attribute name” being “score information” ”And a line 510 is a description about the item whose“ attribute name ”is“ number of times of use ”.
　「属性値の例」502は属性値の一例が記載される。例えば、「属性名」が「日時」であるものの「属性値の例」は例えば「2011/7/7」でありエッジが生成及び更新された日時を示す。「属性名」が「エッジ種別」であるものの「属性値の例」は例えば親子関係、類似関係、同義関係、等であり、当該エッジで接続される接続元ノードと接続先ノードの間の関係を示す。「属性名」が「接続元ノード」であるものの「属性値の例」は例えば「ノード1」であり当該エッジの接続元ノードの識別情報であり、「属性名」が「接続先ノード」であるものの「属性値の例」は例えば「ノード5」であり当該エッジの接続先ノードの識別情報である。「属性名」が「アクセス権限」であるものの「属性値の例」は当該エッジにアクセス可能なユーザのID一覧である。「属性名」が「抽出情報」であるものの「属性値の例」は例えば抽出したキーワード、値、等であり生データから抽出された情報を示す。「属性名」が「スコア情報」であるものの「属性値の例」は例えば「7.3」などの数値であり所定の基準に基づき算出された抽出情報５０８として登録されている情報の確からしさを表す情報である。尚、算出方法例はノードにおける「スコア値」と同等である。「属性名」が「利用回数」であるものの「属性値の例」は例えば「15」などの数値であり、当該エッジの情報がアプリケーションへ出力された回数を示す情報である。 “An example of an attribute value” 502 describes an example of an attribute value. For example, although “attribute name” is “date and time”, “example of attribute value” is “2011/7/7” and indicates the date and time when the edge was generated and updated. Although “attribute name” is “edge type”, “example of attribute value” is, for example, a parent-child relationship, a similarity relationship, a synonym relationship, etc., and a relationship between a connection source node and a connection destination node connected by the edge Indicates. Although the “attribute name” is “connection source node”, the “attribute value example” is, for example, “node 1”, which is identification information of the connection source node of the edge, and the “attribute name” is “connection destination node”. For example, “example of attribute value” is “node 5”, which is identification information of a connection destination node of the edge. Although “attribute name” is “access authority”, “example of attribute value” is a list of IDs of users who can access the edge. Although “attribute name” is “extracted information”, “example of attribute value” indicates, for example, extracted keywords, values, and the like and extracted from raw data. Although “attribute name” is “score information”, “example of attribute value” is a numerical value such as “7.3”, for example, and represents the likelihood of information registered as extracted information 508 calculated based on a predetermined standard. Information. An example of the calculation method is equivalent to the “score value” at the node. Although “attribute name” is “usage count”, “example of attribute value” is a numerical value such as “15”, for example, and is information indicating the number of times the information on the edge is output to the application.
　図6はアプリケーションがデータ解析システムへ発行する要求の一例を示す図である。図６に示すのは、アプリケーションがデータ解析システムへ発行するデータ出力要求で指定するパラメータの一例である。尚、要求内容とアプリケーションの種別によっては、図６に示すすべてのパラメータを指定する必要はない。 Fig. 6 is a diagram showing an example of a request issued by the application to the data analysis system. FIG. 6 shows an example of parameters specified by the data output request issued by the application to the data analysis system. Note that it is not necessary to specify all the parameters shown in FIG. 6 depending on the request contents and the type of application.
　「パラメータ名」601はパラメータの名称である。行603は「パラメータ名」が「要求ID」であるものに関する記述であり、行604は「パラメータ名」が「情報登録許可フラグ」であるものに関する記述であり、行605は「パラメータ名」が「データ探索時間制限」であるものに関する記述であり、行606は「パラメータ名」が「要求秘匿性レベル」であるものに関する記述であり、行607は「パラメータ名」が「正確性レベル」であるものに関する記述であり、行608は「パラメータ名」が「認証情報」であるものに関する記述であり、行609は「パラメータ名」が「キーワード」であるものに関する記述であり、行610は「パラメータ名」が「同義語検索許可フラグ」であるものに関する記述であり、行611は「パラメータ名」が「メタ情報」であるものに関する記述であり、行612は「パラメータ名」が「要求クエリ」であるものに関する記述であり、行613は「パラメータ名」が「出力形式」であるものに関する記述であり、行614は「パラメータ名」が「キャッシュフラグ」であるものに関する記述であり、行615は「パラメータ名」が「先読みフラグ」であるものに関する記述であり、行616は「パラメータ名」が「ファイル」であるものに関する記述である。 “Parameter name” 601 is the name of the parameter. Line 603 is a description regarding what “parameter name” is “request ID”, line 604 is a description regarding what “parameter name” is “information registration permission flag”, and line 605 indicates “parameter name”. Line 606 is a description regarding what “parameter name” is “request confidentiality level”, and line 607 is a description regarding “parameter name” is “accuracy level”. Line 608 is a description regarding what “parameter name” is “authentication information”, line 609 is a description regarding what “parameter name” is “keyword”, and line 610 indicates “ The description of the parameter name is “synonym search permission flag”, line 611 is the description of the parameter name “meta information”, and line 612 is the parameter name “required”. Is a description about what is "query", line 613 is a description about what "parameter name" is "output format", and line 614 is a description about what "parameter name" is "cache flag", Line 615 is a description regarding the “parameter name” being “prefetch flag”, and line 616 is a description regarding the “parameter name” being “file”.
　「パラメータ値の例」602はパラメータ値の一例である。「パラメータ名」が「要求ID」であるものの「パラメータ値の例」は例えば「111」であり、当該要求を一意に識別するための情報である。「パラメータ名」が「情報登録許可フラグ」であるものの「パラメータ値の例」は「許可」または「不許可」であり「許可」の場合は本要求の結果利用されたメタ情報のアクセス回数の更新と、新たなメタ情報が生成された場合はそれをメタ情報リポジトリへ登録することを許可する。「不許可」の場合は前述の処理を実施しないことを示す。「パラメータ名」が「データ探索時間制限」であるものの「パラメータ値の例」は例えば「100ms」などの時間でありアプリケーションがデータ検索時間として許容している時間が指定されている。「パラメータ名」が「要求秘匿性レベル」であるものの「パラメータ値の例」は例えば「5」など要求の秘匿性を示す数値である。「パラメータ名」が「正確性レベル」であるものの「パラメータ値の例」は例えば「4」などアプリケーションが許容するメタ情報の正確性のレベルを示す数値である。この正確性の値として、例えばメタ情報が持つ「スコア値」の大きさを用いる場合が考えられる。「パラメータ名」が「認証情報」であるものの「パラメータ値の例」はアプリケーション利用者の認証情報であり、「パラメータ名」が「キーワード」であるものの「パラメータ値の例」は例えば「”アスピリン”」などアプリケーションが指定する検索キーワードである。「パラメータ名」が「同義語検索許可フラグ」であるものの「パラメータ値の例」は「許可」または「不許可」であり、「許可」の場合にはキーワードの同義語についても検索することを要求しており、「不許可」の場合にはキーワードの同義語は検索しないことを要求している。「パラメータ名」が「メタ情報」であるものの「パラメータ値の例」は例えば「アスピリンの成分を表すメタ情報構造データ」であり、取得すべきメタ情報の種類を特定するための情報である。「パラメータ名」が「要求クエリ」であるものの「パラメータ値の例」は例えば「アスピリンの成分と処方箋一覧を取得するSQL、SPARQL、等」であり、アプリケーションが要求しているクエリ処理を示す。「パラメータ名」が「出力形式」であるものの「パラメータ値の例」は表形式、リスト形式、等アプリケーションが指定する出力形式を示す。「パラメータ名」が「キャッシュフラグ」であるものの「パラメータ値の例」は「キャッシュ要」または「キャッシュ不要」であり、「キャッシュ要」の場合はアプリケーションへ出力したメタ情報の出力用加工結果をキャッシュしておき、以後同等な要求が来た場合にそのキャッシュした情報を返信する。「キャッシュ不要」の場合は、アプリケーションから同一要求が来ても毎回、メタ情報をアクセスし出力用データを生成しなおす。「パラメータ名」が「先読みフラグ」であるものの「パラメータ値の例」は「有効」または「無効」であり、「有効」の場合は、データ出力した内容と類似した構造を持つデータをメタ情報リポジトリ上で探索し、その一覧データを作成し、アプリケーションから同等な要求が来た場合にこの一覧データを用いて応答する。尚、この一覧データは、メタ情報リポジトリ上に保存しても良いし、別の保存部を作成しても良い。 
「無効」の場合は前述の処理を実施しない。「パラメータ名」が「ファイル」であるものの「パラメータ値の例」は例えば「アスピリンの画像」など検索すべきファイルを特定するための情報である。 “Example of parameter value” 602 is an example of a parameter value. Although “parameter name” is “request ID”, “example of parameter value” is “111”, for example, and is information for uniquely identifying the request. Although “parameter name” is “information registration permission flag”, “parameter value example” is “permitted” or “non-permitted”, and in the case of “permitted”, the number of accesses of meta information used as a result of this request Updates and registration of new meta information when it is generated are allowed to be registered in the meta information repository. “Non-permitted” indicates that the above processing is not performed. Although the “parameter name” is “data search time limit”, the “parameter value example” is a time such as “100 ms”, for example, and the time allowed by the application as the data search time is specified. Although “parameter name” is “request confidentiality level”, “example of parameter value” is a numerical value indicating the confidentiality of the request, such as “5”. Although “parameter name” is “accuracy level”, “parameter value example” is a numerical value indicating the accuracy level of meta information allowed by the application, such as “4”. As the accuracy value, for example, the case of using the “score value” of the meta information is considered. Although “parameter name” is “authentication information”, “parameter value example” is application user authentication information, and “parameter name” is “keyword”, but “parameter value example” is, for example, “” Aspirin "" Is a search keyword specified by the application. Although “Parameter name” is “Synonym search permission flag”, “Example of parameter value” is “Permitted” or “Not permitted”. In the case of “not allowed”, it is requested not to search for synonyms of keywords. Although the “parameter name” is “meta information”, the “example of parameter value” is, for example, “meta information structure data representing an aspirin component”, which is information for specifying the type of meta information to be acquired. Although “parameter name” is “request query”, “example of parameter value” is “SQL, SPARQL, etc. for acquiring aspirin components and prescription list”, for example, and indicates the query processing requested by the application. Although “parameter name” is “output format”, “example of parameter value” indicates an output format specified by the application, such as a table format or a list format. Although “Parameter Name” is “Cache Flag”, “Parameter Value Example” is “Cache Required” or “Cache Not Required”. The cached information is cached, and the cached information is returned when an equivalent request is received. In the case of “no cache required”, every time the same request is received from the application, the meta information is accessed and the output data is generated again. Although "Parameter name" is "Prefetch flag", "Parameter value example" is "Valid" or "Invalid". If it is "Valid", data with a structure similar to the data output content is meta information Search in the repository, create the list data, and respond using this list data when an equivalent request comes from the application. The list data may be stored on the meta information repository, or another storage unit may be created.
In the case of “invalid”, the above processing is not performed. Although “parameter name” is “file”, “example of parameter value” is information for specifying a file to be searched such as “aspirin image”.
　図7はアクセス識別ルールの一例を示す図である。 FIG. 7 is a diagram showing an example of an access identification rule.
　これは、アクセス要求識別部がアプリケーションからのアクセス要求を識別するときの判断基準として用いるルール一覧であり、メタ情報リポジトリへ格納される。ただし、ルールが一定の値の場合は、パラメータ列としてアクセス要求識別部へ埋め込んでも構わないし、ルールを保存する専用の記憶部を設けてもよい。
「属性名」701はメタデータに含まれる属性値の一般名称である。行703は「属性名」が「外部委託処理時間閾値」であるものに関する記述であり、行704は「属性名」が「認証必要秘匿性レベル」であるものに関する記述であり、行705は「属性名」が「対応要求クエリ種別一覧」であるものに関する記述である。
「属性値の例」702は属性値の一例である。「属性名」が「外部委託処理時間閾値」であるものの「属性値の例」は例えば「100ms」などの時間であり、「属性名」が「認証必要秘匿性レベル」であるものの「属性値の例」は例えば「2」など秘匿性レベルを示す数値であり、「属性名」が「対応要求クエリ種別一覧」であるものの「属性値の例」はCSV、SQL、SPARQL、…など、データ解析システムが処理可能なクエリの一覧情報である。 This is a rule list used as a criterion for the access request identifying unit to identify an access request from an application, and is stored in the meta information repository. However, when the rule has a constant value, it may be embedded in the access request identifying unit as a parameter string, or a dedicated storage unit for storing the rule may be provided.
“Attribute name” 701 is a general name of an attribute value included in the metadata. Line 703 is a description relating to “attribute name” being “outsourcing processing time threshold”, line 704 is a description relating to “attribute name” being “authentication confidentiality level”, and line 705 is “ This is a description related to “attribute name” being “corresponding request query type list”.
“Example of attribute value” 702 is an example of an attribute value. Although the “attribute name” is “outsourcing processing time threshold”, the “attribute value example” is a time such as “100 ms”, and the “attribute name” is “attribute value required for authentication confidentiality level”. "Example" is a numerical value indicating the level of confidentiality, such as "2". Although "Attribute name" is "Corresponding request query type list", "Example of attribute value" is data such as CSV, SQL, SPARQL, ... This is a list of queries that can be processed by the analysis system.
　図8は生データからメタデータを抽出し、これをメタ情報に変換してメタ情報リポジトリ１１２へ登録する処理を示す図である。この処理は、データ抽出システムがデータ保有システムより取得した生データをデータストアに保存し、この中からデータ解析システムがメタデータを抽出してこのメタデータに基づいてメタ情報を作成し、これをメタ情報リポジトリへ登録する処理である。 FIG. 8 is a diagram showing a process of extracting metadata from raw data, converting it into meta information, and registering it in the meta information repository 112. In this process, the raw data acquired by the data extraction system from the data holding system is stored in the data store, the data analysis system extracts the metadata from this, and creates the meta information based on this metadata. It is a process of registering in the meta information repository.
　ステップ801はデータ抽出システムがデータ保有システムから生データを取得する処理である。 Step 801 is a process in which the data extraction system acquires raw data from the data holding system.
　ステップ802はデータ保有システムが、取得した生データからメタデータとデータ本体を分離し、各々を別々にデータストアに保存する処理である。
尚このステップにおいて、データ保有システムは取得した生データをメタデータとデータ本体に分離せずにデータストアに保存する場合もある。 Step 802 is a process in which the data holding system separates the metadata and the data body from the acquired raw data and stores them separately in the data store.
In this step, the data holding system may store the acquired raw data in the data store without separating the metadata and the data body.
　ステップ803はデータ解析システムのメタデータ抽出部が、データストアからメタデータを取得する処理である。このステップにおいてメタデータ抽出部は、データ抽出システムからデータストアを更新した旨の通知を受けてこれを契機としてメタデータを取得することとしても良いし、データ更新監視部がデータストアの更新有無を監視しておりデータストアの更新を検知した場合にメタデータ抽出部がメタデータを取得することとしても良い。 Step 803 is a process in which the metadata extraction unit of the data analysis system acquires metadata from the data store. In this step, the metadata extraction unit may receive the notification that the data store has been updated from the data extraction system and acquire the metadata in response to this notification, or the data update monitoring unit may determine whether or not the data store has been updated. The metadata extraction unit may acquire metadata when it is monitored and an update of the data store is detected.
　ステップ804はメタデータ抽出部がメタデータをメタ情報に変換する処理である。
例えば、メタデータが持つ、ファイル名、ファイルサイズ、メタデータの保存先といったデータを、属性名と属性値としてもつメタ情報のノードを作成する。あるいは、メタデータ自体を１つの属性値としてもつメタ情報のノードを生成しても良い。 Step 804 is processing in which the metadata extraction unit converts the metadata into metadata.
For example, a node of meta information having an attribute name and an attribute value including data such as a file name, a file size, and a metadata storage destination that the metadata has is created. Alternatively, a meta information node having the metadata itself as one attribute value may be generated.
　ステップ805はステップ８０５でメタデータ抽出部が生成したメタ情報をメタ情報リポジトリへ保存する処理である。ステップ806はデータ解析システムが、引き続きデータ本体からの情報抽出を実行するかどうか調べる処理である。データ本体からの情報抽出を実行するなら、ステップ807へ進む。もしデータ本体からの情報抽出を実行しないなら、処理を終了する。例えば、データ本体が、動画データなどの場合、映像データと音声データが含まれる場合がある。情報抽出はこれらのデータすべてに対して実施する必要がある。ここで言う情報抽出の実行判断処理は、データ本体に含まれる解析対象データの解析がすべて完了したかを判断する処理を示す。　ステップ807は外部委託判定部がデータストアから、データ本体を読み出す処理である。 Step 805 is processing for storing the meta information generated by the metadata extraction unit in step 805 in the meta information repository. Step 806 is processing for examining whether or not the data analysis system continues to extract information from the data body. If information extraction from the data body is executed, the process proceeds to step 807. If the information extraction from the data body is not executed, the process is terminated. For example, when the data body is moving image data, video data and audio data may be included. Information extraction needs to be performed on all of these data. The information extraction execution determination process referred to here indicates a process of determining whether or not the analysis of all the analysis target data included in the data body has been completed. Step 807 is a process in which the outsourcing determination unit reads the data body from the data store.
　ステップ808は外部委託判定部が外部ネットワークの外部情報抽出部１０３と連携可能かどうか調べる処理である。もし外部ネットワークの情報抽出部と連携可能であれば、ステップ809へ進む。もし外部ネットワークの情報抽出部と連携可能でなければ、終了する。例えば、データ解析システムが外部ネットワークに接続されているか否か、外部ネットワークへのアクセスが許可されているかといった判断基準で連携可能性を判断する。 Step 808 is a process for checking whether or not the outsourcing determination unit can cooperate with the external information extraction unit 103 of the external network. If it is possible to cooperate with the information extraction unit of the external network, the process proceeds to step 809. If it is not possible to cooperate with the information extraction unit of the external network, the process ends. For example, the possibility of cooperation is determined based on criteria such as whether the data analysis system is connected to an external network or whether access to the external network is permitted.
　ステップ809は外部委託判定部がメタデータをチェックする処理である。 Step 809 is a process in which the outsourcing determination unit checks the metadata.
　ステップ810は外部委託判定部が外部委託が必要かどうか、すなわち外部情報抽出部１０３にデータ本体からメタ情報を作成してもらう必要があるか調べる処理である。もし外部委託が必要であれば、ステップ811へ進む。もし外部委託が必要でなければ、ステップ813へ進む。この判断基準として、例えばデータ解析システム内部の情報抽出部が対応していないデータ形式の場合、内部の情報抽出部で処理すると時間がかかり処理効率が悪い場合などが考えられる。 Step 810 is processing for examining whether or not the outsourcing determination unit needs to be outsourced, that is, whether the external information extraction unit 103 needs to create meta information from the data body. If outsourcing is necessary, go to Step 811. If outsourcing is not necessary, go to Step 813. As this judgment criterion, for example, in the case of a data format that is not supported by the information extraction unit in the data analysis system, it may take a long time to process by the internal information extraction unit, and the processing efficiency may be poor.
　ステップ811は外部委託判定部が、データストアから取得したメタデータとデータ本体を外部情報抽出部へ送付し情報抽出（メタ情報の作成）を依頼する処理である。依頼を受けた外部情報抽出部は、後述のステップ８１３で情報抽出部１１４が行う処理と同様の方法で、メタデータとデータ本体とから生データの内容を表す意味情報を抽出する。さらにこの意味情報が生データの内容をどの程度正確に表現しているかを表す指標として、所定の方法でスコア値（スコア情報）を生成し、スコア情報と意味情報とを有するメタ情報を作成する。このステップにおいて、外部への情報を漏えいを防ぐため、暗号化通信を用いる場合もある。 Step 811 is a process in which the outsourcing determination unit sends the metadata acquired from the data store and the data body to the external information extraction unit and requests information extraction (meta information creation). The external information extraction unit that has received the request extracts semantic information representing the content of the raw data from the metadata and the data body in the same manner as the processing performed by the information extraction unit 114 in step 813 described later. Further, as an index indicating how accurately this semantic information represents the content of the raw data, a score value (score information) is generated by a predetermined method, and meta information having score information and semantic information is created. . In this step, encrypted communication may be used to prevent leakage of information to the outside.
　ステップ812は外部情報抽出部が、ステップ８１１の依頼に基づき抽出したメタ情報を外部委託判定部へ返信する処理である。このステップにおいて、外部への情報漏えいを防ぐため、暗号化通信を用いる場合もある。 Step 812 is processing in which the external information extraction unit returns the meta information extracted based on the request of Step 811 to the outsourcing determination unit. In this step, encrypted communication may be used to prevent information leakage to the outside.
　ステップ813はデータ解析システムが情報抽出部１１４を呼び出し、メタデータとデータ本体からのメタ情報を作成する処理である。情報抽出部１１４はメタデータとデータ本体から生データの内容を表す意味情報を抽出する。さらにこの意味情報が生データを表す度合いを示すスコア値を算出しこれをスコア情報とする。そして意味情報とスコア情報とを有するメタ情報を作成する。例えば情報抽出部１１４が画像の構造を解析して、それを意味するキーワードを属性値として生成する機能を提供するとした場合、この情報抽出部１１４のモデル１２１に記録された画像パターンデータ群と、解析対象データがどれだけ類似しているかを判定し、もっとも類似性が高い画像パターンデータに対応するキーワードと、その類似度を「スコア値」とした属性値を持つメタ情報を生成することで、解析対象の画像が何を意味するのかを説明するメタ情報を生成できる。音声、センサ情報などの解析についても同等な方法が考えられる。 Step 813 is a process in which the data analysis system calls the information extraction unit 114 to create metadata and meta information from the data body. The information extraction unit 114 extracts semantic information representing the content of the raw data from the metadata and the data body. Further, a score value indicating the degree to which the semantic information represents raw data is calculated and used as score information. Then, meta information having semantic information and score information is created. For example, when the information extraction unit 114 analyzes the structure of the image and provides a function for generating a keyword meaning it as an attribute value, the image pattern data group recorded in the model 121 of the information extraction unit 114, By determining how similar the analysis target data is, by generating meta information with the keyword corresponding to the image pattern data with the highest similarity and the attribute value with the similarity as “score value” Meta information explaining what the analysis target image means can be generated. An equivalent method can be considered for analysis of voice, sensor information, and the like.
　ステップ814は抽出情報関連付け部がメタデータを変換したメタ情報と情報抽出部が抽出したメタ情報を関連付けてメタ情報リポジトリへ格納する処理である。　図9はアプリケーションとデータ解析システムの連携動作を示す図である。 Step 814 is a process of associating the meta information converted by the extraction information association unit with the meta information extracted by the information extraction unit and storing it in the meta information repository. Fig. 9 is a diagram showing the cooperative operation between the application and the data analysis system.
　ステップ901はアプリケーションがデータ解析システムへ処理要求を発行する処理である。アプリケーションが発行する処理要求の一例は図６に示した通りである。このステップにおいて、処理要求の形式としては、独自形式、Webサービス呼び出し、SQL等、様々な形式が考えられる。 Step 901 is a process in which the application issues a processing request to the data analysis system. An example of the processing request issued by the application is as shown in FIG. In this step, various formats such as a unique format, a Web service call, and an SQL can be considered as the format of the processing request.
　ステップ902はアクセス要求識別部が、アクセス要求識別ルールに従って要求内容を判断する処理である。詳細は図１０を用いて詳述する。 Step 902 is a process in which the access request identification unit determines the request content according to the access request identification rule. Details will be described in detail with reference to FIG.
　ステップ903は、ステップ902における判断に基づいて、処理要求がメタ情報管理部で処理(アクセス)できないかどうか調べる処理である。もしメタ情報管理部で処理できない場合は、ステップ904へ進む。もしメタ情報管理部で処理できる場合は、ステップ905へ進む。 Step 903 is a process for checking whether the processing request cannot be processed (accessed) by the meta information management unit based on the determination in step 902. If the meta information management unit cannot process, the process proceeds to step 904. If it can be processed by the meta information management unit, the process proceeds to step 905.
　ステップ904は外部情報抽出部１０３または情報抽出部１１４がアクセス用メタ情報（すなわち検索条件として用いられるメタ情報）を生成する処理である。詳細は図１１を用いて後述する。 Step 904 is processing in which the external information extraction unit 103 or the information extraction unit 114 generates access meta information (that is, meta information used as a search condition). Details will be described later with reference to FIG.
　ステップ905はメタ情報管理部が、アプリケーションの要求に応じたメタ情報をメタ情報リポジトリから取得する処理である。具体的にはアプリケーションの要求の中で指定されているキーワードや、要求クエリの種別、メタ情報、ファイルなどの条件に基づいて、この条件に合致するメタ情報をメタ情報リポジトリ内のメタ情報を検索することにより取得する。また、ステップ９０４でメタ情報が生成されている場合には、メタ情報管理部は生成されたメタ情報を検索条件としてメタ情報リポジトリ内のメタ情報を検索する。なお、アプリケーションの要求の中に正確性レベル607が指定されている場合には、メタ情報管理部はスコア情報409または509が正確性レベル607以上であるメタ情報のみを検索対象としてメタ情報の検索を行う。また、アプリケーションの要求の中でキーワード６０９と共に同義語検索許可フラグ６１０の「許可」が設定されている場合には、メタ情報管理部はメタ情報リポジトリが備えている同義語辞書（複数のキーワード各々について、その同義語が登録されている辞書である）を用いて、キーワードの同義語をまず抽出し、抽出された一または複数の同義語各々を検索条件として用いたメタ情報の検索も行ってメタ情報を取得する。ステップ906はデータ整形部および出力抑制パターン判定部が、取得したメタ情報のデータを整形し、出力抑制判断する処理である。詳細は図１２を用いて後述する。 Step 905 is a process in which the meta information management unit acquires meta information corresponding to the application request from the meta information repository. Specifically, based on keywords specified in the application request, request query type, meta information, file, etc., search meta information in the meta information repository for meta information that matches this condition. To get it. If the meta information has been generated in step 904, the meta information management unit searches the meta information repository using the generated meta information as a search condition. When the accuracy level 607 is specified in the application request, the meta information management unit searches the meta information only for the meta information whose score information 409 or 509 is the accuracy level 607 or higher. I do. In addition, when “permission” of the synonym search permission flag 610 is set together with the keyword 609 in the request of the application, the meta information management unit sets the synonym dictionary (each of the plurality of keywords included in the meta information repository). The synonym of the keyword is first extracted, and the meta information is searched using each of the extracted one or more synonyms as a search condition. Get meta information. Step 906 is processing in which the data shaping unit and the output suppression pattern determination unit reshape the acquired meta information data and determine output suppression. Details will be described later with reference to FIG.
　ステップ907はアプリケーションに整形済みデータ出力する処理である。 Step 907 is a process for outputting the formatted data to the application.
　ステップ908はアプリケーションからの処理要求の先読みフラグ615が有効かどうか調べる処理である。もしアプリケーション要求の先読みフラグが有効であれば、ステップ909へ進む。もしアプリケーション要求の先読みフラグが有効でなければ、終了する。 Step 908 is a process for checking whether the prefetch flag 615 of the processing request from the application is valid. If the prefetch flag of the application request is valid, the process proceeds to step 909. If the prefetch flag of the application request is not valid, the process ends.
　ステップ909は、ステップ９０７で出力したメタ情報と類似した構造を持つメタ情報をメタ情報リポジトリ上で探索し、検索して得られるメタ情報の識別情報一覧を作成してメタ情報リポジトリに格納しておく処理である。 Step 909 searches the meta information repository for meta information having a structure similar to the meta information output in step 907, creates a list of identification information of the meta information obtained by searching, and stores it in the meta information repository. Process.
　図10はアクセス要求識別の振り分け判断を示す図である。 FIG. 10 is a diagram showing an access request identification determination.
　この処理フローは図９のステップ９０２の詳細に相当し、アクセス要求識別部がアプリケーションからの要求を振り分ける処理である。 This processing flow corresponds to the details of step 902 in FIG. 9 and is a process in which the access request identifying unit distributes requests from applications.
　ステップ1001はアプリケーションからアクセス要求（処理要求）を取得する処理である。 Step 1001 is a process for acquiring an access request (processing request) from the application.
　ステップ1002は当該処理要求に設定された要求秘匿性レベル606が、アクセス識別ルールとして設定されている認証必要秘匿性レベル７０４以上かどうか調べる処理である。もし要求秘匿性レベル６０６が認証必要秘匿性レベル７０４以上であれば、ステップ1003へ進む。もし要求秘匿性レベル６０６が認証必要秘匿性レベル７０４以上でなければ、ステップ1008へ進む。 Step 1002 is a process of checking whether the request confidentiality level 606 set in the processing request is equal to or higher than the authentication confidentiality level 704 set as the access identification rule. If the requested confidentiality level 606 is the authentication required confidentiality level 704 or higher, the process proceeds to step 1003. If the requested confidentiality level 606 is not higher than the authentication confidentiality level 704, the process proceeds to step 1008.
　ステップ1003はアクセス要求に認証情報608があるか調べる処理である。もし認証情報があれば、ステップ1004へ進む。もし認証情報ありでなければ、ステップ1007へ進む。 Step 1003 is a process for checking whether there is authentication information 608 in the access request. If there is authentication information, the process proceeds to step 1004. If there is no authentication information, the process proceeds to step 1007.
　ステップ1004は認証処理部に認証情報６０８を渡して認証処理の実行を要求する処理である。 Step 1004 is a process of requesting execution of authentication processing by passing authentication information 608 to the authentication processing unit.
　ステップ1005は認証処理部から受け取った認証結果に基づき認証エラーが発生しているかどうか調べる処理である。もし認証エラー発生であれば、ステップ1006へ進む。もし認証エラー発生でなければ、ステップ1008へ進む。 Step 1005 is a process for checking whether an authentication error has occurred based on the authentication result received from the authentication processing unit. If an authentication error has occurred, go to Step 1006. If no authentication error has occurred, the process proceeds to step 1008.
　ステップ1006はアプリケーションへエラーを返す処理である。 Step 1006 is a process for returning an error to the application.
　ステップ1007はアプリケーションへエラーを返す処理である。 Step 1007 is a process for returning an error to the application.
　ステップ1008はアクセス要求に検索用メタ情報６１１の指定があるかどうか調べる処理である。もし検索用メタ情報の指定があれば、ステップ1009へ進む。もし検索用メタ情報指定ありでなければ、ステップ1010へ進む。 Step 1008 is a process for checking whether or not the search meta information 611 is specified in the access request. If the search meta information is designated, the process proceeds to step 1009. If no search meta information is specified, the process proceeds to step 1010.
　ステップ1009は、メタ情報管理部で検索用のメタ情報６１１を検索条件として用いたメタ情報リポジトリの検索が可能と判断する処理である。この場合、先述の図９のステップ９０３ではメタ情報管理部で処理可能だと判断される。 Step 1009 is a process in which the meta information management unit determines that the meta information repository can be searched using the search meta information 611 as a search condition. In this case, it is determined in step 903 of FIG. 9 that the meta information management unit can process.
　ステップ1010はアクセス要求にキーワード６０９の指定があるかどうか調べる処理である。もしキーワードの指定があれば、ステップ1011へ進む。もしキーワードの指定がなければ、ステップ1014へ進む。 Step 1010 is a process for checking whether or not the keyword 609 is specified in the access request. If a keyword is specified, the process proceeds to step 1011. If no keyword is specified, the process proceeds to step 1014.
　ステップ1011はアクセス要求において同義語検索が許可されているか、すなわち同義語検索許可フラグ６１０が許可となっているか調べる処理である。もし同義語検索が許可されていれば、ステップ1013へ進む。もし同義語検索が許可されていなければ、ステップ1012へ進む。 Step 1011 is a process for checking whether the synonym search is permitted in the access request, that is, whether the synonym search permission flag 610 is permitted. If the synonym search is permitted, the process proceeds to step 1013. If the synonym search is not permitted, the process proceeds to step 1012.
　ステップ1012はキーワードを用いてメタ情報管理部がメタ情報をメタ情報リポジトリから取得可能（アクセス可能）と判断する処理である。この場合、先述の図９のステップ９０３ではメタ情報管理部で処理可能だと判断される。 Step 1012 is processing in which the meta information management unit determines that the meta information can be acquired (accessed) from the meta information repository using the keyword. In this case, it is determined in step 903 of FIG. 9 that the meta information management unit can process.
　ステップ1013はメタ情報リポジトリのアクセス用同義語抽出処理の実行が必要と判断する処理である。この場合、先述の図９のステップ９０３ではメタ情報管理部で同義語抽出の処理が必要だと判断される。 Step 1013 is processing for determining that it is necessary to execute access synonym extraction processing of the meta information repository. In this case, in step 903 of FIG. 9, the meta information management unit determines that the synonym extraction process is necessary.
　ステップ1014はアクセス要求において要求クエリ612の指定があるか調べる処理である。もし要求クエリの指定があれば、ステップ1015へ進む。もし要求クエリの指定がなければ、ステップ1018へ進む。 Step 1014 is a process for checking whether the request query 612 is specified in the access request. If a request query is specified, the process proceeds to step 1015. If no request query is specified, the process proceeds to step 1018.
　ステップ1015は図７に示すアクセス識別ルールの対応要求クエリ種別一覧705に、ステップ１０１４で確認した要求クエリが登録されているか調べる処理である。もしアクセス識別ルールの対応要求クエリ種別一覧に当該要求クエリが登録されていれば、ステップ1017へ進む。もしアクセス識別ルールの対応要求クエリ種別一覧に当該要求クエリの登録がなければ、ステップ1016へ進む。 Step 1015 is a process for checking whether or not the request query confirmed in Step 1014 is registered in the corresponding request query type list 705 of the access identification rule shown in FIG. If the request query is registered in the corresponding request query type list of the access identification rule, the process proceeds to step 1017. If the request query is not registered in the corresponding request query type list of the access identification rule, the process proceeds to step 1016.
　ステップ1016は情報抽出部でクエリの解析が必要と判断する処理である。この場合、先述の図９のステップ９０３ではメタ情報管理部で処理できないと判断されるので、ステップ９０４に進み、指定されたクエリを外部情報抽出部や情報抽出部などで実行して検索にメタ情報の抽出が行われることになる。このクエリ処理によって得られたメタ情報は、検索条件として後に図９のステップ９０５でメタ情報管理部に用いられることになる。 Step 1016 is processing in which the information extraction unit determines that query analysis is necessary. In this case, since it is determined in step 903 of FIG. 9 that the meta information management unit cannot process, the process proceeds to step 904, where the specified query is executed by the external information extraction unit, the information extraction unit, etc. Information will be extracted. The meta information obtained by this query processing will be used later as a search condition by the meta information management unit in step 905 of FIG.
　ステップ1017はメタ情報管理部がクエリ処理を実行することで、検索用のメタ情報をメタ情報リポジトリから取得可能（アクセス可能）と判断する処理である。この場合、先述の図９のステップ９０３では、まずメタ情報管理部がクエリ処理を実行して検索用のメタ情報を取得してからこれを用いてメタ情報リポジトリを検索する必要があると判断される。　ステップ1018はアクセス要求においてファイル616の指定があるかどうか調べる処理である。もしファイルの指定があれば、ステップ1019へ進む。もしファイルの指定がなければ、ステップ1020へ進む。 Step 1017 is processing in which the meta information management unit executes query processing to determine that search meta information can be acquired (accessed) from the meta information repository. In this case, in step 903 of FIG. 9 described above, it is first determined that the meta information management unit needs to search the meta information repository by using the query processing to acquire the meta information for search and then using it. The Step 1018 is a process for checking whether the file 616 is specified in the access request. If a file is specified, the process proceeds to step 1019. If no file is specified, the process proceeds to step 1020.
　ステップ1019はファイルからメタデータを抽出しこれをメタ情報に変換してメタ情報リポジトリに格納する処理である。尚、メタ情報とスコア値の算出は前述のとおりである。このステップにおいて、もしメタデータ抽出に失敗した場合はエラーを返す。 Step 1019 is a process of extracting metadata from the file, converting it into meta information, and storing it in the meta information repository. The calculation of the meta information and the score value is as described above. In this step, if metadata extraction fails, an error is returned.
　ステップ1020は、メタ情報管理部がステップ１０１９で作成されたメタ情報を検索用メタ情報として用いてメタ情報リポジトリからメタ情報を取得することが可能（アクセス可能）と判断する処理である。この場合、先述の図９のステップ９０３ではメタ情報管理部で処理可能だと判断される。　図11はアクセス用メタ情報生成を示す図である。この処理は図９のステップ９０４の詳細フローであり、アプリケーションからの要求に含まれる解析対象データに基づいて、メタ情報リポジトリへのアクセス用のメタ情報を抽出する処理の流れを示す。 Step 1020 is a process in which the meta information management unit determines that the meta information can be acquired (accessed) from the meta information repository using the meta information created in step 1019 as the search meta information. In this case, it is determined in step 903 of FIG. 9 that the meta information management unit can process. FIG. 11 is a diagram showing access meta information generation. This process is a detailed flow of step 904 in FIG. 9 and shows a flow of a process of extracting meta information for access to the meta information repository based on the analysis target data included in the request from the application.
　ステップ1101はメタデータ抽出部が外部委託判定部へ解析対象データを送信する処理である。解析対象データには、例えば要求クエリ６１２などのほか、アプリケーションからの要求に添付される音声データや画像データなどの添付データが含まれる。ステップ1102はメタ情報抽出部が解析対象データのメタデータを抽出する処理である。 Step 1101 is a process in which the metadata extraction unit transmits the analysis target data to the outsourcing determination unit. The analysis target data includes, for example, a request query 612 and the like, and attached data such as voice data and image data attached to a request from the application. Step 1102 is a process in which the meta information extraction unit extracts metadata of analysis target data.
　ステップ1103は外部委託判定部が解析対象データが外部委託対象かどうか調べる処理である。このステップにおいて、解析対象データのメタデータの内容を情報抽出部利用ルールと比較して、外部委託対象か否か判断する。情報抽出部利用ルールには例えば外部情報抽出部１０３で処理可能なクエリの種別が登録されている。従ってこれを参照して、アプリケーションからの処理要求によって指定されるクエリ処理が外部情報抽出部１０３で実行可能か判断し、実行可能であれば外部委託対象としても良い。また、情報抽出部利用ルールには要求秘匿必要レベルがあらかじめ登録されており、アプリケーションからの要求に含まれる要求秘匿性レベル606が、この要求秘匿必要レベルよりも低い場合に外部委託対象とすることにしても良い。もし外部委託対象であれば、ステップ1104へ進む。もし外部委託対象でなければ、ステップ1106へ進む。ステップ1104は外部委託判定部が外部情報抽出部へ、解析対象データを送付する処理である。ステップ1105は外部委託判定部が外部情報抽出部から解析対象データの解析結果（すなわち抽出したメタ情報）を受信する処理である。 Step 1103 is a process in which the outsourcing determination unit checks whether the data to be analyzed is an outsourcing object. In this step, the content of the metadata of the analysis target data is compared with the information extraction unit usage rule, and it is determined whether or not it is an outsourcing target. For example, the type of query that can be processed by the external information extraction unit 103 is registered in the information extraction unit usage rule. Therefore, referring to this, it is determined whether the query processing specified by the processing request from the application can be executed by the external information extraction unit 103, and if it can be executed, it may be set as an outsourcing target. In addition, the required level of required confidentiality is registered in advance in the information extraction unit usage rule, and if the required confidentiality level 606 included in the request from the application is lower than this required level of required confidentiality, the information is to be outsourced. Anyway. If it is outsourced, the process proceeds to step 1104. If it is not outsourced, the process proceeds to step 1106. Step 1104 is processing in which the outsourcing determination unit sends analysis target data to the external information extraction unit. Step 1105 is a process in which the outsourcing determination unit receives the analysis result (that is, the extracted meta information) of the analysis target data from the external information extraction unit.
　ステップ1106は外部委託判定部が情報抽出部へ解析対象データを入力し、これを受信した情報抽出部が情報抽出を実行してメタ情報を抽出する処理である。尚、抽出方法は前述で例示したとおりである。ステップ1107は解析対象データ内に未解析のデータが存在かどうか調べる処理である。このステップにおいて、解析対象データが、例えば動画、音声等複数のデータから構成されているときはそのすべてについて処理が実行されたか確認する。もし解析対象データ内に未解析のデータが存在であれば、ステップ1103へ戻る。一方解析対象データ内に未解析のデータが存在でなければ、ステップ1108へ進む。 Step 1106 is a process in which the outsourcing determination unit inputs analysis target data to the information extraction unit, and the information extraction unit that receives the data extracts information by executing information extraction. The extraction method is as exemplified above. Step 1107 is processing for examining whether or not unanalyzed data exists in the analysis target data. In this step, when the analysis target data is composed of a plurality of data such as moving images and voices, it is confirmed whether or not the processing has been executed for all of them. If unanalyzed data exists in the analysis target data, the process returns to step 1103. On the other hand, if unanalyzed data does not exist in the analysis target data, the process proceeds to step 1108.
　ステップ1108は抽出した全メタ情報を抽出情報関連付け部で統合する処理である。統合された情報はメタ情報リポジトリへ格納される。 Step 1108 is a process of integrating all extracted meta information in the extracted information association unit. The integrated information is stored in the meta information repository.
　図12はデータ整形と出力抑制処理の概要を示す図である。この処理は図９のステップ９０６の詳細に相当し、データ整形部１１１および出力抑制パターン判定部１１０によってアプリケーションの要求に応じた出力形式にメタ情報を加工し、加工結果がセキュリティの観点から問題ないか判断して出力を制御する処理である。 Fig. 12 is a diagram showing an overview of data shaping and output suppression processing. This process corresponds to the details of step 906 in FIG. 9, and the data shaping unit 111 and the output suppression pattern determination unit 110 process the meta information into an output format according to the application request, and the processing result is satisfactory from the viewpoint of security. This is the process of controlling whether the output is determined.
　ステップ1201はステップ９０５でメタ情報管理部がメタ情報リポジトリから取得した出力対象メタ情報を、データ整形部が取得する処理である。 Step 1201 is processing in which the data shaping unit acquires the output target meta information acquired from the meta information repository by the meta information management unit in step 905.
　ステップ1202はデータ整形部が、アプリケーションからのアクセス要求に応じたスキーマ定義をメタ情報リポジトリから取得する処理である。このステップにおいて、スキーマ定義は、メタ情報を表形式、リスト形式、XML形式等に変換するルールが記載されており、アプリケーションのアクセス要求の中で出力形式613として指定されている出力形式に応じたものをメタ情報リポジトリから取得する。 Step 1202 is a process in which the data shaping unit acquires the schema definition corresponding to the access request from the application from the meta information repository. In this step, the schema definition describes the rules for converting meta information into table format, list format, XML format, etc., according to the output format specified as output format 613 in the application access request Get things from the meta information repository.
　ステップ1203はデータ整形部がスキーマ定義あるいは出力対象メタ情報にデータ本体参照の指定があるかどうか調べる処理である。もしスキーマ定義あるいは出力対象メタ情報にデータ本体参照の指定があれば、ステップ1204へ進む。もしスキーマ定義あるいは出力対象メタ情報にデータ本体参照の指定がなければ、ステップ1205へ進む。 Step 1203 is a process in which the data shaping unit checks whether the data body reference is specified in the schema definition or the output target meta information. If the data body reference is specified in the schema definition or the output target meta information, the process proceeds to step 1204. If the data body reference is not specified in the schema definition or the output target meta information, the process proceeds to step 1205.
　ステップ1204はデータ整形部がデータストアから参照指定されているデータ本体を取得する処理である。 Step 1204 is a process in which the data shaping unit acquires the data body whose reference is specified from the data store.
　ステップ1205はデータ整形部が、スキーマ定義に基づき出力データを生成する処理である。 Step 1205 is a process in which the data shaping unit generates output data based on the schema definition.
　ステップ1206は出力抑制パターン判定部が出力データ形式をチェックする処理である。
このステップにおいて、出力抑制パターン判定部は、メタ情報リポジトリ内の出力抑制パターン判定用ルールに記載されたルールに基づいて出力データの形式をチェックする。 Step 1206 is a process in which the output suppression pattern determination unit checks the output data format.
In this step, the output suppression pattern determination unit checks the format of the output data based on the rules described in the output suppression pattern determination rules in the meta information repository.
　ステップ1207は出力抑制パターン判定部が出力抑制パターン判定用ルールに基づいて、出力データに出力禁止構造が含まれるかどうか調べる処理である。もし出力データに出力禁止構造が含まれる場合は、ステップ1208へ進む。もし出力データに出力禁止構造が含まれない場合は、ステップ1210へ進む。 Step 1207 is a process in which the output suppression pattern determination unit checks whether the output data includes an output prohibition structure based on the output suppression pattern determination rule. If the output prohibition structure is included in the output data, the process proceeds to step 1208. If the output prohibition structure is not included in the output data, the process proceeds to step 1210.
　ステップ1208は出力抑制パターン判定部が出力データ内の出力禁止構造に対応する部分をダミーデータに変換する処理である。このステップにおいて、割り当てるダミーデータは、例えば文字列、値など種別に応じて事前に定めた値を割り当てるなどの方法が考えられる。 Step 1208 is a process in which the output suppression pattern determination unit converts a portion corresponding to the output prohibition structure in the output data into dummy data. In this step, for the dummy data to be assigned, for example, a method of assigning a predetermined value according to the type such as a character string or a value can be considered.
　ステップ1209は出力抑制パターン判定部がアプリケーションへ警告つきで出力データを出力する処理である。 Step 1209 is a process in which the output suppression pattern determination unit outputs output data with a warning to the application.
　ステップ1210は出力抑制パターン判定部がアプリケーションへ出力データを出力する処理である。 Step 1210 is a process in which the output suppression pattern determination unit outputs output data to the application.
　図13はアプリケーションが入力したメタ情報をメタ情報リポジトリへ登録する処理の流れを示す図である。 FIG. 13 is a diagram showing a flow of processing for registering meta information input by the application in the meta information repository.
　ステップ1301はアプリケーションがメタ情報登録要求をデータ解析システムへ発行する処理である。このメタ情報登録要求には登録されるべきメタ情報が含まれる。 Step 1301 is a process in which the application issues a meta information registration request to the data analysis system. This meta information registration request includes meta information to be registered.
　ステップ1302はアクセス要求識別部がアプリケーションから受信したメタ情報を検索キーにしてメタ情報リポジトリを検索する処理である。 Step 1302 is a process in which the access request identifying unit searches the meta information repository using the meta information received from the application as a search key.
　ステップ1303は受信したメタ情報と部分的に一致する構造がメタ情報リポジトリ内に見つかるかどうか調べる処理である。もし部分的に一致する構造が見つかる場合は、ステップ1304へ進む。もし部分的に一致する構造が見つかない場合は、ステップ1305へ進む。 Step 1303 is a process for checking whether a structure partially matching the received meta information is found in the meta information repository. If a partially matching structure is found, go to step 1304. If a partially matching structure is not found, go to step 1305.
　ステップ1304は一致する構造の最上位のノードにアプリケーションからのメタ情報を追加する処理である。 Step 1304 is processing for adding meta information from the application to the top node of the matching structure.
　ステップ1305は新たなデータ構造として登録する処理である。 Step 1305 is a process of registering as a new data structure.
　図14は、データストア内のデータが更新されたことを検出して、情報抽出を実行する処理の流れを示す図である。 FIG. 14 is a diagram showing a flow of processing for detecting that data in the data store has been updated and performing information extraction.
　ステップ1401はデータ更新監視部が、データストアの更新を検出する処理である。 Step 1401 is a process in which the data update monitoring unit detects a data store update.
　ステップ1402は更新データについて、メタデータ抽出と情報抽出処理（更新データについてのメタ情報の作成・更新）を実行する処理である。 Step 1402 is processing for executing metadata extraction and information extraction processing (creation / update of meta information for update data) for the update data.
　ステップ1403はメタ情報リポジトリ内の該当データに対応するメタ情報を更新（反映）する処理である。 Step 1403 is a process of updating (reflecting) the meta information corresponding to the corresponding data in the meta information repository.
　図15は、学習用サンプルデータを用いた情報抽出部の学習処理の流れを示す図である。 FIG. 15 is a diagram showing the flow of the learning process of the information extraction unit using the learning sample data.
　ステップ1501は情報抽出部が学習を開始する処理である。 Step 1501 is a process in which the information extraction unit starts learning.
　ステップ1502は外部データの取り込みが可能かどうか調べる処理である。もし外部データの取り込みが可能であれば、ステップ1503へ進む。もし外部データの取り込みが可能でなければ、ステップ1509へ進む。外部データ取り込みの判断基準として、例えば外部ネットワークにデータ解析システムがアクセス可能学習用データが存在するという情報が、データ解析システムに登録されており、かつ、外部ネットワークに接続可能であるという場合などが考えられる。 Step 1502 is a process for checking whether external data can be imported. If external data can be imported, the process proceeds to step 1503. If external data cannot be imported, the process proceeds to step 1509. As a criterion for external data capture, for example, information that learning data that can be accessed by the data analysis system in the external network is registered in the data analysis system and can be connected to the external network. Conceivable.
　ステップ1503は外部データ取り込み部が外部参照データを取り込む処理である。 Step 1503 is a process in which the external data capturing unit captures external reference data.
　ステップ1504は学習用サンプルデータと外部データ取り込み部が取り込んだ取り込みデータを照合する処理である。 Step 1504 is a process for collating the learning sample data with the captured data captured by the external data capturing unit.
　ステップ1505は外部データ取り込み部が取り込んだ取り込みデータと学習用サンプルデータとの間に矛盾する部分がありかどうか調べる処理である。もし矛盾する部分がありであれば、ステップ1506へ進む。もし矛盾する部分がなければ、ステップ1509へ進む。この矛盾とは、例えば、内部学習データでは「犬は動物である」「動物と植物は異なる」で、外部データでは「犬は植物である」と定義されていた場合、論理的な矛盾が生じる場合もあるし、「富士山の高さは3776ｍ」「富士山の高さは3022ｍ」といった、単純な値の矛盾も考えられる。 Step 1505 is a process for checking whether there is a contradictory portion between the fetched data fetched by the external data fetching unit and the learning sample data. If there is a contradiction, the process proceeds to step 1506. If there is no conflict, go to Step 1509. This contradiction means, for example, that the internal learning data is defined as “dogs are animals” and “animals and plants are different”, and external data is defined as “dogs are plants”. In some cases, simple values such as “Mt. Fuji is 3776m high” and “Mt. Fuji is 3022m high” can be considered.
　ステップ1506は矛盾個所について、外部データを学習用サンプルデータに置き換えする処理である。 Step 1506 is a process for replacing the external data with the learning sample data for the inconsistent portion.
　ステップ1507は置き換え不可かどうか調べる処理である。もし置き換え不可であれば、ステップ1508へ進む。もし置き換え不可でなければ、ステップ1509へ進む。 Step 1507 is a process for checking whether or not replacement is possible. If it cannot be replaced, the process proceeds to step 1508. If not replaceable, go to step 1509.
　ステップ1508は外部参照データの利用を中止する処理である。 Step 1508 is a process for canceling the use of the external reference data.
　ステップ1509は情報抽出部内の利用可能な学習用サンプルデータを用いて、学習機構が学学習処理を実行し、抽出モデルを修正する処理である。 Step 1509 is a process in which the learning mechanism executes a learning process using the available learning sample data in the information extraction unit to correct the extraction model.
101…データ解析システム
102…外部ネットワーク
103…外部情報抽出部
104…データ保有システム
105…生データストア
106…アクセス要求識別部
107…メタ情報管理部
108…アプリケーション
110…出力抑制パターン判定部
111…データ整形部
112…メタ情報リポジトリ
113…出力抑制ルール
114…情報抽出部
115…外部委託判定部
116…抽出情報関連付け部
117…データ抽出システム
118…学習機構
119…データストア
120…外部参照データ
121…抽出モデル
122…アクセス識別ルール
123…メタデータ抽出部
124…メタ情報
125…スキーマ定義
128…外部データ取り込み部
129…学習用サンプルデータ
130…学習制限ルール
133…生データ
134…データ本体
135…メタデータ
139…情報抽出部利用ルール
142…外部抽出モデル
143…データ更新監視部
144…認証処理部
201…ノード１
202…エッジ
301…属性名
302…属性値の例
401…属性名
402…属性値の例
501…属性名
502…属性値の例
601…パラメータ名
602…パラメータ値の例
701…属性名
702…属性値の例 101 ... Data analysis system
102 ... External network
103 ... External information extraction unit
104 ... Data retention system
105 ... Raw data store
106: Access request identification section
107… Meta information management department
108… Application
110 ... Output suppression pattern judgment unit
111 ... Data shaping section
112 ... Meta information repository
113… Output suppression rules
114 ... Information extraction unit
115 ... Outsourcing decision section
116 ... Extraction information association part
117 ... Data extraction system
118 ... Learning mechanism
119… Data store
120… External reference data
121 ... Extraction model
122 ... Access identification rule
123 ... Metadata extraction unit
124… Meta information
125 ... Schema definition
128 ... External data capture unit
129 ... Sample data for learning
130 ... Learning restriction rules
133… Raw data
134 ... Data body
135 ... Metadata
139 ... Information extraction unit usage rules
142 ... External extraction model
143 ... Data update monitoring unit
144 ... Authentication processing section
201 ... Node 1
202 ... Edge
301 ... Attribute name
302 ... Example of attribute value
401 ... Attribute name
402 ... Example of attribute value
501 ... Attribute name
502 ... Attribute value example
601 ... Parameter name
602 ... Parameter value example
701 ... Attribute name
702 ... Example of attribute value

Claims (6)

  1. 　メタデータとデータ本体とを有する非構造データを格納する記憶装置と接続されるデータ解析システムであって、
    　前記記憶装置から前記非構造データの前記メタデータを取得して当該メタデータから第一のメタ情報を作成するメタデータ抽出部と、
    　前記記憶装置から前記非構造データの前記メタデータと前記データ本体を取得して、当該メタデータと当該データ本体から前記非構造データの内容を表す意味情報を抽出して当該意味情報を有する第二のメタ情報を作成する情報抽出部と、
    　前記第一のメタ情報と前記第二のメタ情報とを関連付けて格納するメタ情報リポジトリと、
    　アプリケーションからの要求に応じて、前記メタ情報リポジトリに格納されているメタ情報を抽出するメタ情報管理部と、
    　前記メタ情報管理部が抽出したメタ情報を前記要求によって指定される方法で加工して出力データを作成し、当該出力データを前記アプリケーションへ出力するデータ出力部とを有し、
    　前記情報抽出部は、前記非構造データの意味情報を抽出した際に、当該意味情報が前記非構造データの内容を表す度合いを示すスコア値を計算し、このスコア値を前記第二のメタ情報に含ませることを特徴とするデータ解析システム。     A data analysis system connected to a storage device for storing unstructured data having metadata and a data body,
    A metadata extraction unit that obtains the metadata of the unstructured data from the storage device and creates first metadata from the metadata;
    The metadata and the data body of the non-structured data are acquired from the storage device, semantic information representing the contents of the non-structured data is extracted from the metadata and the data body, and the second has the semantic information An information extraction unit that creates meta information of
    A meta information repository that stores the first meta information and the second meta information in association with each other;
    In response to a request from an application, a meta information management unit that extracts meta information stored in the meta information repository;
    Processing the meta information extracted by the meta information management unit by a method specified by the request to create output data, and a data output unit for outputting the output data to the application,
    When the information extraction unit extracts semantic information of the non-structured data, the information extraction unit calculates a score value indicating a degree that the semantic information represents the content of the non-structured data, and uses the score value as the second meta information. Data analysis system characterized by being included in.
  2. 　請求項１記載のデータ解析システムであって、さらに、
    　前記アプリケーションからの要求には、データの正確度を表す正確性レベルが指定されており、
    　前記メタ情報管理部は、前記メタ情報リポジトリに格納されているメタ情報のうち、前記スコア値が前記正確性レベルより大きいメタ情報のみを対象として、メタ情報を抽出することを特徴とするデータ解析システム。     The data analysis system according to claim 1, further comprising:
    The request from the application specifies an accuracy level representing the accuracy of the data,
    The meta-information management unit extracts meta-information only for meta-information stored in the meta-information repository with the score value being greater than the accuracy level. system.
  3. 　請求項２記載のデータ解析システムであって、
    　前記アプリケーションからの要求には、データ探索の時間制限を示す制限時間値が指定されており、
    　前記メタ情報管理部は、前記制限時間値に基づいて、検索対象とするメタ情報のスコア値の下限を定めることを特徴とするデータ解析システム。     A data analysis system according to claim 2, comprising:
    In the request from the application, a time limit value indicating a data search time limit is specified,
    The meta information management unit determines a lower limit of a score value of meta information to be searched based on the time limit value.
  4. 　請求項３記載のデータ解析システムであって、さらに、
    　前記要求を当該データ解析システムに接続されている外部情報抽出部に処理させるか否かを判断する外部委託判断部を有しており、
    　前記アプリケーションからの要求には、当該要求の秘匿性レベルが含まれており、
    　前記外部委託判断部は、前記要求の秘匿性レベルが予め定められた秘匿性のレベルよりも高い場合には、当該要求の処理を前記外部情報処理部に処理させないよう制御することを特徴とするデータ解析システム。     The data analysis system according to claim 3, further comprising:
    An outsourcing determination unit that determines whether or not to allow the external information extraction unit connected to the data analysis system to process the request;
    The request from the application includes the level of confidentiality of the request,
    The outsourcing determination unit controls the external information processing unit not to process the request when the confidentiality level of the request is higher than a predetermined confidentiality level. Data analysis system.
  5. 　請求項４記載のデータ解析システムであって、
    　前記メタ情報リポジトリは、各キーワードについてその同義語を登録した同義語辞書を有しており、
    　前記メタ情報管理部は、前記アプリケーションから受信した要求に、キーワードの指定と当該キーワードの同義語を検索する旨の指定が含まれている場合に、前記同義語辞書を用いて当該キーワードの同義語を取得し、当該キーワードと当該同義語とを用いて前記メタ情報リポジトリを検索してメタ情報を取得することを特徴とするデータ解析システム。     The data analysis system according to claim 4, wherein
    The meta information repository has a synonym dictionary that registers synonyms for each keyword,
    When the request received from the application includes a keyword specification and a specification to search for a synonym of the keyword, the meta information management unit uses the synonym dictionary to synonym the keyword A data analysis system characterized by acquiring meta information by searching the meta information repository using the keyword and the synonym.
  6. 　請求項５記載のデータ解析システムであって、さらに、
    　前記アプリケーションからの要求に基づき、当該要求の内容を識別するアクセス要求識別部を有しており、
    　当該アクセス要求識別部は、前記アプリケーションからの要求が前記メタ情報管理部で処理できないと判断した場合に、前記要求の内容を前記情報抽出部に引き渡し、
    　前記情報抽出部は、受信した前記要求の内容から意味情報を抽出してメタ情報を作成し、当該メタ情報を前記メタ情報リポジトリに格納し、
    　前記メタ情報管理部は、前記情報抽出部が抽出したメタ情報を用いて、前記メタ情報リポジトリを検索することを特徴とするデータ解析システム。     The data analysis system according to claim 5, further comprising:
    Based on the request from the application, it has an access request identifying unit that identifies the content of the request,
    When the access request identification unit determines that the request from the application cannot be processed by the meta information management unit, the access request identification unit delivers the content of the request to the information extraction unit,
    The information extraction unit extracts semantic information from the received request content to create meta information, stores the meta information in the meta information repository,
    The meta information management unit searches the meta information repository using the meta information extracted by the information extraction unit.
PCT/JP2011/003695 2011-06-29 2011-06-29 Unstructured data analysis system WO2013001571A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2011/003695 WO2013001571A1 (en) 2011-06-29 2011-06-29 Unstructured data analysis system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2011/003695 WO2013001571A1 (en) 2011-06-29 2011-06-29 Unstructured data analysis system

Publications (1)

Publication Number Publication Date
WO2013001571A1 true WO2013001571A1 (en) 2013-01-03

Family

ID=47423520

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2011/003695 WO2013001571A1 (en) 2011-06-29 2011-06-29 Unstructured data analysis system

Country Status (1)

Country Link
WO (1) WO2013001571A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015015559A1 (en) * 2013-07-30 2015-02-05 株式会社日立製作所 Search system and search method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004102767A (en) * 2002-09-11 2004-04-02 Nippon Hoso Kyokai <Nhk> Metadata collection/distribution system, metadata collection server, metadata collection method, metadata collection program, and recording medium having metadata collection program recorded therein
JP2005092295A (en) * 2003-09-12 2005-04-07 Toshiba Corp Meta information generating method and device, retrieval method and device
JP2011034394A (en) * 2009-08-03 2011-02-17 Fujitsu Ltd Content providing device, content provision program, and content providing method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004102767A (en) * 2002-09-11 2004-04-02 Nippon Hoso Kyokai <Nhk> Metadata collection/distribution system, metadata collection server, metadata collection method, metadata collection program, and recording medium having metadata collection program recorded therein
JP2005092295A (en) * 2003-09-12 2005-04-07 Toshiba Corp Meta information generating method and device, retrieval method and device
JP2011034394A (en) * 2009-08-03 2011-02-17 Fujitsu Ltd Content providing device, content provision program, and content providing method

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015015559A1 (en) * 2013-07-30 2015-02-05 株式会社日立製作所 Search system and search method

Similar Documents

Publication Publication Date Title
US10599719B2 (en) System and method for providing prediction-model-based generation of a graph data model
Ben Ellefi et al. RDF dataset profiling–a survey of features, methods, vocabularies and applications
US11573996B2 (en) System and method for hierarchically organizing documents based on document portions
US11727000B2 (en) System and method for reducing resource usage in a data retrieval process
US10223417B1 (en) System and method for reducing query-related resource usage in a data retrieval process
US9348943B2 (en) Method for analyzing time series activity streams and devices thereof
US20160314121A1 (en) System and method for natural language querying
CN108268600B (en) AI-based unstructured data management method and device
US10942977B2 (en) Systems and methods for targeting, reviewing, and presenting online social network data by evidence context
US20100153331A1 (en) System and method for managing semantic and syntactic metadata
Meroño-Peñuela et al. CEDAR: the Dutch historical censuses as linked open data
Sheth Semantic Services, Interoperability and Web Applications: Emerging Concepts: Emerging Concepts
CN110362727A (en) Third party for search system searches for application
WO2008069125A1 (en) Data management device
US20160110471A1 (en) Method and system of intelligent generation of structured data and object discovery from the web using text, images, video and other data
US9940355B2 (en) Providing answers to questions having both rankable and probabilistic components
Sleeman et al. Entity type recognition for heterogeneous semantic graphs
Färber et al. The data set knowledge graph: Creating a linked open data source for data sets
US20230205824A1 (en) Contextual Clarification and Disambiguation for Question Answering Processes
JP2024041902A (en) Multi-source interoperability and/or information retrieval optimization
US10866944B2 (en) Reconciled data storage system
WO2023236257A1 (en) Document search platform, search method and apparatus, electronic device, and storage medium
WO2013001571A1 (en) Unstructured data analysis system
US20220156228A1 (en) Data Tagging And Synchronisation System
US10579738B2 (en) System and method for generating a multi-lingual and multi-intent capable semantic parser based on automatically generated operators and user-designated utterances relating to the operators

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11868593

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 11868593

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP