JP2008511936A - Method and system for semantic identification in a data system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data integration system tool that enables use, reuse, and change of functions in a changing business environment.
Semantic identifiers that enable identification of items based on their relationship to other items without the need for additional data, data, metadata, semantic identifiers and other items in a certain format, language , And / or methods associated with the conversion engine that can convert from the data model to the other, and the abstract property level of the hub or database that allows for the distinction of multiple instances or forms of items and A system is provided.
[Selection] Figure 1

Description

  The present invention relates to the field of information technology, and more particularly to the field of data integration systems.

  With the advent of computer applications, many business processes have become faster and more efficient. However, with the proliferation of different computer applications using different data structures, communication protocols, languages and platforms, the typical enterprise IT infrastructure has become extremely complex. Different business processes within a typical enterprise may use computer applications that are each developed and optimized for a specific business process rather than the entire enterprise. For example, a company may have a specific computer application for tracking payment accounts and a computer application for tracking the history of customer contacts. In fact, it maintains a centralized customer contact database, but when employees maintain their own contact information, such as in a personal information manager, a company may have more than one computer application, even in the same business process. May be used.

  Special-purpose computer applications offer the advantage of being able to provide a tailored solution to the customer, but the more specific-use computer applications, the same data is repeatedly entered throughout the enterprise This may lead to inefficiencies such as the need to process, or when a company executes other processes that benefit from data associated with one process. For example, if the payment account process is separated from the supply chain and ordering process, the company may accept and respond to orders from customers with a credit history that the company will reject the order. is there. There are many other examples where companies can benefit from consistent access to all of the data across various computer applications.

  Many companies have recognized and addressed the need to share data between different applications in the enterprise. In this way, enterprise application integration or EAI has emerged as a message-based strategy for processing data from different sources. As the complexity and number of computer applications increase, EAI efforts address the need to handle different protocols, the need to handle increasing data and transactions, and the demand for faster integration of increasing data. Face many challenges, including Various approaches to EAI have been implemented, including the lowest common denominator approach, atomic approach and bridged approach. However, EAI is based on communication between individual applications. A significant challenge is that the complexity of the EAI solution increases geometrically with the linear addition of platforms and applications.

  While data integration systems provide useful tools to address enterprise needs, such systems are typically deployed as customer solutions. Such a system involves a long development cycle and may require advanced technical training to respond to changes in business structure and information requirements. There remains a need for data integration system tools that allow the use, reuse, and change of functionality in a changing business environment. One such tool is a semantic identifier that allows an item to be uniquely identified based on relationships with other items without requiring additional data. A conversion engine is a tool that can convert data, metadata, semantic identifiers, and other items from one format, language, and / or data model to another. Ultimately, the abstract property level of the hub or database allows the distinction between multiple instances or types of items.

  There can be a semantic identifier for the item. Item is object, data item, data, column, row, table, database, instance, attribute, metadata, concept, topic, subject, semantic identifier, other identifier, RFID tag, vendor, supplier, customer, person Teams, organizations, users, networks, systems, equipment, families, stores, products, production lines, product characteristics, product specifications, product attributes, prices, costs, material specifications, shipping data, tax data, courses, educational programs, It can be a location, map, department, organization, organic organization, process, rule, law, rating system, product, service and service offering, or other item or concept. An item can be associated with a data integration job and / or a data integration platform. A semantic identifier can identify an item based on the relationship between the item and one or more other items. A relationship can also be the absence of a relationship. The relationship can be based on meaning. The relationship can include the position of the item in the relationship hierarchy.

  The semantic identifier can be a unique identifier for the item. It is possible to consider a relationship where the number of unique semantic identifiers for an item is less than all relationships between the item and other items. In order to ensure uniqueness, it is advantageous to create semantic identifiers based on a minimum number of relationships. The number of relationships required to create a unique semantic identifier for an item can vary from context to context. The semantic identifier can be context dependent. The semantic identifier can be dynamic.

  Semantic identifiers are stored, maintained, recorded, processed, and / or interpreted in a syntax that can be stored, maintained, recorded, processed, and / or interpreted in a string structure or format. can do. The syntax and / or string structure or format can be parsed. The syntax and / or string structure or format can be truncated, modified, shortened, parsed, or rearranged. It is possible to truncate, modify, shorten, or rearrange the syntax and / or string and still retain the semantic identifier. In certain contexts, shorter syntaxes and / or strings are useful and can increase performance.

  A semantic identifier is a step in an enterprise method, data in a database, data in a row or column, row or column in a table, row or column in a database, data in a table, table in data, meta in database Data, items in the hub or repository, items in the database, items in the table, items in the column, items in the row, people in the organization, sender or receiver of the communication, users on the network, systems on the network Devices on the network, people in the family, items in the store, dishes on the menu, products in the production line, products in the product offering, courses or steps in the educational or training program, location on the map, item Location, organizational unit, team person, rules in the rules system, services Services in the business suite, entities in the corporate organizational hierarchy, entities in the supply chain, customers in the market, purchasers in purchase decisions, prices of goods or services, costs of goods or services, manufacturing or system components, methods And / or a semantic context such as a member of a group.

  In an embodiment, the database may have a table with columns. The unique semantic identifier for the column can be “column name of table name of database name”. This unique semantic identifier is stored, maintained, recorded, processed, and / or interpreted using the following syntax: “column name :: table name :: database name”. The syntax and / or any associated string can be parsed and unnecessary elements can be removed. For example, if there is only one database, the following syntax can still generate a unique identifier for the column :: column name :: table name. Database relationships are not required to create unique semantic identifiers. In another example, since the database can have only one table, the following syntax can be a unique identifier for column :: column name :: database name. Table relationships are not required to create unique identifiers. By using shorter syntax and / or strings, the number of operations is reduced and efficiency is increased.

  The conversion engine can perform conversion operations on one or more semantic identifiers, databases, databases containing semantic identifiers, information systems, semantic identifiers, or information systems including other items. The conversion operation may convert or otherwise modify the format, language, and / or data model of the semantic identifier. A conversion operation is a conversion or conversion between one or more data tools, languages, formats, and / or data models, and at least one other data tool, language, format, and / or data tools. Mapping can be included. For example, the conversion operation is DataStage7, QualityStage, BusinessObject, IBM-DB2 CubeViews, UML1.1, UML1.3, ERSStudio, ProfileStage, PowerDesigner (Packages and ExtendedAttached support) Conversions to and from MicroStrategies can be included. The conversion engine and / or the conversion operation can optionally be implemented in a metabroker. The conversion engine, mapping of the conversion operation, or conversion operation can trace data that is converted back and forth between the original semantic context and the converted semantic context in the execution of the operation. Conversion operations can be performed, performed, and / or performed in batch, in real time, or continuously. The conversion operation can be provided or made available as a service, eg, as part of a service-oriented architecture.

  If there is a conversion operation for a semantic identifier, a database, a database that includes one or more semantic identifiers, an information system, an information system that includes one or more semantic identifiers, or other items, this conversion operation may be designated as any other meaning. An identifier, a database, a database containing one or more semantic identifiers, an information system, one or more semantic identifiers, or other items that share at least one conversion operation, map to, and bind to Can be used with, or associated with, this.

  Items can exist in many forms or instances, such as physical modeling activities and / or logical modeling activities. In a database and / or hub, items that contain any relevant data or metadata can exist in many forms or instances. To distinguish between different types or instances of an item, the level of abstraction, position in the hierarchy, relationship to other items, one or more distinct attributes of the item, the context in which the item is found, the physical location in which the item is found Any distinguishing property can be used, such as.

  In one embodiment, an item such as a table named “employee” may be placed in the hub. A hub collector can have two types or instances of “employees” within the hub, one corresponding to a physical database instance and the other corresponding to a logical modeling activity. The abstract property level of hub data collection allows a distinction between a physical model and a logical model instance or form.

  When performing a transformation operation that can be responsive to a query, the transformation engine can grab, load, or obtain all of the items from the hub or database. The transformation engine filters, selects, stores, and transforms items based on distinctive properties such as abstraction level, position in hierarchy, relationship to other items, item attributes, physical location, etc. Can be modified, manipulated, or otherwise manipulated. Alternatively, when performing a transformation operation that can be responsive to a query, the transformation engine filters, selects, and stores items that contain any data and / or metadata at the hub or database. Can be converted, modified, or otherwise manipulated to grab or acquire only items with an associated level of abstraction or with associated attributes, positions, relationships, positions, etc. it can. Filtering, selection, storage, conversion, modification, or other operations can be performed at run time and design time, and can be performed in batch, in real time, or continuously. In an embodiment, the filtering, selection, storage, transformation, modification, or other operation is a transformation engine at development time, design time, or runtime, such as a data model, data model mapping, identifier syntax distinguishing characteristics, etc. And / or based on information or input obtained by the system. Information can be updated dynamically in real time. Thus, in one preferred embodiment, the system selects from a database based on a well-known mapping of the database, such as selecting a logical item and omitting a physical item, or selecting a physical item and omitting a logical item. The selection command for selecting data can be refined.

  In some cases, the closer the filtering, selection, or other operation is to the hub or database, the more efficient and faster the operation is throughout the process. The transformation engine can perform a transformation operation on the query itself, resulting in a revised query or select command that can be sent directly to the hub or database. The revised query or select command can be in a format that is directly compatible with the hub or database.

  In other aspects, a computer program product can include a computer-usable medium that includes computer program code, where the computer-readable program code when executed on one or more computers. To perform any one or more of the above methods.

  As used herein, “International Business Machines” or “IBM” refers to International Business Machines Corporation, Armonk, NY.

  As used herein, “data source” or “data target” is the broadest consistent with these terms, unless a specific meaning is otherwise indicated or the context of the phrase is otherwise required. Intended to be meaningful, database, multiple databases, repository information manager, queue, message service, repository, data mechanism, data storage mechanism, data provider, website, server, computer, computer storage Mechanism, CD, DVD, mobile storage mechanism, central storage mechanism, hard disk, multiple coordinated data storage mechanism, RAM, ROM, flash memory, memory card, temporary memory mechanism, permanent memory mechanism, magnetic tape, local connection computing Networking mechanism, remote connection computing mechanism, wireless mechanism, wired mechanism, mobile mechanism, central mechanism, web browser, client, laptop, personal digital assistant ("PDA"), telephone, mobile phone, mobile phone, information platform, Analysis mechanism, processing mechanism, business enterprise system or other mechanism or data or other information processing data and structured or unstructured data or any streaming data used in any of the above systems , Other mechanisms adapted to store any file or file type to hold messaged data, event driven data or source data, and any combination of the above. A storage mechanism is a mechanism that can function as some physical or logical device, resource, or data source or data target, or otherwise store data in a searchable format.

  “Enterprise Java® Bean (EJB)” includes a server-side component architecture for the J2EE platform. EJB supports fast and easy development of distributed Java applications, transactional Java applications, secure and portable Java applications. EJB supports a container architecture that allows concurrent processing of messages and supports distributed transactions, so database updates, message processing, and connections to enterprise systems using the J2EE architecture are in the same transaction context. It becomes possible to get involved.

  “JMS” refers to Java® Message Service, an enterprise message service for Java®-based J2EE enterprise architecture. “JCA” means J2EE Connector Architecture of the J2EE platform described in more detail below. EJB, JMS, and JCA are software tools commonly used in modern distributed transaction environments, but any platform, system, or architecture that provides similar functionality is described herein. Note that it can be used with.

  As used herein, “real time” includes time intervals close to business transactions or business durations, as opposed to those performed offline, such as batch processing operations performed at night. Or with processes or services that take place during the business process. Depending on the duration of the business process, real time may include seconds, moments, minutes, hours, or even days.

  As used herein, “business process”, “business logic” and “business transaction” include sales, marketing, fulfillment, inventory management, pricing, product design, professional services, financial services, management, Finance, underwriting, analysis, contracts, information technology services, data storage, data mining, information distribution, product routing, scheduling, communications, investment, transactions, provision, promotion, advertising, bids, engineering, manufacturing , Supply chain management, personnel management, data processing, data integration, workflow management, software generation, hardware production, new product development, research, development, strategic functions, quality control and assurance, packaging, logistics, customer relationship management , Rebates and returns processing, customer support Theft, product maintenance, telephone solicitation, corporate communications, including investor relations activities, the present invention is not limited to these, any method can be performed by the company, service, is intended to include operational, process, or the transaction.

  As used herein, “service oriented architecture (SOA)” includes services that form part of an enterprise's infrastructure. In SOA, services can be a building block for application development and deployment that enables rapid application development and avoids redundant code. Each service can embody a set of business logic or business rules that can be coupled to the surrounding environment, such as a source of data input for the service or a target of data output for the service. Various examples of SOA are provided in the following description.

  As used herein, “metadata” refers to data that introduces context into the data being processed, data about the data, information about the context of the related information, information about the origin of the data, information about the location of the data, and the meaning of the data Information, information about the elapsed time of the data, information about the headings of the data, information about the units of the data, information about the fields of the data, and / or information about any other information related to the context of the data.

  As used herein, “WSDL” or “Web Service Description Language” is a network service (often a web service) as a set of endpoints that operate on messages that contain either document-oriented or procedure-oriented information. XML format for describing (service). Operations and messages are described abstractly and then combined into a specific network protocol and message format to define the endpoint. Related concrete endpoints are combined into abstract endpoints (services). WSDL is extensible to allow the description of endpoints and their messages regardless of which message format or network protocol is used for communication.

  A “metabroker” as used herein can include a system or method that can include a conversion engine or other means for performing data or metadata conversion operations or other operations. Transformation operations and other operations can include the transformation of data or metadata from one or more formats, languages, and / or data models to one or more formats, languages, and / or data models.

  Throughout the following description, unless otherwise indicated, numbers for like elements are intended to refer to like elements.

  The invention disclosed herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In preferred embodiments, the present invention is implemented in software, including but not limited to firmware, resident software, microcode, and the like.

  Furthermore, the present invention provides a computer program product accessible from a computer usable or computer readable medium that provides program code for use by or in connection with a computer or any instruction execution system. Can take form. For purposes of this description, a computer usable or computer readable medium includes, stores, communicates, propagates, or transfers a program for use by or in connection with an instruction execution system, apparatus, or the like. It can be any device that can.

  The medium can be an electronic system, a magnetic system, an optical system, an electromagnetic system, an infrared system, or a semiconductor system (or apparatus or device) or a propagation medium. Examples of computer readable media include semiconductor memory or solid state memory, magnetic tape, removable computer diskette, random access memory (RAM), read only memory (ROM), magnetic hard disk and optical disk. Current examples of optical disks include CD-ROM, CD-R / W and DVD.

  A data processing system suitable for storing and / or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory element has at least some local memory used during actual execution of the program code, mass storage, and at least some so as to reduce the number of times code must be obtained from the mass storage during execution. And cache memory providing a temporary storage location for program code.

  Coupling input / output devices or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) to the system either directly or through intervening I / O controllers Can do.

  Network adapters can also be coupled to the system so that the data processing system can be coupled to other data processing systems or remote printers or storage devices through private or public networks. Modems, cable modems and Ethernet cards are some of the currently available types of network adapters.

  FIG. 1 represents a platform 100 for facilitating the integration of various enterprise data. The platform includes a plurality of business processes, each of which can include a plurality of different computer applications and data sources. The platform can include a number of data sources 102 that can be data sources as described above. These data sources can include various data types from various physical locations. For example, the data source can be provided from a system that can include a provider such as Sybase, Microsoft, Informix, Oracle, Inlover, EMC, Trillium, First Logic, Siebel, PeopleSoft, IBM, Apache, or Netscape. Data sources 102 may include systems that use database products or standard technologies such as IMS, DB2, ADABAS, VSAM, MD Series, UDB, XML, composite flat files, or FTP files. Data sources 102 include files created or used by applications such as Microsoft Outlook, Microsoft Word, Microsoft Excel, Microsoft Access, and files in standard formats such as ASCII, CSV, GIF, TIF, PNG, etc. be able to. The data source 102 can be located at various locations or it can be centrally located. The data supplied from the data source 102 can be in various formats and can have different formats that are compatible or incompatible.

  Data targets are described later in this document, but in general, these data targets can be any of the data sources 102 described above. Such differences in term usage are generally attributed to whether the data system provides or receives data in the data integration process. However, in a typical data integration system, a data source can receive data and a data target can provide data, so this distinction is different from data sources unless otherwise stated. Note that it is not intended to give a difference in capability between data targets.

  The platform shown in FIG. 1 also includes a data integration system 104. The data integration system can facilitate data collection from the data source 102 as a result of, for example, a query or search command received by the data integration system 104. The data integration system 104 can send commands to one or more data sources 102 such that the data source provides data to the data integration system 104. Since the received data can be in a number of formats including various metadata, the data integration system can reconstruct the received data so that it can be later combined for the integration process. it can. The functions that can be implemented by the data integration system 104 are described in more detail below.

  The platform 100 also includes a search system 108. The search system 108 can include a database or processing platform that is used to further manipulate data transmitted from the data integration system 104. For example, the data integration system 104 organizes and combines the data received from the data source 102 so that the search system 108 can use the processed data to generate a business useful report 110; Can be converted or otherwise manipulated. The report 110 can be used to report data relevance, answer complex queries, answer simple queries, or create other reports useful to business or users. The report 110 can include raw data, tables, charts, graphs, and any other representation of data from the search system 108.

  The platform 100 may also include a database or database management system 112. Database 112 can be used to store information as temporary or permanent or long-term storage. For example, the data integration system 104 can collect data from one or more data sources 102 and convert the data into a format that is compatible with each other or that can be combined with each other. Once the data is converted, the data integration system 104 can store the data in the database 112 in a decomposed form, combined form, or other form for later retrieval.

  FIG. 2 is a schematic diagram illustrating data integration between multiple entities and business processes in an enterprise. In the illustrated embodiment, the data integration system 104 facilitates the flow of information between the user interface system 202 and the data source 102. The data integration system 104 can receive data from the interface system 202 to extract and possibly convert data present in one or more data sources 102. The interface system 202 is a data integration system such as a laptop or desktop computer, a mobile phone, a personal information terminal ("PDA"), a networked platform, and a web browser that operates on devices attached thereto. Any device and program for communicating with 104 or any other device or system that interfaces with data integration system 104 may be included.

  For example, a user can operate a PDA to request information from the data integration system 104 via a WiFi or wireless access protocol / wireless markup language (“WAP / WML”) interface. The data integration system 104 can receive the request and generate any necessary queries to access information from other data sources 102 such as websites or FTP file sites. Data from the data source 102 is extracted and converted into a format compatible with the requesting interface system 202 (PDA in this example) and then sent to the interface system 202 for viewing and manipulation by the user. can do. In other embodiments, the data can be pre-extracted from a data source and stored in a separate database 112, which can be a data warehouse or other data device used by the data integration system 104. The data can be stored in the database 112 in the converted state or in its original state. For example, the data can be stored in a transformed state so that data from many data sources 102 can be combined in other transformation processes. For example, a query from a PDA can be sent to the data integration system 104, which can extract information from the database 112. After extraction, the data integration system 104 can convert the data to a combined format compatible with the PDA before returning it to the PDA.

  FIG. 3 is a schematic diagram illustrating an architecture for providing data integration for a plurality of enterprise data sources 102. Embodiments of the data integration system 104 include a data discovery stage 302 that performs extraction of data from a data source and analysis of column values and table structure for the source data (possibly during other processing). Can be included. The data discovery stage 302 can also generate recommendations regarding the table structure, relationships and keys for the data target. More advanced profiling and auditing features can include date range validation, calculation accuracy, if-then evaluation accuracy, and so on. The data discovery stage 302 can normalize the data, such as by eliminating redundant dependencies and other anomalous parts of the source data. The data discovery stage 302 can provide additional functions such as drilling down exceptions within the data source 102 for further analysis or allowing direct profiling of mainframe data. Commercially available forms of data discovery stage 302 include, but are not limited to, IBM's Websphere ProfileStage product.

  The data integration system 104 also includes a data preparation stage 304 that prepares, standardizes, collates, or otherwise manipulates the data to produce quality data that will be converted later. You can also The data preparation stage 304 includes general data quality functions such as adjusting inconsistencies in the data or performing an exact match (including one-to-one matching, one-to-many matching and deduplication). Can be executed. The data preparation stage 304 can also provide specific data enhancement functions. For example, the data preparation stage 304 may ensure that the address meets multilateral postal standards for improved international communications. The data preparation stage 304 can adapt the location data to a multinational geocoding standard for spatial information management. The data preparation stage 304 changes or adds to the address to ensure that the address information can receive a US Postal Service postage discount by a US address correction approved by the US government. be able to. Similar analysis and data correction can be implemented in Canadian and Australian postal systems that offer discounted rates for properly addressed mail. Commercially available forms of data preparation stage 304 include, but are not limited to, IBM's Websphere QualityStage product.

  The data integration system may also include a data conversion stage 308 that converts the converted data and delivers it with enhanced quality. The data conversion stage 308 may perform migration services such as data reconstruction and reformatting and may also perform calculations based on system user business rules and algorithms. Data transformation stage 308 can also organize target data into subsets known as data marts or cubes for more sophisticated reconciliation processing of data in a particular analysis context. Data transformation stage 308 bridges various software and hardware architectures of various data sources and data targets used by data integration system 104 (as described generally below). ) Bridges, translators, or other interfaces can be used. Data conversion stage 308 can include a graphical user interface, a command line interface, or a combination thereof to design data integration jobs across platform 100. Commercially available forms of data conversion stage 308 include, but are not limited to, IBM's Websphere DataStage product.

  The stages 302, 304, 308 of the data integration system 104 can be performed using the parallel execution system 310 in succession or in combination to optimize the performance of the system 104.

  The data integration system 104 can also include a metadata management system 312 for managing metadata associated with the data source 102. In general, the metadata management system 312 can provide metadata exchange, integration, management, and analysis across tools in a data integration environment. For example, the metadata management system 312 is different from IBM's Websphere ODBC MetaBroker, CA ERwin, IBM Websphere ProfileStage, IBM Websphere DataStage, IBM Cosphere Quality Stage, IBM CoVu Quality Stage, IBM DBVu ug Common views can be provided. The metadata management system 312 may also provide analytical tools for data lineage and impact analysis. In addition, the metadata management system 312 can be used to create a business data glossary of data definitions, algorithms, and business context for data in the data integration system 104, which is used throughout the enterprise. Can be made public. A commercially available form of the metadata management system 312 includes, for example, IBM's Websphere MetaStage product, but is not limited thereto.

  Referring to FIG. 4, items associated with a company can be described in terms of various contexts or hierarchies, such as to obtain the semantic context of the item. Thus, FIG. 4 shows the semantic identifier for the item. Items can be other items or concepts including objects, classes, attributes, data items, data models, models, definitions, identifications, structures, languages, mappings, relationships, instances and other semantic identifiers. A semantic identifier can identify an item based on the item's attributes, the physical location of the item, the relationship between the item and one or more other items in the hierarchy, and the like. In some cases, a relationship can be defined as the absence of some particular relationship. The relationship can be based on meaning. The relationship can include the position of the item in the relationship hierarchy. For example, in FIG. 4, item 1 5202 can be identified based on relationships with other related items. Item 1 5202 is directly related to item 2 5204, item 3 5208 and item 4 5210, indirectly related to item 5 5212, and to item 6 5214 via item 5 5212 and item 4 5210. It can be identified as indirectly related. Item 1 can also be identified as directly related to item 2 5204, item 3 5208, and item 4 5210. In an embodiment, the indirect relationship between item 1 5202 and item 5 5212 and item 6 5214 can be obtained in the relationship between item 1 5202 and item 4 5210. This identification of concatenation type or recursion type allows the realization of dynamic identifiers in addition to static identifiers. For example, if the relationship between item 4 5210 and item 6 5214 changes, the semantic identifier for item 1 5208 incorporating item 2 5204, item 3 5208, and item 4 5210 will change this through the incorporation of item 4 5210. Incorporation, there is no need to make an update to account for changes in item 6 5214 as if item 6 5214 was included directly within the semantic identifier.

  FIG. 5 shows a more specific example of the semantic identifier. Jim can be identified as a Jim who resides in Sakai Street, Sakaimachi, U.S. 111, USA and has a telephone number 555-555-5555 and a social security number 012-34-5678. Alternatively, Jim can be identified in terms of relationships with others. As shown in FIG. 5, Jim can be identified as Betty's son, Larry and Jeff's brother, Jessica's father, and Frank's nephew.

  The semantic identifier can be a unique identifier for one item. In the example of FIG. 5, if there is only one person in the world, Betty's son, Larry and Jeff's brother, Jessica's father, and Frank's nephew, this semantic identifier is a unique identifier for Jim. It is also possible to consider the case where the unique semantic identifier for an item is less than all of its relationships with other items. If there is only one Jim in the world, Betty's son, Larry's brother, and Jessica's father, the existence of these relationships is sufficient to create a unique semantic identifier. There is no need to consider the relationship between Jim and Jeff and Frank. It is advantageous to create semantic identifiers based on a minimum number of relationships that guarantee uniqueness. For example, if semantic identifiers are stored in the database 112 or processed by the data integration system 104, less complex semantic identifiers require less space and allow faster processing.

  The number of relationships required to create a unique semantic identifier for an item can vary based on context. FIG. 6 shows two items of interest, item 1 5402 and item 7 5404. In context A 5408, item 1 5402 can be distinguished from item 7 5404 by the relationship between item 1 5402 and item 5 5410 and item 6 5412. That is, in context A, the unique semantic identifier for item 1 5402 is directly related to items 2, 3 and 4, indirectly related to item 5 5410 through item 4, and item 6 through item 5 5410 and item 4 5412 may be indirectly related. In context A, the unique semantic identifier for item 7 5404 may be directly related to items 2 and 3 only. FIG. 7 shows item 1 5402 in a different context, context B 5414. In order to uniquely identify item 1 5402 in context B 5414, the direct relationship of item 1 5402 with item 4, the absence of a direct relationship with item 6, or indirect with item 5 Any one or more of the relationships can be considered. In context B 5414, item 1 5402 can be uniquely and semantically identified as directly related to items 2 and 3, but not directly related to item 6. Thus, the unique identifier for item 1 is different for context A 5408 and context B 5414. Thus, in the data integration method and system embodiments described herein, a semantic identifier for an item, such as an item associated with a data integration job or data integration platform, is given a context sensitive identifier for that item. be able to. In embodiments, such context sensitive identifiers can be stored in an atomic format, such as in a data repository.

  In other embodiments, Context A 5408 and Context B 5414 are two different imports, mappings, execution versions, models, metabroker models, instances, tools, views, objects, classes, items, relationships, attributes, or the above Any combination can be used. The matching or comparison mechanism should compare the syntax of identifying items in different imports, execution versions, models, metabroker models, instances, tools, and / or items, and what action to take based on the comparison A determination as to whether or not to take action can be determined. For example, the matching engine can compare the model used by Import Instance A with the model used by Meta Broker B. Based on this comparison, it is determined that Metabroker B can access the data and metadata of Import Instance A without conversion or modification, and that the comparison mechanism can instruct MetaBroker B to continue. . In another example, tool A 5408 can be compared to tool B 5414, and execution of object merging between tools can be determined, with each tool accessing and using objects in other tools. . In an embodiment, the comparison mechanism triggers the conversion mechanism to convert based on different syntax for the processing of identification of a particular item in each of the respective tools, or other between tools determined by comparison. It can help merge objects between tools such as establishing bridges, metabrokers, hubs, etc., to help transform any object that needs transformation, such as transformation based on differences.

  In an embodiment, the semantic identifier is stored, maintained, recorded, processed in a syntax that can be stored, maintained, recorded, processed, and / or interpreted in a string structure or format, And / or can be interpreted. FIG. 8 shows an example of a syntax and a corresponding character string constructed in the syntax. The syntax 5502 can be a column name :: table name :: database name. This syntax can be associated with, for example, a semantic identifier that identifies a column of a table in the database. The string constructed in this syntax 5504 can be age :: employee :: employee database. This string can be associated, for example, with a semantic identifier that identifies the age of the employee in a particular employee database. In the example of FIG. 7, the character string corresponding to the semantic identifier for item 1 5402 in context B is a direct relationship with item 2 :: a direct relationship with item 3:: a direct relationship with item 4. Relationship can be. The semantic identifier and the corresponding string can also incorporate the lack of a direct relationship between item 1 5402 and item 6.

  In FIG. 9, the semantic identifier in string format for item 9 5602 is directly related to item 2 :: directly related to item 4 :: indirectly related to item 5 5604 Can be. The string can be parsed. The syntax and / or string can be truncated and modified, and / or the elements of the syntax and / or string can be rearranged. In FIG. 10, a character string 5702 is a character string 5604 truncated, a character string 5704 is a character string 5604 truncated, modified, and / or rearranged, and a character string 5708 is a character string 5606 has been modified and / or rearranged. The conversion engine can perform truncation, correction, and / or rearrangement. Because of the uniqueness of semantic identifiers, it is useful to truncate the syntax and / or string when you do not need all the relationships contained within the syntax and / or string. Assume that in the given context of string 5604, all items are directly related to item 3, ie, for example, item 3 was a database that stores all items. The character string 5604 can be truncated to leave a unique semantic identifier, such as creating a character string that omits the relationship involving item 3. Syntax and / or string truncation can reduce storage requirements and increase processing efficiency. For example, it may be useful to change the order of the relationships in the syntax and / or strings to reduce processing time for the data integration process. If a less common relationship is processed first, the system will likely need to access and process fewer relationships associated with the item to identify the item. For example, if there are few items associated with item 3, few are associated with item 4, and many items are associated with item 2, depending on the context, string 5708 may be shorter than string 5604. It becomes possible to identify the item 9. It may be that only the first two elements of string 5708 are needed and the first three elements of string 5604 are needed to uniquely identify item 9 in context.

  The conversion engine may perform a conversion operation on one or more semantic identifiers, a database 112, a database 112 including semantic identifiers, an information system, an information system including one or more semantic identifiers, or other items. FIG. 11 shows a conversion engine 5802 that operates on a semantic identifier embodied as a character string 5804 and a semantic identifier embodied as a character string located in the database 5808. The conversion operation may convert or otherwise modify the format, language, and / or data model of the semantic identifier. A transformation operation is a transformation or mapping between one or more data tools, languages, formats, and / or data models, with at least one other data tool, language, format, and / or data tools. Conversions or mappings between can be included. For example, the conversion operations are: WebSphere Data Stage7 from IBM, WebSphere Quality Stage from IBM, Business Object Tool for Business, IBM-DB2 Cube Views, UML1.1, UML1.3, ERSStudio, WebPier DataStage. And / or conversion or mapping to, or between, well-known data integration tools, such as MicroStrategie tools. The conversion engine and / or the conversion operation can optionally be embodied in a metabroker. Conversion operations can be performed, performed, and / or performed batch, real-time or continuously. The conversion operation can be provided or made available as a service, eg, as part of a service-oriented architecture. The SOA can be part of the infrastructure of a business enterprise enterprise computer system. In SOA, services become a building block for application development and deployment, enabling rapid application development and avoiding redundant code. Each service embodies a set of business logic or business rules that are independent of the surrounding environment, such as a data input source for the service or a data output target for the service. As a result, the service can be reused with various applications if appropriate inputs and outputs are established between the service and the application. A service-oriented architecture allows services to be protected against environmental changes, so the architecture works even if the surrounding computing environment changes. As a result, there is no need to record services as a result of infrastructure changes, which saves time and effort. The SOA can be for web services and can include three entities: a service provider, a service requester, and a service registry. The registry may be public or private. The service requester can search the registry for an appropriate service. Once an appropriate service is found, the service requester can receive code, such as Web Services Description Language (“WSDL”) code, necessary to invoke the service. WSDL is a programming language traditionally used to describe web services. The service requester then invokes the message in an appropriate format (such as the Simple Object Access Protocol (“SOAP”) format) for web service messages. The service provider can be connected through such as. The SOAP protocol is the preferred protocol for transferring data in web services. The SOAP protocol defines a message exchange format between a web service client and a web service server. The SOAP protocol uses an eXtensible Markup Language (“XML”) schema, which is a common language specification commonly used in web services for tagging data, but uses other markup languages You can also

  If there is a conversion operation for the semantic identifier, the database 112, the database 112 including one or more semantic identifiers, the information system, the information system including one or more semantic identifiers, or other items, this conversion operation is Conversion between the semantic identifier of the database 112, the database 112 including one or more semantic identifiers, the information system, the information system including one or more semantic identifiers, or other items sharing at least one conversion operation. Can be mapped to, combined with, used with, or associated with. In an embodiment where an atomic data repository such as a hub is used for the transformation operation, the mapping of the transformation operation includes, among other things, the original semantic context and the transformed semantic context in performing the operation. Data that is converted back and forth between can be traced. Depending on the context, the syntax and / or string may be altered or truncated to allow more efficient storage or faster processing, or the relationship used to form a unique identifier where the semantic context changes The appropriate identifier of the data item may change. Thus, dynamic identifiers can combine the benefits of retraceable transformations with the advantages of fast processing, efficient data processing, and efficient manipulation in the various contexts in which the data items are used.

  A given item, such as an item having an identity in the model, can exist in many forms or instances, such as physical instances and logical modeling instances. FIG. 12 shows a table of items, ie employee information 5902. However, the concept or entity “employee” may exist in many different forms within an enterprise. For example, employee table 5902 may exist as a physical table that stores values associated with employees in a physical data storage facility. On the other hand, entity employees may also be represented as logical entities, such as icons or text representing employees in logical modeling activities 5908 or in various other forms or instances. That is, the same item, including any related data or metadata, exists in various forms or instances across views, models, structures, or data integration environments, for example in databases, data repositories, models, hubs, etc. Can do. FIG. 13 shows an employee table 5902 in one form or single instance in database 6002 and / or in two or more forms or instances in database 6004 or hub 6008.

  To distinguish between different types or instances of items, the level of abstraction, item physical properties, item location in the hierarchy, item location in the database, the context in which the item is found, item syntax, item and other Some distinguishing characteristics can be used, such as item relationships, item attributes, item classes, or other characteristics. For example, referring again to FIG. 5, it is possible to distinguish items, or individuals in this case, based on age, gender, hair color, IQ, political affiliation, and / or the number of times the doctor has been seen in the last three months. it can. For example, if age is selected as a product distinguishing factor, Jessica is the only individual under 10 years old, Betty is the only individual from 57 to 67 years old, Jim is 37 years old There is only one individual. In other examples, different types or instances of items can exist at different levels of abstraction or in different contexts. For example, the employee table may be a logical employee table 5904, such as used to store values in a database related to employee-related data, and logical processes such as those used in connection with employee-related processes. It can exist in numerous forms or instances within the hub 6102, such as a unique employee model 5908.

  Differentiating between different instances of a particular item identified allows for various other methods and processes. For example, in one embodiment, an item such as a table named “employee” may be placed in the hub. A hub collector can have two types or instances of “employees” within the hub, one corresponding to a physical database instance and the other corresponding to a logical modeling activity. Differentiating characteristics, such as the properties of items attributed to items in the hub, allow a distinction between physical instances and logical model instances or forms. In the embodiment, the distinction characteristic may be a so-called abstraction level, such as for distinguishing between a logical level and a physical level of abstraction. In other cases, the hub may associate other characteristics with items such as different types of identifiers, relationships, classes, attributes, physical locations, logical locations, models, and the like.

  As shown in FIG. 15, when performing operations such as selecting data to be loaded into a database, transforming data, generating queries, etc., a system such as the transformation engine 6204 can be accessed from a hub 6208 or a database 6210. All items can be grabbed, loaded, or acquired. The system can select or filter items based on some distinguishing characteristic. For example, the system has a physical abstraction level, has a specific relationship with other items, has a logical abstraction level, was created before a specified date and time, or any other Instances or types that have distinct characteristics can be selected or filtered. Thus, the methods and systems described herein provide for selective processing of instances of the same item or entity based on any distinguishing characteristic.

  As shown in FIG. 16, when performing a data integration operation, such as a conversion operation, that may be responsive to a query 6202, the conversion engine 6204 may receive any data and / or data in the hub 6208 or database 6210. Or items that contain metadata can be filtered or selected, and only those items at the relevant level of abstraction can be grabbed, loaded, or acquired. For example, the transformation engine can filter or select instances or forms that have a logical abstraction level, and keep only those that have a physical abstraction level. Filtering or selection can be done at run time and design time, and can be done in batch, in real time, or continuously. In an embodiment, such methods of filtering and selection can be provided as RTI services in a service-oriented architecture.

  Filtering or selection is acquired by the transformation engine and / or system at development time, design time, or runtime, data model mapping, metadata model mapping, distinctive characteristics, relationship between items and other items, item Based on information such as the attribute or identifier syntax. In embodiments, the information can be updated dynamically in real time.

  The closer the filtering or selection is to the hub or database throughout the process, the more efficient and faster the operation. As shown in FIG. 17, the transformation engine 6204 performs a transformation operation on the query 6202 itself and implements an improved query 6402 that can be sent for further processing, such as sent directly to the hub 6208 or database 6210. Can do. For example, the revised query 6402 can be in a format that is directly compatible with the native format of the hub 6208 or database 6210. For example, by putting the query in the native format of the database 6210, the system can increase the processing efficiency for the query. Similarly, query 6402 can be filtered or commands such as select commands can be generated to hold logical modeling entities rather than physical entities, in which case query 5402 is suitable for the database. Format suitable for logical modeling activities (such as a graphical user interface). Of course, not only queries but also other messages and operations can be filtered according to the level of abstraction so that the same entity can be traced across the data integration platform and processed according to the appropriate operating environment for the particular data integration activity. It is.

  Using the methods and systems described herein, semantic context is captured and objects, data items, data, columns, rows, tables, databases, instances, attributes, metadata, concepts, topics, subject matter, semantic identifiers, Other identifiers, RFID tags, vendors, suppliers, customers, people, teams, organizations, users, networks, systems, equipment, families, stores, products, production lines, product characteristics, product specifications, product attributes, prices, costs, Such as material specifications, shipping data, tax data, courses, educational programs, locations, maps, departments, organizations, organic organizations, processes, rules, laws, evaluation systems, goods, services, and / or service offerings, Data integration tasks can be handled for a wide range of items related to the enterprise.

  The methods and systems described herein include steps in an enterprise method, data in a database, data in a row or column, rows or columns in a table, rows or columns in a database, data in a table, data in a database Table, metadata in database, item in hub or repository, item in database, item in table, item in column, item in row, person in organization, sender or receiver of communication, on network Users, systems on the network, devices on the network, people in the family, items in the store, dishes on the menu, products in the production line, products in the product offering, courses or steps in the educational or training program, maps Top position, item position, organizational department, team person, Rules in the law system, services in the service suite, entities in the corporate organizational hierarchy, entities in the supply chain, customers in the market, buyers in purchase decisions, prices of goods or services, costs of goods or services, manufacturing or systems Can be used in a variety of semantic contexts, such as components, method steps, group members, or many others.

  Although the invention has been described with reference to certain preferred embodiments, it is noted that other embodiments are recognized by those skilled in the art and are intended to be included within the scope of the invention.

1 is a schematic diagram of a business enterprise having multiple business processes, each of which can include multiple different computer applications and data sources. 1 is a schematic diagram illustrating data integration across multiple business processes of a business enterprise. FIG. 1 is a schematic diagram illustrating an architecture for providing data integration of multiple data sources to a business enterprise. FIG. Show items in relation to other items. Show items in relation to other items. Indicates an item in a specific context. Indicates an item in a specific context. Indicates a specific string. Indicates an item and the corresponding string. Indicates a string and a specific variant. Indicates a conversion engine that operates on a specific string. Indicates an item that can exist in multiple forms or instances. Indicates an item that can exist in multiple forms or instances in a hub or database. Fig. 4 illustrates items at various levels of abstraction in the hub. Shows the conversion process where all items are grabbed in the database or hub. Fig. 4 illustrates a conversion process in which all items are filtered in a database or hub. Indicates the conversion process in which the query is converted.

Claims (35)

Providing a semantic identifier for identifying the item based on relationships with other items;
Obtaining a mapping of the data model to enable determination of the semantic identifier for an item in the data model;
Associating the mapping with a data integration function performed based on at least one of a mapping and the semantic identifier;
including,
A method for data integration.   The item is an object, data item, data, column, row, table, database, instance, attribute, metadata, concept, topic, subject, identifier, semantic identifier, RFID tag, vendor, supplier, customer, person, Team, organization, user, network, system, equipment, family, store, product, production line, product characteristics, product specification, product attribute, price, cost, material specification, shipping data, tax data, course, education program, location The method of claim 1, comprising one or more of: a map, a department, an organization, an organic organization, a process, a rule, a law, a rating system, a product, a service, and a service offering.   The method of claim 1, wherein the relationship includes an item's position in a relationship hierarchy.   The method of claim 1, wherein the semantic identifier is a unique identifier for an item.   The method of claim 1, wherein the semantic identifier is based on a sufficient number of relationships for the identifier to be less than all relationships with other items of the item.   The method of claim 1, wherein the semantic identifier is based on a minimum number of relationships necessary for the identifier to be unique.   The method of claim 1, wherein the semantic identifier is a context sensitive identifier for an item.   The method of claim 1, wherein the semantic identifier is stored in an atomic format.   The method of claim 1, wherein the semantic identifier is stored in a data repository in an atomic format.   The method of claim 1, wherein the semantic identifier is dynamic.   The method of claim 1, wherein the semantic identifier varies with context. A method for performing a data integration process,
Associating a model with a data set;
Forming a selection command for selecting an item from the data set based on a distinguishing characteristic for the item determined from the model;
including,
Method.   13. The method of claim 12, wherein the formation of the selection command / query is performed during execution of a process that uses the selection command / query.   13. The method of claim 12, wherein the forming of the selection command / query is performed at the time of designing a process that uses the selection command / query. A method for performing a data integration process,
Associating a model with a data set;
Forming a query for querying the data set based on a distinguishing characteristic for the item determined from the model;
including,
Method.   The method of claim 15, wherein the formation of the selection command / query is performed during execution of a process that uses the selection command / query.   16. The method of claim 15, wherein the forming of the selection command / query is performed at the time of designing a process that uses the selection command / query. A system for data integration,
A semantic identifier to identify the item based on its relationship to other items,
Mapping of the data model to enable determination of the semantic identifier for items in the data model;
A mechanism for associating the mapping with a data integration function performed based on at least one of the mapping and the semantic identifier;
A system comprising:   The system of claim 18, wherein the relationship includes a position of an item in a relationship hierarchy.   The system of claim 18, wherein the semantic identifier is a unique identifier for an item.   19. The semantic identifier is based on a number of relationships that is less than all relationships with other items of the item but sufficient to ensure that the identifier is unique. System.   The system of claim 18, wherein the semantic identifier is based on a minimum number of relationships that ensure that the identifier is unique.   The system of claim 18, wherein the semantic identifier is a context sensitive identifier for an item.   The system of claim 18, wherein the semantic identifier is stored in an atomic format.   The system of claim 18, wherein the semantic identifier is stored in a data repository in an atomic format.   The system of claim 18, wherein the semantic identifier is dynamic.   The system of claim 18, wherein the semantic identifier varies with context.   The semantic identifier recursively obtains an indirect relationship to the second item by obtaining a direct relationship to the first item that has a direct relationship to the second item; The system of claim 18.   The system of claim 18, wherein the semantic identifier is obtained as a string and the string is truncated if not all elements are required for a unique identifier.   The system of claim 18, wherein the data integration function is a conversion operation.   31. The system of claim 30, wherein the conversion operation modifies one or more of the format of semantic identifiers, the language of semantic identifiers, and the data model of semantic identifiers.   31. The system of claim 30, wherein the mapping of the conversion operation can trace data that is converted back and forth between the original semantic context and the converted semantic context in performing the operation.   The system of claim 18, wherein the conversion operation is provided as a service in a service oriented architecture.   The system of claim 18, further comprising a filter for selectively filtering instances of logical entities based on the distinguishing characteristics of the entities.   35. The system of claim 34, wherein the distinguishing characteristic is obtained from at least one of the mapping and the semantic identifier.

Images