AU2011213842B2 - A system and method of managing mapping information - Google Patents

Abstract

A centralized version managed system and method for managing centralized mapping specification by applying customized templates and mapping rules which will help to maximize subsequent reuse of mapped information in data migration or data archival 5 projects. Built in workflows and well defined process flow ensure all time compliance to mapping process thereby improving the data quality and reducing efforts and time involved in transformation of legacy source data to target data entities. Progress trackers help in tracking the progress of the mapping process almost real-time. Create and manage Project Create and manage users and project administrators Create and manage global data sources 5 - Create/Import and manage global compatibility rules Create and manage Project oles/Permissions -*Create and manage Project users Create/Import and manage project data sources - Create/Import and manage compatibility rules - Configure Market, LoB and business entity Create/Import and manage source/target Systern invent or y Create/import and manage DB Table & Column Info 10 - Create & manage metadata, data analysis defns Capture Mapping Decisions * Record mapping conditions Capture mapping definitions, rules & relationships - Review mapping information - Approve mapping information - Capture tacit knowledge around Inventory/mapping as Q&A - Monitor& govern progress - Derive relationships between business entities, source & target inventory in graphical form - Generate & export reports for further analysis * Generate & export eports for further reuse in the next phases. Fig.2

Description

- 1 AUSTRALIA PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT ORIGINAL Name of Applicant/s: Tata Consultancy Services Limited Actual Inventor/s: Ragunathan Revathi and Javalirao Akshata and Vijayanathan Ajitha and Ramakrishnan Ramesh Kumar and Surendra Babu Muruga and Mysore Raghavendra Address for Service is: SHELSTON IP 60 Margaret Street Telephone No: (02) 9777 1111 SYDNEY NSW 2000 Facsimile No. (02) 9241 4666 CCN: 3710000352 Attorney Code: SW Invention Title: A system and method of managing mapping information The following statement is a full description of this invention, including the best method of performing it known to me/us: File: 71501AUPOO -2 A SYSTEM AND METHOD OF MANAGING MAPPING INFORMATION FIELD OF THE INVENTION: The invention relates to a structured, version managed and centrally governed web based 5 solution for supporting and managing data mapping between desperate databases from varied platforms by applying customized templates, in built workflows and set of defined rules. Embodiments of the invention more particularly relate to maximize subsequent reuse of the mapping information and data relationships managed by the version system for data migration/ archival projects. 10 BACKGROUND OF THE INVENTION: Any discussion of the prior art throughout the specification should in no way be considered as an admission that such prior art is widely known or forms part of common general knowledge in the field. In data warehousing and large data migration programs, different entities and different 15 lines of business are mapped and analyzed by different teams. The outputs of the teams are merged in Excels or in MS- Access databases to create different mapping views in later phases. The current approach of managing the mapping information however poses several difficulties. Some programs may encounter complexities due to poor manageability and 20 various other issues. These issues may be related to the effort intensive nature of the program and may also result in loss of information due to non persistent character of the captured mapping information. This also bears the high risk of being overridden which leads to increased loss of information. Further, the current approach does not offer a holistic view of the data relationships as 25 information is often distributed and constant rework and extensive effort is expended in manually integrating the information to determine the data relationships in broader context, in verifying the mapping information and establishing relationships between the entities offline which may be prone to defects. More particularly, manual offline process for post mapping analysis may be a failure which requires specialized skill sets and -3 resolving complicated redundancies and inconsistencies. The lack of up-to- date collective and consistent information of the systems further hampers reuse of information in further phases. Moreover, the information managed in excels and other offline mechanisms pose the 5 risk of different users working on different versions or information recorded in different forms owing to complicated and inconsistent distributed mechanism. In the same respect, reviews, testing and driving integrated test scripts or cases to validate the ETL (Extraction, Transformation and Loading) process is a tedious, time consuming process which accordingly should be substituted by a solution to maximize reusability of 10 mapping information and hence contribute to the improved productivity of data migration programs. Non unified view for assessing the progress of mapping may lead improper standardization of legacy data, validating which again will be a tedious task. The existing prior arts also fail to provide personalized or role defined user - friendly 15 information view thus impacting the productivity and reuse of information as there is only a single way of information representation which might not make sense for different users. Further it is noteworthy that tracking the progress of data mapping activity apart from governance and control of the process is a challenge in the current approach as it increases the efforts management need to expend in tracking through 20 offline mechanism. Also, the prior art fails to capture the knowledge passed between business, analyst and IT team while performing the data mapping exercise thus resulting in a loss of knowledge and complicating the validation process further apart from hampering reuse of the knowledge. Accordingly, it would be desirable to provide improved technique that addresses data 25 profiling, characterization and migration deficiencies associated with the prior art approaches. For instance, US Patent 5909570 discloses a method and architecture for mapping data between fixed structure datasets employing mapping template which operates from an embedded knowledge of the structure rules for the data being exchanged.

-4 At another instance, US Patent Application No. 20060235899 provides a system and method for migrating legacy database systems to modern database comprises generally of the steps of gathering design information about the legacy database system; analyzing the metadata, data fields, and processes of the legacy system; iteratively creating 5 business objects to represent the migrated data; iteratively associating each of the fields of the legacy database system to one or more of the business objects; creating a data migration script to effect the migration of data; and resolving inconsistencies between the legacy database systems and the one or more target database systems. Therefore, the problems associated in the prior art with unmanageability of data; 10 susceptibility to increased error due to offline manual procedures, lack of unified personalized role and entity based views, constant reworks and involvement of extensive efforts faced by the technical team during the preparatory phases i.e. mapping and designing of data migration project is attempted to be solved by the present invention and thereby provide a system which improves manageability, control, progress tracking 15 or process governance, reutilization of mapped information and associated knowledge and data relationships and ease of information search and retrieval during subsequent data migration or data archival projects. OBJECT OF THE INVENTION: In accordance with an embodiment of the present invention, a structured version 20 managed and centrally governed web based solution for efficiently managing and tracking the data mapping process that captures and derives data mapping rules, specifications, relationships and associated knowledge for subsequent reuse of the mapping information. It is an object of embodiments of the present invention to manage inventory details of 25 source and target data fields and set of data rules governing them. It is yet another object of embodiments of the present invention to graphically depict relationship between source and target data entities. Yet another object of embodiments of the present invention is to configure custom roles, users and permission for the roles.

-5 It is an object of embodiments of the invention to manage market/line of business/business entity wise mapping rules accompanied with extraction, transformation, loading, cleansing, verification and validation, business and reconciliation rules between source and target data fields. 5 It is yet another object of embodiments of the present invention to enable configuration of custom database definitions that can be imported and reused by projects. It is another object of embodiments of the present invention to manage datatype compatibility rules between different databases that can be imported by projects. It is further object of embodiments of the present invention to provide high level 10 validation of rules against set compatibility rules. One of the objects of embodiments of the present invention is to enable the user to use platform independent semantic editor to standardize the definition of the transformation rules across the project for easy and accurate interpretation. Another object of embodiments of the present invention is to provide for built in 15 workflows and well defined process flows to enable quick tracking, assessing and monitoring of the mapping process real time for ensure compliance to the mapping specifications. One of the objects of embodiments of the present invention is to generate reports for identifying any unmapped or incompatible data fields. 20 It is yet another object of embodiments of the present invention to help in isolating and analysis of unmapped or inconsistent data fields for correction and validation of standardized data. One more object of embodiments of the present invention is to ease the information search and retrieval procedure through intelligent search, personalized views, filter and 25 sort mechanisms through which the system provides the analyst with intelligent suggestions on related source fields for every target field to be mapped thereby speeding up the mapping process.

-6 Yet another object of embodiments of the invention is to provide personalized role based view of information and action in addition to exportation of information to users based on their preference. It is a further object of embodiments of the present invention to provide a design 5 document containing recorded data relationships and defined rules to further understand the behavior of the system and entities. Yet another object of embodiments of the present invention is to provide an aggregation of business data by accessing the data and applying the corresponding business logic as required by the application. 10 Another object of embodiments of the present invention is overall and business entity wise progress bar for tracking and monitoring apart from facilitating bulk upload facility from external sources and providing Q&A forums to capture knowledge on data fields, entities and their relationships beside generating mail alerts. Yet another object of embodiments of the present invention is to record migration 15 decisions by the business and the rationale behind the decision for further reuse. Further, one more object of embodiments of the invention is to provide bulk upload facility from external sources into one system to minimize efforts involved in manually entering the data. It is thus a main object of an embodiment of the present invention to provide a system 20 for managing mapping information between different platforms to maximize reusability of mapping information, reduce rework and appraisal activities in order to improve productivity of data migration programs. It is an object of the present invention to overcome or ameliorate at least one of the disadvantages of the prior art, or to provide a useful alternative. 25 SUMMARY OF THE INVENTION: The underlying invention, in embodiments, relates to a structured, version managed and centrally governed web based system and a method to store and manage the source and target data structures and their relationships along with annotations and mapping rules -7 which will help to effectively manage the mapping process and maximize subsequent reuse of the mapping information in data warehousing and data migration or archival projects. The present system efficiently manages and tracks the data mapping progress that captures and derives market/ line of business/business entity wise mapping rules, 5 specifications, relationships and associated knowledge for subsequent re-use in later phases of data migration/archival projects. The solution can be used by any IT team involved in data migration, data archival and data warehousing projects for managing and sharing information pertaining to data profiling, data analysis, mapping specifications and high level design. The captured 10 information is further referred to by the developers to build the ETL programs and by testers to build test scripts and validate the data migration process. The solution captures tacit knowledge around mapping specifications as queries and their answers and discussions which can be used in the long run as system documentation for the support or maintenance teams. The graphical relationships generated can be used for impact 15 analysis and to analyze data relations. The IT teams which use this platform for ETL projects realize the following benefits: e Productivity improvement through information reuse, improved usability, process compliance, collaborative working, reduction in rework/ appraisal effort and effective information management. 20 e The built-in intuitive mechanism to depict relationships between source, entities and targets helped the team in understanding the information system architecture and design better. e Better project management through controlled information management, real time progress tracking and through built in workflows. 25 e Ability to keep track of changes in the mapping rules over a period of time via rollback mechanism. e Better collaboration among distributed teams through real time information sharing.

-8 According to a first aspect of the present invention, there is provided a centralized version managed computer implemented system for managing mapping information to be subsequently used in transformation from source data field to target data field is provided, the said system comprising of a data mapping management unit operatively 5 coupled to a storage system and plurality of input/output interfacial components within a network to execute the programmable instructions, the said system further comprising of: a reception subunit receiving inventory information and mapping information associated with at least one data field captured from the source and target data databases 10 including the set of defined rules governing it and configures the project with related information; an analysis subunit assessing received inventory information and mapping information and deriving at least one rule governing the received information; a management subunit deriving and verifying compatibility rule between at least 15 one of the associated data source and target system; mapping rule, transformation rule, cleansing rule, extraction, loading, validation and verification rules, business and reconciliation rules between the said selected databases and establishing corresponding mapping relationship between at least one column of one data source and associated at least one column of the target data; the said management module further facilitating 20 mapping process between at least one of the received and analyzed mapping information; a monitoring and validation subunit supported with inbuilt set of at least one process flow architecture for verifying mapping information and reporting for any unmapped data fields in a real time mapping process in a suitable format; and 25 a storage system to provide aggregation of mapping specifications and rules to be used in migration of at least one data source to at least one corresponding target system. According to a second aspect of the present invention, there is provided a method of managing mapping information for transformation between source data field to target data field, the said method comprising the steps of: 30 receiving the inventory information, mapping information associated with at least one data field captured from the source and target data databases including the set of defined rules governing it, mapping specifications, annotations and configuring the -9 project with related information; assessing received mapping information; defining the database definitions and deriving at least one rule governing the associated received information; deriving compatibility rule between at least one of the associated data source and 5 target system and configuring to the project; optionally deriving transformation rule between the said selected databases and establishing corresponding mapping relationship between at least one column of one data source and associated at least one column of the target data when the source and target data field are incompatible; the said management module further facilitating 10 mapping process between at least one of the received and analyzed mapping information using the reports generated by the system; verifying mapping information and reporting for any unmapped data fields in a real time mapping process in a suitable format by generating reports; and presenting aggregation of processed data for migration of at least one data source 15 to at least one corresponding target system; reports showing custom mapping specifications, unmapped or incompatible fields, mapping decision reports and graphical representation between business entities and source and target data sources. Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise", "comprising", and the like are to be construed in an inclusive 20 sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to". BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS: The foregoing summary, as well as the following detailed description of preferred embodiments, are better understood when read in conjunction with the appended 25 drawings. For the purpose of illustrating the invention, there is shown in the drawings example constructions of the invention; however, the invention is not limited to the specific methods and system disclosed. In the drawings: Fig. 1 represents the well delineated architectural view of the present invention. Fig. 2 highlights the primary process flow and the key functional components of the 30 solution offered by the present invention.

-10 Fig 3 illustrates the functional architecture of the present invention as implemented across the architecture of Fig 1. DETAILED DESCRIPTION OF THE INVENTION: Some embodiments of this invention, illustrating all its features, will now be discussed 5 in detail. The words "comprising," "having," "containing," and "including," and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. 10 It must also be noted that as used herein and in the appended claims, the singular forms "a,'' "an," and "the" include plural references unless the context clearly dictates otherwise. Although any systems and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, the preferred, systems and methods are now described. 15 Fig. I is a diagram showing the system configuration of an embodiment of the computer system according to the invention. A computing system of the present invention comprises of a data mapping management unit 102 for managing the mapping information, a storage system 103 and input/output systems or terminal 101. The data mapping management unit 102 may further be comprised of multiple subunits assisted 20 by processors while the input/output systems 101 include various input interfaces, say for example keyboards; one or more display interfaces connected to display monitors, one or more communication interfaces connected to communication devices so as to interact with programmable instructions executable on the give system. The data mapping management unit 102, storage system 103 and the input /output devices 101 are 25 communicatively coupled to each other by a network. The computer implemented system of the present invention derives data mappings by processing stored data mappings. The derived data mappings are system generated data mappings. The system herein comprises a plurality of stored data mappings, source and target data structures with annotations and mapping rules, mapping specifications, 30 relationship between source and target entities, validation routines, set of audit rules and - 11 a data mapping report generator which act as design documents referred by the users in understanding the behavior of systems and entities for subsequent reuse of the stored information in data migration or archival projects. The system is a web based solution supported by Apache HTTP Web server. It is therefore provided a means to reuse prior 5 data mappings to generate newer data mappings thereby increasing the efficiency of data migration process and reducing the errors and wasted efforts in data migration or archival process. The data mapping management unit 102 of the present invention is comprised of a processing unit entrusted with the task of accessing the business data and applying the 10 business logic required by the application. The data mapping management unit 102 is also connected through a network to input/output systems 101 comprising primarily of an input interface and display screen. The input unit may further comprise of a keyboard, mouse and the like. The network connection between the data mapping management unit and set of input /output systems may be an internal bus while the display screen and the 15 data mapping management unit may be integrally constructed. The storage system 103 in one of the preferred embodiment of the present invention is responsible for storing data mappings from prior data analysis and mapping efforts, a data mapping report generator and data mapping tool to generate derived data maps using the stored data mappings from prior data migration efforts. Derived data mappings 20 may also be added to the repository of stored data mappings. The storage system thus provides an aggregation of the standardized business data to be stored, managed and retrieved. Fig. 2 illustrates the schematic representation of the computer implemented data mapping management unit in accordance with the embodiments of the present invention. 25 The unit comprises of subunits to incrementally process the information received by the unit from an input interface within a communication network by applying the corresponding business logic to the received mapping information. The subunits contained within data mapping management unit includes: a reception subunit which receives inventory information and mapping 30 information associated with at least one data field captured from at least one of the distributed source and target databases in addition to reception of contextual -12 unstructured knowledge gathered in the form of question and answers and discussions on mapping specifications, business entities and data elements during data mapping from discussion platforms from a user via input interface say for example keyboard or mouse or the like. This information is accompanied with 5 added mapping annotations, rules, specifications, relationship between data and business entities, business entity definition along with metadata, market entities, line of business entities and knowledge around the data fields and business entities which govern the management of entire mapping information. Further the reception subunit configures the project with inventory information 10 and with roles assigned to users and the user is allowed to have a role based personalized view of the information an analysis subunit entrusted with the task of assessing the received mapping information and its data fields thoroughly for its structure, relationship, integrity, storage, management and retrieval. The analysis subunit additionally effectuates 15 deriving of one or more rules governing the associated captured information from the storage system of the present invention. The analysis subunit provides additional analysis inferences about the data fields in the source and target systems as tags and metadata to help in identifying related data fields in the target system. This helps in retrieval of related source data fields as suggestions for the 20 business entity in question while mapping a target field via an intelligent search facility based on identified tags or metadata in order to speed up the mapping process. The said information is stored within the storage system which can be used for future projects to decide as to which source fields are required to be mapped to target data fields. 25 The analyzed information is graphically displayed on the display screen to depict the relationship between business entities and the source or target database inventory while performing impact analysis to understand the behavior of the systems and entities. The subunit further allows users to configure mapping decisions along with reasoning post the definition of inventory to decide whether 30 a particular source or target field needs to be migrated or mapped. Only those data -13 fields which are configured as "to be mapped" will appear during the mapping process. A management subunit provided in the data mapping management unit executes the primary function of managing and controlling centralized mapping 5 specifications to maximize its subsequent reuse in data migration and archival programes. The subunit receiving the analyzed mapping information defines different databases and verifies compatibility rule between datatypes of one or more associated data source and target system obtained globally and also prepares its corresponding entity wise mapping specification. The importation of these 10 compatibility rules into configured projects allows further customization of the mapping rules locally. The subunit provides the ability to export database definitions and derived compatibility rules to the global space for other projects to reuse. Derivation of compatibility rules between datatypes of varied disparately located databases is a onetime activity which once defined can be imported and 15 reused by any project leveraging this tool to manage the preparatory phases (mapping and design stage) of data migration project. The project is finally configured with defined compatibility rules and gets stored in the storage system for further use. The management subunit in combination with the storage system supports basic 20 built in customizable templates for capturing inventory details and the metadata, business entity details and source-to-entity-to-target mapping information itself. These defined templates act as analysis or mapping design documents for reuse by other allied projects in addition to tasks associated with data cleansing, extraction, transformation, load, verification & validation or reconciliation. These templates 25 could be further extended by projects to capture additional information in the form of metadata based on their needs. The management subunit further facilitates framing of transformation rules between the said selected databases as its core objective while recording mapping specification. However such transformation rules are derived when more than one 30 source field is mapped to target field or if the data types of the source and target fields are incompatible. So, some fields can also have a direct mapping with a -14 source field or might directly be assigned a default or calculated value. The superadmin or the project admin defines the transformation semantics along with the appropriate help option. This is a onetime activity which can be reused as-is drawn from the storage system or customized and extended as per the project 5 requirement. The management subunit captures transformation rule or semantic; transformation description or help; number of translation arguments and help content for the arguments. The superadmin or the project admin is enabled to define fixed or variable set of arguments for any transformation rule. Argument help content is provided as a feature to give the user an idea about the arguments, 10 values to be given to the argument and the method to go about it. Transformation description gives the definition of each transformation rule and instruction on when and how to use it. The transformation rules schemed by the management subunit allows addition or edition for any assigned project using semantic editor. Use of a platform 15 independent custom defined semantic editor for data transformation rule definition in a structured universal format ensures uniform depiction and interpretation of transformation rules in simple language. The user is provided with an option to select multiple transformation rules according to the mapping for depicting a composite transformation rule/specification and for each function 20 in the transformation rule the user can specify the number of arguments provided the argument count is not fixed for that transformation rule via input interface. Additionally the user can specify the argument names (table/column names) against each argument for convenience. The help content option is provided against each transformation rule along with the argument definition. This helps 25 the user to understand about the argument of each function in the transformation rule. Thereafter the rule will be automatically generated and pre-populated once the user has given the argument names and number of arguments for each transformation rule. It uses a rich text editor for display on the display screen and hence gives the user the facility to modify the auto-generated rule accordingly, if 30 needed. Optionally, the user can also give some extra information along with the auto-generated rule for future mapping purpose. This feature comes handy while defining certain functions not configured in the semantic editor. Thus the - 15 generated rule can be saved along with the mapping specifications and modified later on with or without the semantic editor in the storage system of the present invention. For ex, if the transformation rule involves looking up a particular value in an array it can be depicted as Lookup ("Array", "Lookup Value", "Output 5 column position"). The said management subunit implements the mapping process between at least one of the captured and analyzed mapping information and compatibility for the same is checked during the data mapping stages. If the datatypes of columns are not compatible, it is mandatory to have a transformation rule for that mapping. 10 Each mapping has a flexibility to associate any number of files which should be uploaded to the storage system of the present system prior to association. A monitoring and validation subunit supported with inbuilt set of at least one process flow architecture and workflows receives the mapped information for its verification, review and refinement. The built in workflows ensures real time 15 tracking, monitoring and compliance with mapping process along with a version for its control to assess overall project-wise and business entity wise progress of data mapping. The report for any unmapped or incompatible data fields in a real time mapping process is generated and presented to the user on a display screen in a suitable format for either of approval or rejection of mapping specification. As 20 discussed earlier, if the datatypes of columns are inconsistent or incompatible, it is essential to have a transformation rule for that mapping. For the same, the user is issued a warning message detailing incompatibility of selected datatypes which mandates the user to specify a transformation rule while mapping incompatible source and target data fields. If the column datatypes are compatible, 25 transformation rule is not mandatory to be captured during mapping. The monitoring and validation subunit generates reports which include custom mapping specifications or design documents; unmapped data fields; incompatible fields; and mapping decision reports defining fields which are to be mapped and which need not be mapped. Progress bar depicting the progress of mapping 30 process is continually displayed. These reports help in distribution or reuse of the mapping information available in the storage system; performing further analysis -16 on data structures and their relationships along with regular monitoring of the progress made and validation of the mapping specification and the process thereby speeding up the entire mapping process. The user is also allowed to customize the report fields prior to exporting based on their preference. 5 The rules are hereafter stored in the storage system and re-used via report generation while capturing the mapping specification for validating the presence of a valid transformation rule for incompatible data fields. The information after been processed by the data mapping management unit in combination with the storage system can be displayed on the display screen of the 10 system. The system also facilitates bulk load inventory to avoid manual entering of the data which may include system details along with metadata. In a large enterprise, bulk entering of the data manually requires intensive efforts and time. The system provides Microsoft excel based templates which can be used to upload data in bulk. The system can also upload inventory information in bulk from XML based data definition 15 languages. The present invention thereby proposes a platform independent XML based inventory which is DDL compatible and is capable of importing details into the desired system. Fig. 2 details a primary process flow to achieve solution to the problems posed in the existing prior arts by means of a flow chart in accordance with the invention. As 20 indicated, the system allows for the creation of new project and suitably assigns a user for effective management of the project. Further, global data sources can be optionally created, edited or deleted by the means of controls provided in the interface as per the requirement. In accordance with the embodiment of the present invention, data mapping management 25 unit executes when the information received by the unit from the administrator through the input system is incrementally processed within the unit by applying the corresponding business logic to the received mapping information. At this time the data mapping management unit causes necessary information to be indicated on the display screen.

-17 The information received by the data mapping management unit includes at least one data field captured from at least one of the distributed source and target databases in addition to contextual unstructured knowledge gathered in the form of question and answers and discussions on mapping specifications, business entities and data elements 5 during data mapping from discussion platforms. This information is accompanied with added mapping annotations, rules, specifications, relationship between data and business entities, business entity definition along with metadata, market entities, line of business entities and knowledge around the data fields and business entities which govern the management of entire mapping information. 10 The process initiates with configuration of project with project details along with project metadata, market/line of business/entity definitions, database and datatype definitions and set of compatibility rules as stored in storage system. The project details are thereafter processed and correlated based on captured order data by the input unit. The user then assigns roles and project to different users. The projects configured with users 15 and also with information pertaining to the role to be performed by the user are generated as a system output. Further, the data mapping management unit configures the project with the source and target database information in addition to previously attributed information, the said information retrieved from the storage system of the present invention. 20 The assignment of user to the new project is followed by a process outlining the details of effective management of the project undertaken. System facilitates direct importation of DDLs (Data definition Language) exported from databases. For example it may be imported from a plurality of global data sources by either of public or private access or optionally and a new data source can be created. System sources may be any type of data 25 source, warehouse or any system. The data so collected or created can be in any of the type or format, however capable of being utilized by the system of the present invention. The roles of the users assigned to the project are defined and accordingly the user is allocated to the project. For every role, the permissions in the system can also be defined. Any user assigned to a particular role will thereafter assume the permissions 30 defined for the role. In addition, the system generates compatibility rules between the source and target entities which can either be added or deleted.

-18 It is however to be noted that the system provides personalized role based view to different genre of users at any point of time. For example the views can be any of the entity-wise view, source system view, target system view, business analyst's view, tester's view or a project leader view and the users are provided with a view of the most 5 updated version. In the present invention, the data can be consolidated from disparate systems which undergo incremental processing to eventually attain a formalized and structured form favorable for storing in the storage system of the present invention thereby reducing the workload. The standardized structures so obtained can be effectively used for quick and 10 efficient data extract for subsequent reuse. The data storage system supports applications that provide analysis and monitoring of stored data of business relevance. It is advantageously designed and tuned for fixed set of instructions or applications so that the data migration process is effectively executed independent of structure or arrangement of source data so obtained. 15 An inventory of the source and target information is collated which can be edited or deleted. In the present embodiment, the table/ column information is managed and commonalities are enforced in the information that appears across multiple disparate sources and prepares it to be stored in the inventory. In accordance with the present embodiment, receiving of the legacy source data and 20 target data is followed by receiving mappings to represent a desired movement of flow of data from legacy source data source to target data source. The mapping between the source and target columns is inserted, which is initiated for review, reviewed and approved by the user allows designing a transformation program for transformation of data form one or more source files to target files to minimize migration errors. The 25 mapping rules are formulated and stored in storage system wherein a known mapping is identified and all the associated source data can be migrated directly to the target file without further processing thereby increasing speed and automation of migration process. The rules captured include extraction rules, business rules, data loading rules, verification & reconciliation rules and data cleansing rules apart from transformation 30 rules.

-19 In one of the preferred embodiments specific data items may be identified and labeled by deriving relationships between the specific data elements and corresponding fields in customer database which can readily make the data elements available for migration process. In other words a user may be allowed to specify known values for certain 5 significant data elements within the source data flexible for editing, deletion or revision, if required. Other preferred embodiment of the present invention provides version control capability to manage changes or conversions in specification. The system also allows reuse of specification from past processing to be exported as design documents in a defined 10 format for use by data migration developers without incurring much time and efforts on re-building the already existing specifications. The validation operation is performed by monitoring and control subunit of the data mapping management unit, which reports the appropriate results and selects the unmapped or inconsistent component in the file format for further correction using a 15 review and approval workflow. This enables de-duplication, reconciliation and auditing of the standardized data. Moreover, the unmapped and inconsistent reports also assist in validation, reporting to external stakeholders and in monitoring the progress of the mapping process. The system thereby addresses the significant technical problems associated with immensely labor intensive, complex and error prone efforts of manually 20 off line creating the technical specification. There is also a considerable improvement in time saving and data quality in terms of accuracy and completeness as measures are adopted to track and assess the mapping progress real time, identify the gaps and the progress for each business entity. The system enables all time compliance to the mapping process through built in 25 workflows and a well defined flow process which renders minimal scope for an error to occur thereby improving the data quality. The built in workflows allows the system to generate data mappings for correlating between disparate data sources, also providing for entity wise mapping, its compatibility beside giving an option to generate field reports for any unmapped or inconsistent data fields. 30 The present solution also provides for methods and systems that optionally include security features for authentication and authorization of the user and to prevent -20 unauthorized access to the related data, and components for workflows or alerts, import/ export features, sorting or filtering of data, file attachments, rich text editor, versioning, metadata, syntax/ semantics for rules defined in case of mappings and DDL compatibility which offers convenience for reuse when referred by other teams in 5 understanding the behavior of the system and entities. Fig 3 schematically illustrates working of the present invention that may be implemented across various constituting units of the present system as expressed in Fig. 1. As depicted, a new project can be configured by the superadmin or the project admin using the project metadata and accordingly the system stores the project details for subsequent 10 use. The users along with their defined roles are assigned to their respective projects by the superadmin and consequently the project configured with users, roles and other inputted details is stored within the system. The user configured project is thereafter loaded with information pertaining to legacy source and target databases and one time compatibility rules between source and target 15 entities, following which project configured with compatibility rules along with other user related information and their associated roles is generated by the system. In addition, an inventory of source and target information is collated and maintained within the system as shown in Fig 3. This is followed by receiving of formulated mapping information captured from distributed source and target databases in addition to 20 contextual unstructured knowledge gathered in the form of question and answers and discussions on mapping specification, business entities and data elements during data mapping from an integrated discussion platforms and mapping rules as shown in Fig 3 using which a known mapping is identified and all the associated source data can be migrated directly to the target file without further processing. The system, in turn, allows 25 reuse of mapping information in a defined format by data migration developers. The mapped details are thus obtained as system output to be used as a set of customized specification for future use. The system generates new versions for every change and the user will have the privilege to rollback to any of the prior versions as desired. The system in addition provides an 30 ability wherein the users can subscribe for an email alert whenever any change is made to their entity or source or target data fields of interest.

-21 The system as shown in Fig 3 enables all time compliance to the mapping process through built in workflows and a well defined flow process which renders minimal scope for an error to occur thereby improving the data quality. The built in workflows allows the system to generate data mappings for correlating between disparate data 5 sources, also providing for entity wise mapping, its compatibility beside giving an option to generate field reports for any unmapped or inconsistent data fields. A real time progress tracker in the form of progress bar depicts overall mapping completion status for individual business entities and overall. This constitutes a unique feature as it enables real time monitoring and validation of the mapping process. 10