Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/25—Integrating or interfacing systems involving database management systems
  • G06F16/258—Data format conversion from or to a database
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units

Definitions

• the present invention relates to a computer-implemented system which is able to retrieve information stored in one or more of a number of different sources and which may be in any of a number of different formats and/or provide reports and analysis based on the information, and in particular to a computer method and apparatus which can automatically retrieve database information stored in any of a plurality of formats, including structural and/or relational information, without the need for relying on human analysis ofthe source data.
• a number of ways of organizing computer-accessible information have developed, such as relational or hierarchical database management systems, flat file data systems, spreadsheet systems, and the like. These systems are used for storing, manipulating and displaying a myriad of types of information, including accounting or other financial information, scientific or technical data, corporate or business data, name, address and telephone data and statistical data. Many formats and data structures have been developed, and this situation has both desirable and undesirable ramifications.
• On the positive side by having a multiplicity of different types of systems, it is possible to provide different systems which are optimized for different purposes (e.g., optimized for data entry or storage vs. speed or flexibility of data analysis and reporting, optimized for accounting data vs.
• Examples of such situations include: (1 ) an accountant who wishes to produce standardized reports but who has multiple clients, each of whom keeps its accounting data in a different type of data source; (2) a co ⁇ oration with several divisions which wishes to produce uniform reports, but in which different divisions use different corporate or financial software; (3) a co ⁇ oration which wishes to produce uniform reports, but which keeps its accounting information on a first type or brand of database (or other data source), and its co ⁇ oration information on a second and different type of database; (4) a group of scientists investigating a common problem, each of whom stores or has access to data kept in a different type or brand of database or other data source.
• Other examples will occur to the reader after understanding the present disclosure.
• Such situations present difficulties for a number of reasons, including the difference in manners of organizing information and differences between types of data sources.
• similar categories of information may be organized in different ways, even if the same database software is being used.
• a user might organize a company's personnel records such that all ofthe company's personnel names are stored in a first table or list, all ofthe addresses are stored in a second table or list, and all ofthe telephone numbers are stored in a third table or list, and pointers or links are stored to indicate which names are associated with which addresses and which phone numbers.
• each line or "record" of information includes a name, an address and a telephone number, thus without any links or pointers from a record in one table to a record to another table.
• different types of data sources may have different structures and/or different data storage formats or schemes.
• some database packages are organized in a hierarchical manner (e.g., in a tree- fashion), while others may be organized as relational databases (modeled on two-dimensional tables of rows and columns).
• information may be stored in forms that are not, strictly speaking, database forms such as storing data in a "flat file” form, as a spreadsheet, and the like.
• different types of data sources may store the data in various formats. For example, some database products store each table, each reporting format and each query as a separate file on a storage device such as a hard disk, while other software may store all tables, relationships, queries, report formats, etc., in a single file.
• Some products may store each record and/or field as fixed length data and or at a fixed position in a file, while others may use delimiters to distinguish between one record and the next or between one field and the next within a record. Even if two different software products store a particular type of information at a predetermined location, such location may be different for the different software products.
• data may be encoded differently in different software products, such as using ASCII encoding in one product and multi-lingual (multi-byte) characters in another product. In some cases, data may be compressed and/or encrypted.
• a consultant or other expert individually or “manually” analyzed each "source” data file or database to understand its structure, relationship data storage format, the organization ofthe data within the database, and the like. The expert would then construct some manner of import or querying of the data in the source data file or database in order to achieve the desired access, coordination or combination.
• the present invention relates to a system which achieves access to stored information, e.g., for accessing information or for achieving coordination and/or combination of information in two different information storage systems.
• some or all the analysis involved is performed automatically (i.e., without the need for human analysis), in one embodiment, using a properly programmed computer.
• the system is flexible in that it i not inherently limited in the data formats it can access but can be configured to obtain data from virtually any computer-readable information source.
• the system is extensible, (more preferably, modularly extensible in that components can be added to permit it to access additional types, formats or organizations of data.
• the access, coordination or combination of data is accompanied by an enhancement of data analysis, i.e., providing types of data analyses and/or reporting not found or used in the original data source.
• the system can be used to provide for standardization of data analysis or reporting across several types of data sources.
• the system uses the contents ofthe source data files or databases, as well as information about the structure, in order to achieve the desired results (such as by using text recognition, artificial intelligence, and/or expert systems).
• the system uses such information to at least partially control the manner in which data is made available for analysis or reporting.
• the system uses such information in providing such analysis or reports.
• a data source which may be any of two or more types of source data, in a standardized or uniform manner.
• a plurality of drivers are provided specific to different types of source data which include programming for identifying structural or other characteristics ofthe various data sources, e.g. for use in defining a new database.
• the new database is configured to permit highly flexible and/or rapid output or reporting or is otherwise optimized for reporting pu ⁇ oses.
• the present invention includes conversion of one or more data sources into one or more uniform databases, preferably generating one or more key categories for organizing the data, optionally generating category groupings or rollups and additional data or optional references.
• the present invention creates or populates a database, based on accounting or other data converted from existing data files, such as data files created by previous accounting or other software.
• the system is configured to facilitate updating some or all portions ofthe new database such as by storing one or more profiles of data which should be updated and/or creating or defining a schedule to automatically execute update procedures at predefined times or intervals.
• enhancements are provided which automatically identify a desirable view ofthe data such as by generating a view when more than one value is found for a given field or parameter ofthe data.
• one or more verification or audit tools are provided to detect potential errors or problems.
• Fig. 1 is a schematic depiction of an example of flat file data storage
• Fig. 2 depicts a directory structure of a type usable in connection with the data storage depicted in Fig. 1 ;
• Figs 3A - 3C depict examples of data storage formats usable in connection with the data storage depicted in
• Figs. 4A - 4F are schematic depictions of an example of data stored in tables of a relational database ;
• Fig. 5 depicts a directory structure of a type usable in connection with the data storage depicted in Figs 4 A
• Fig. 6 is a schematic depiction of an example of flat file data storage
• Figs. 7A - 7D are schematic depictions of data stored in tables of a relational database
• Fig. 8 is a block diagram of a system for information retrieval according to an embodiment ofthe present invention.
• Fig. 9 is a schematic depiction ofthe contents of a function module according to an embodiment ofthe present invention.
• Fig. 10 is a flowchart of a process for information retrieval according to an embodiment ofthe present invention.
• Figs. 1 1 A and 1 IB depict pseudocode procedures for selecting or searching directories, according to an embodiment ofthe present invention.
• Fig 12 is a schematic depiction of data stored in tables of a database 808 provided according to an embodiment ofthe present invention.
• Table I provides a comparison of types of data that might be stored by two different co ⁇ orations. Table I is intended to indicate the conceptual organization of accounting and other information for two co ⁇ orations, and is not necessarily information that would be stored in a database (although it could be, if desired).
• co ⁇ oration no. 1 retains Account information, employee information, project information and product information and thus has four account parts.
• the Account information of co ⁇ oration no. 1 includes only three components: cash, accounts payable, and accounts receivable.
• the co ⁇ oration maintains a list of its salesmen, tracks information by two types of projects (research and sales) and maintains a list of its products.
• the second example of Table I is a somewhat lengthier (but still simplified) example.
• the Account has multiple components in a hierarchy.
• co ⁇ oration no. 2's account has categories of cash, accounts payable and accounts receivable, each of these categories has subcategories and some ofthe subcategories have even finer divisions.
• the employee, project and product information is broken into several categories and subcategories, and co ⁇ oration no. 2 further has additional items which are tracked, such as subsidiary companies.
• Table I illustrates that, even before considering differences among data sources and/or data storage formats, a co ⁇ oration's structure and/or the manner in which it chooses to organize its information will cause differences from one system to another. For example, if an accountant had co ⁇ oration no. 1 and co ⁇ oration no. 2 as clients and wished to use a uniform or standard system of reporting and analysis or for these two co ⁇ orations, there would be significant difficulties in doing so, even if co ⁇ oration no. 1 and co ⁇ oration no. 2 used the same database software, and even organized accounting information within that software in a similar fashion.
• FIG. 1 is a schematic depiction of how a plurality of flat files might be used to store information for co ⁇ oration no. 2 of Table I.
• Fig. 1 shows the information as it might appear in written form on a plurality of sheets of paper, the data will, in fact, be stored on a computer readable medium such as a hard disk, e.g., as described below.
• the format of Fig. 1 is intended to depict the logical structure ofthe data organized into a plurality of files 101a through 101 f, each file having file identification information 104, depicted in Fig. 1 as title or header information 104a, 104b. and a plurality of records, depicted in Fig.
• FIG. 1 as lines of information 106a, 106b, 106c, each record having a plurality of fields (organized into columns 108a, 108b, 108c, 108d in the depiction of Fig. 1).
• Methods and apparatus for storing and accessing data to have or reflect the logical row and column structure depicted in Fig. 1 are well known to those of skill in the programming art.
• the present invention can be used in connection with a number of flat file information storage programs. Examples of such information storage programs include those sold under the tradenames, Simply Accounting and MAS-90TM.
• the data organized in the logical structure depicted in Fig. 1 may be stored in a number of different formats.
• the data in each flat file 101 a through 10 l f is stored in a separate file on a hard disk of a personal computer.
• Fig. 2 depicts a directory/file structure which might be used for storing such files, in which all files depicted in Fig. 1 are stored in a single subdirectory.
• the various files even though organized in a directory hierarchy as depicted in Fig. 2, may be stored, physically on a hard disk in a plurality of separated locations.
• a number of formats may be used for storing the data in a file. Examples are depicted in Figs.
• the file includes header information, followed by storage ofthe first record 106a, the second record 106b, and so on.
• a fixed length data format is used in which each record 106a, 106b, has an identical length 304 (i.e., occupies a fixed number of bits).
• each field within each record also has a fixed length 308a through 308d.
• Fig. 3B depicts another fixed length data storage method in which data is stored in column order rather than row order (all dates sequentially, then all descriptions sequentially, etc.).
• Fig. 3A depicts another fixed length data storage method in which data is stored in column order rather than row order (all dates sequentially, then all descriptions sequentially, etc.).
• a particular piece of data will be located a given distance (i.e., a given number of bits) from the beginning ofthe data.
• a given distance i.e., a given number of bits
• the data information for the first record 106a will necessarily be found beginning at byte number 5
• the data information with the second record 106b will necessarily be found beginning at byte number 13, and so forth.
• Fig. 3C depicts storage of data in delimited fields rather than fixed-length fields.
• a special symbol i.e. a pattern of bits which is different from any pattern used for storing data, is used to mark the en or beginning of a record and or a field.
• two different special symbols are used, one to mark the beginning of a record and another to mark the beginning of a field. These symbols are indicated in Fig. 3C, respectively, as a colon and a semicolon, which may stand for any bit pattern or symbol.
• Fig. 3C In th data format of Fig. 3C.
• Figs. 4A through 4F depict one possible organization of information in the context of a relational database.
• a first transactions table 402 is stored, having a plurality of records 406a through 406d.
• the records depicted in Fig. 4A are similar in some respects to those depicted in Fig. 1 , i.e., include a data field, description field, quantity field, and amount field.
• an additional index field 408 is provided for each record.
• FIG. 4A does not include the header information 104a ofthe type depicted in Fig. 1, and only a single transaction table Fig. 4 A is provided (rather than a multiplicity of tables 101a through lOlf in the embodiment of Fig. 1).
• additional tables are provided which may reflect the organization depicted in Table I.
• an Account table 412 includes a list of all the categories defined in Table I, with an index 414 being associated with each account.
• a employee table 416 includes the names ofthe employee of Table I, each having an index 418 associated therewith. Also in Fig.
• 4C is an indication, for each name, of whether the person is associated with the sales force or the research force (reflecting the hierarchy shown in Table I). Furthermore, a field is included to indicate the location ofthe employee. Additional tables (not depicted) may be provided to list the various projects, products and subsidiaries of co ⁇ oration no. 2, reflecting the organization of Table I.
• Fig. 4D depicts a link table 422, which indicates, for each record of a transaction table 402, any desired links to other tables.
• a record 428 will be provided, indicating that for the transaction record having index value no. 1 , the appropriate account reference is that having the index no. 424.
• links may be made to the employee table 416, or other tables (not shown).
• a single transaction table 402 is required in the embodiment of Figs. 4A through 4F, with the link table Fig. 422 providing the information which, in the example of Fig. 1 , is obtained by knowing in which flat file a transaction is stored.
• Fig. 4E depicts a table which stores a plurality of such queries using, e.g., structured query language (SQL).
• SQL structured query language
• the queries which are used in a particular database system may reflect the manner in which the data is analyzed or organized by a company. And thus, an accountant who may have an interest in standardizing reporting and analysis based on information in such a database may wish to know about and/or be able to reproduce the data analysis of the type represented by various stored queries (Fig. 4E).
• database permit a user to design a report (either for display or printing) and, in some cases, to store information defining such a report, e.g. for repeated use.
• another table or set of tables may be stored as part of or in connection with a relational database to preserve information regarding such reports.
• Figs. 4A through 4F may be stored in a number of different fashions.
• Fig. 5 depicts a directory/file hierarchy which may be used for storing a plurality of data tables, link tables, query tables, and/or report formats. Such data may be stored in a number of different data formats, such as any of those depicted in Figs. 3A through 3C, or others, as will be known to those with skill in the art.
• Another example of information which may be stored in various formats is scientific or technical information.
• Fig. 6A a flat file system is provided for storing surface temperature information for, e.g., meteorological research.
• Fig. 6A a flat file system is provided for storing surface temperature information for, e.g., meteorological research.
• each file 602a, 602b, 602c stores information for a particular location and unit of measure (e.g., Fahrenheit or Celsius), indicated in the header thereof 604.
• unit of measure e.g., Fahrenheit or Celsius
• a relational database system for storing this type of data is depicted in Figs. 7A through 7D.
• a data table 702 contains all ofthe observations and, for each observation, unit of measure 704a, along with an index 704a.
• Tables 706 (Fig. 7B) and 708 (Fig. 7C) store information relating to location and hours ofthe day.
• Each line 722, 724 ofthe link table (Fig. 7D) can be used to indicate, for any data point stored in Fig. 7A (as indicated by its index value 704b), the index value for the location associated with that data point and the time of day associated with that data point
• Figs. 6 and 7A through 7D are simplified, it will be apparent to those of skill in the art how to construct information systems for storing more complicated sets of data such as meteorological data, including wind speed and wind direction, radio sound data, and the like.
• meteorological data including wind speed and wind direction, radio sound data, and the like.
• a researcher wished to correlate information from two meteorological data sources, one of which stored information in the system depicted in Fig. 6 and another storing information in the system depicted in Fig. 7A through 7D
• Fig. 8 is a block diagram of a system, according to one embodiment ofthe present invention, for use in overcoming the difficulties described above in relation to accessing, coordinating, or combining data on different information systems.
• a main process 802 selectively activates various drivers 804a, 804c, 804d, as needed in connection with analyzing and/or accessing information in a plurality of data sources 806a, 806b, 806c, 806d.
• the source data 806a through 806d, depicted in Fig. 8 can be data which is stored or created by any of a number of programs or systems for organizing or storing data such as flat files systems, databases, spreadsheets, etc., as discussed above.
• Fig. 8 may reside on one or many computers.
• the process is implemented in the context of a local-area network (LAN), having a network server computer, and associated disk or other storage devices, and one or more client computers.
• the main process 802 is executed on a client computer while the information sources 806 and the data files for the new database 808 are stored on disk (or other data storage) associated with the network server.
• the invention can be implemented in many other architectures, such as on a single computer, on multiple non- networked computers, using computers which are linked by a communications link such as a wide area network, modem communications, via the Internet, etc.
• a number of types of computers can be used to implement the invention, including mainframe and personal computers such as those produced by International Business Machines (e.g. 386, 486 or Pentium or Pentium Plus based computers), Apple Co ⁇ . (such as Macintosh computers) and clones of such computers.
• the processes are implemented using a DOS operating system and or a Microsoft* Windows or Windows 95 user interface.
• the items referred to herein as drivers 804a through 804d should not be confused with a data filter of a type provided in certain database programs.
• a data filter is, typically, a type of query or logical test for selecting certain records and or fields, according to user determinable criteria.
• the drivers 804a through 804d are processes having multiple functions for analyzing and accessing different types of source data.
• the function modules 804 are provided as dynamic link libraries (DLLs) in a manner that will be understood by those of skill in the art after understanding the present disclosure.
• the drivers 804a through 804d are configured for operation with one or more types of data sources, such as a data file produced using a particular database program.
• the source data 806a through 806d depicted in Fig. 8 may, in general, be any computer readable information source. Examples include flat file source data, hierarchical databases, relational databases, spreadsheets, and the like. Although Fig. 8 depicts an embodiment in which four data sources are shown, the present invention could be used in context in which there was only a single data source, or in which there were five or more data sources. Although the present invention may be used in situations in which each data source 806a through 806d is produced using a different type or brand of software, it is also possible to use the present invention in situations in which two or more ofthe data sources are produced by the same brand or type of database or other software.
• the first driver 804a might be configured for use in retrieving information from data files that were produced using dBase II ®
• the second driver might be configured for use in retrieving information from data files that were produced using dBaselll ®
• the third driver might be configured for use in retrieving information from data files that were produced using a flat-file system such as Simply AccountingTM
• the fourth driver might be configured for use in retrieving information from data files that were produced using Microsoft Access ® .
• the results of such analysis can be used in a number of ways, including providing for user-access to the information in the data sources for viewing or editing, copying some or all ofthe data and, preferably enhancements thereof (as described below) to create a new database, creating data reports (for viewing, printing, storage, transmission, etc.), queries and the like.
• the main process 802 using the drivers 804, performs an analysis ofthe source data 806a through 806d, it may create one or more new databases 808, containing data from one or more of the various data sources 806a through 806d.
• one new database is created for each data source.
• one database 806 may contain information from two or more data sources (e.g. if a company uses one database or other data source for storing sales information and another database or other data source for storing employee information).
• the new database or databases 808 may, if desired, be used for generating reports, e.g., using a report writer 810, and may, if desired, be used for entering, viewing or analyzing data, e.g., using a database management system 812 or other software 814.
• the database 808 is a Microsoft ® Access database which includes base code having one or more wizards, templates, filters and or tool kit software (as those terms are understood by those skilled in Microsoft* Access programming) e.g. for providing database reporting and analysis such as outputting standard financial reports.
• financial and management reporting software is provided as an extension or modification of that available under the tradename MVTM available from Timeline, Inc. of Bellevue, WA.
• the analysis system includes a module configured for generating or providing reports or screens for specific pu ⁇ oses or groups. For example, in the embodiment of Fig.
• an executive information system (EIS) 815 is provided preferably with an easy-to-use user interface and configured to output analysis of the data in various spreadsheet, presentation graphics, or printed forms, preferably using navigation tools or graphic selectors (e.g., to select between bar charts and pie charts, to select reporting periods, to select quarterly v. weekly reports and the like).
• the executive is presented with a menu of various views of the data, including views which may have been automatically generated or detected using the automatic roll-up generation procedure as described above.
• the information in the new database or databases 808 is enhanced, e.g. by main process 802 in the sense that it is configured to generate, display or output analysis or relationships of data which were not displayed or output by or using the source data 806a through 806d
• Fig. 9 is a schematic depiction of various drivers 804a through 804d.
• Each driver includes a plurality of defined processes or functions 901 through 909.
• Each function may include computer program instructions 912, e.g., to implement and carry out one or more ofthe steps described below and depicted in Fig. 10.
• each function 901 - 909 is a callable subroutine or procedure.
• the functions 901 through 909 defined in a given driver 804b include functions which must be performed or carried out differently depending on the type of source data 806a, 806b.
• function one 901 which is a function designed to select certain directories on a hard disk or other information storage device where desired information is stored
• the procedure for selecting directories will differ depending on the type of source data 806, as can be seen, e.g., from a comparison ofthe example of Fig. 2 with the example of Fig. 5.
• the programming 912 which implements function one in the first driver 804a may be different from the programming code which implements a corresponding function in the second driver 804b.
• each driver defines one or more processes for performing a function with such procedure being configured to accommodate the different characteristics of two or more different types of source data.
• FIG. 1 IA depicts a portion of a procedure, expressed in pseudo ⁇ code, of a type which might be used in connection with selecting and/or searching directories in connection with the directory structure depicted in Fig. 2, while Fig. 1 IB shows corresponding portions of pseudo-code for a procedure which might be used in connection with selecting or searching directories for the directory structure depicted in Fig. 5.
• Fig. 9 depicts a function module having nine functions, the present invention could be used in connection with a function module having more or fewer functions.
• FIG. 10 A number of procedures can be used to start 1002 the method which is depicted in Fig. 10.
• the method of Fig. 10 is implemented using a computer program stored on a medium such as a hard disk, CD-ROM or other non-volatile medium, and the method is started by issuing instructions to the computer (e.g. via a keyboard, mouse, etc.) to launch the program, i.e., to load the program into memory and execute the program.
• the program may be launched by another program.
• the new database 808 is a Microsoft Access Database which can include a routine, such as a so-called "wizard,” for launching the program, with the program (Fig. 10), in turn, accessing data in the information sources 806 to populate or update the database 808.
• a routine such as a so-called "wizard”
• the wizard it may be useful to employ the wizard to display prompts or "dialog boxes" for the pu ⁇ ose of soliciting user input as needed (e.g. for step 1020), so that the user interface will have an appearance which is consistent with the user interface for the database 808.
• the first step after the procedure starts 1002 is to identify and initialize the dynamic drivers 1004.
• the drivers 804 are considered to be dynamic in the sense that drivers can be modularly added or deleted, e.g., to accommodate a new or different type of data source.
• a user may initially be provided with a system such as that depicted in Fig. 8 having four drivers, but may at some future time add additional drivers through purchase from a software vendor, downloading from an information service, network, Internet connection and the like, or by writing a custom driver. Because of the modular and dynamic nature ofthe drivers, it will not be known, in advance, which drivers are available, and thus, when the program is started 1002. the program identifies the drivers that are available to it.
• this is performed by searching a disk or a directory thereof for files having a predetermined (partial) file name or file extension.
• the program may further analyze selected portions of each file, e.g., header information, to verify that files identified by such file name and/or extension are desired drivers.
• Initializing the drivers generally includes identifying and linking the driver functions and initializing data within each driver.
• an import a procedure is first done in which all or a majority ofthe data and structure in the data source is accessed and saved to the new databases.
• an update a procedure is done in which only a selected portion ofthe data and/or structure are accessed, e.g., to assure that the information in new databases 808 reflects recent changes or additions that may have been made to the data source 806.
• an import will be performed the first time the system of Fig. 8 accesses or uses information from a given data source, or if relatively major changes or additions have been made to the data source.
• an update will be performed on a regular basis (e.g., daily, weekly, etc.) in order to synchronize the data in the source data 806 with the data in the new databases 808.
• the selection to import or update 1006 is performed automatically, i.e., by performing an update unless this is the first time the procedure has accessed the particular data source, ln another embodiment, a user is permitted to choose between import and update by providing input, e.g., via a keyboard selection, using a pointer device, and the like.
• the main procedure 802 will initiate the execution of a function of one or more ofthe drivers to select directories to search 1010.
• the driver 804 which is loaded or called by the main process 802 w ill depend on which type of source data is being accessed. In particular, for a given data source 806a, the main process 802 will employ the driver 804b, which is configured to accommodate that type of source data 806a. If more than one data source 806 is to be accessed, the main process 802 will use whichever driver 804 is configured for each type of source data 806.
• the type ofthe data source is determined automatically based on characteristics such as the names (or "extensions") ofthe files and/or directories, the number, size and structure ofthe files, header or other information in the files.
• a user is permitted or requested to indicate the type of data source (e.g. by identifying the brand name and version number ofthe software that was used to create the data source files or by indicating whether the user wishes the procedure to search only local disk files or to perform a search mat includes network files).
• the main process 802 will have access to a stored list of the directories which are to be searched for all data sources 806 as identified by drivers 804 which were identified in step 1004.
• the main process 802 loads or activates or calls another function 902 ofthe function modules 804a through 804d, in order to search the directories which were selected in step 1010 for the data which is to be imported 1012.
• the search of directories 1012 is performed in a manner which will depend on how the information is stored in different data sources 806. For example, for some types of source data it may be sufficient to identify files having a certain file name and/or certain file extension.
• source data 806 it may be necessary to scan the data in various files to identify files which have certain structures or contents, e.g., in a header portion of a file, or elsewhere.
• different modules 804 will be configured to provide the "search directories" function 902 in different manners, to accommodate different data sources 806.
• an update 1014 rather than an import 1008 is to be performed, it is not necessary to select and search directories, since, preferably, when an import is initially performed, the results ofthe functions 901 , 902 for selecting and searching directories are stored in a manner which can be accessed by the main process 802 at a later time. Thus, using such stored information the main process 802 is able to identify data which was previously imported or updated. In one embodiment, this is useful to prevent loading of redundant data, i.e., data which is already present in the new databases 808. In general, for a full update it is desired at step 1016 to identify data which is new or changed since the last import or update such that at least some data already in the databases 808 will not be re-loaded.
• the system in order to prevent redundant data loads, the system will attempt to identify data which has not changed since the last import or update. In general, if this procedure is followed, at the end ofthe procedure the data in the new databases 808 will be synchronized with the information in the source data 806, i.e., it will contain information which accurately indicates the structure and data ofthe data sources 806 in their current state.
• an identification ofthe data which is to be imported or updated is displayed 1018 in order to give the user an opportunity to select the data to be imported or updated or to choose to refrain from importing or updating certain data.
• the manner in which the indication of data to be updated or imported is organized for display will depend on what type of data source is being accessed, and thus will be provided in response to a call or activation of a function in one of the drivers 804 (902a).
• a function 902a of a driver which is configured for use in connection with the data source depicted in Figs. 4A through 4F might display a list of the subsidiary companies drawn from table 430 (Fig. 4F) in order to give the user an opportunity to import or update data for some companies but not for others.
• the user may select one or more companies from a displayed list.
• the display in selection steps 1018, 1020 can be repeated for other types of organizations or data in the data source, e.g., to display and permit selection of data specific to certain employee 416, to certain accounts 412, depending on how the display function 902a of the driver is written or configured.
• the user will continue to use the original information sources 806a - 806d for creating, storing, editing and, in some cases, viewing, data.
• the present invention is compatible with situations in which the user wishes to continue to use the databases 806, user interfaces and the like, with which the user is already familiar.
• the frequency with which the new database 808 must be updated depends upon the frequency with which various uses are made ofthe database 808. Thus, it may make little sense to update the database 808 on a daily basis if it is accessed only once per week. Moreover, the particular information included in the update may vary according to the use made of the new database 808. Thus, if factory production data is reviewed by management only on a quarterly basis, there is no need to include such data in the weekly updates which may be used, e.g., for reviewing regional sales.
• one or more profiles defining various types of updates are created and stored and, preferably, a scheduler is also created and stored.
• the scheduler is used to define the times at which to automatically execute such updates (i.e., without requiring a person to "calendar" or remember to initiate such updates).
• different update profiles are scheduled to be executed at different times or intervals, such as by providing a scheduler which automatically executes, e.g., regional sales update procedures weekly before a weekly sales meeting and which schedules quarterly updates of production data prior to quarterly production reviews.
• the profiles may be stored along with the new database 822, the old database 824a - 824d (if there is appropriate "write permission" for storing information in the information sources 806a - 806d) or both.
• Various schemes can be used to create profiles. It is possible to include commands (such as menu selections or buttons) to allow a user to request creation of an update profile (e.g., via a predetermined series of screen forms or dialog boxes, such as provided in Microsoft® Access Wizards). It is also possible to automatically prompt a user for a decision regarding creation or editing of a profile, e.g., whenever a user requests a screen or printed report which uses data not currently in a defined profile (or which uses data more frequently than the current profile update frequency for that data).
• the profile 822 is created by creating a database table which lists each field ofthe database 808 and, for each such field, indicates whether there is an existing profile for updating that database, the identity or name of such profile and the current update frequency or interval.
• the user ofthe system of Fig. 8 is preferably also provided with an option to request an update at any time, preferably with the option to select among any ofthe predefined update profiles or request a full update.
• the system may prompt the user for a decision regarding whether this set of data to be updated should be stored as a profile (as described above).
• the system may also provide a list of already-existing or defined profiles so that the user may select among predefined profiles rather than selecting among displayed data (such as a list of subsidiary companies).
• the function 902a for the applied driver can merely return program flow to the main process 802.
• the user may be provided with a display indicating the location ofthe data which corresponds to the user's selection, such as a display ofthe directories, sub-directories and files containing the information, and may optionally be given an opportunity to select which directories, files, etc. are to be accessed.
• the data which is to be imported or updated general information is loaded 1022. If it is desired to access information from two or more data sources, this can be done either serially (i.e., performing steps 1022 through 1046 on a first data source using a first module, followed by performing steps 1022 through 1046 on a second data source using the appropriate driver, and so on) or in parallel (i.e., performing steps on each desired data source using the appropriate drivers before performing later steps on each information module).
• General information includes information about the structure ofthe data in the data source.
• the type of general information which is loaded in this step 1022 will vary for various types of source data.
• the general information may include, e.g. an identification of accounts or other categories used in the data source 806.
• the function 903 of a driver 804 is configured or written for use in connection with the data source depicted in Fig. 1 , it may still be desired to determine how many components are used in the data source but, in this case, this information will be determined by the number of fiat files 101a through lOl f found in the data source 806.
• General information may also include information such as how many projects 1 12, how many products 1 14 and/or product lines 1 16, or how many subsidiaries 1 18, are defined in the data source 806.
• General information can also include company name, first month ofthe fiscal year and, in general, any other information that can be loaded once (as opposed to, e.g. information loaded in steps 1024, 1033 and 1036, typically loaded in a loop). If a "load general information" function 903 is provided in a driver configured for use in connection with the data source depicted in Fig. 6, general information such as the number of locations 612 in the database might be loaded in step 1022.
• the main procedure 802 also calls or activates a function 904 ofthe appropriate driver or modules 804 to load data definitions 1024.
• Data definition can include information such as the text name stored as an identifier for a particular class or category of data in the data source 806, the field size, type of data (string, integer or decimal; number of decimal places) and similar characteristics for various categories of data.
• loading of the data definition includes interrogation ofthe data to obtain information necessary to store an indication of the architecture or structure ofthe information in the data source and the data elements in the data source as required to generate one or more new databases 808 which will contain all ofthe structure and data needed for the type of reporting or analysis to be performed on the new database.
• the interrogation ofthe data in the "load data definition" step is intelligent interrogation in the sense that it can conform to virtually any data source and identify what is required to store a standard form ofthe data source, e.g. for reporting and analysis.
• information needed to indicate the architecture ofthe source data would include, e.g., the names ofthe four account parts (Account, Company, Employee and Location) as well as the data type (e.g. numeric or string) and length required to store any string account parts.
• information needed to indicate the architecture of that data source would include storing the names ofthe account parts (Location and Date) as well as the names of the references used for this data (Unit).
• Interrogation may include identifying other optional data that can be loaded, e.g. invoice numbers.
• the particular type of interrogation performed will depend on the characteristics of the particular data source being analyzed and thus is different for each driver 804. In general, steps of loading data or information 1022, 1024, 1033, 1036 are performed by functions in drivers 804, while steps of saving information 1026, 1028, 1030, 1032, 1034, 1038 are performed by the main process 802.
• the main process 802 then identifies or creates a database repository 1026, i.e., identifies or creates a file or other data storage structure in the new databases 808 which will serve as the location where information loaded from the data source or sources 806 will be saved.
• An update typically does not need to create a new database or database tables, since updates usually simply add to already-existing tables within an already-existing database.
• the main procedure 802 then calls the appropriate function 905 of one or more drivers to create database tables 1028 which will be used for storing data saved from the source data into the new databases 808.
• the manner in which the database tables are created preferably takes into account both the structure ofthe data and the data source or sources 806 and the manner in which the new databases 808 will be used, e.g., for analysis, generating reports, etc. Because the particular tables which are created will vary depending on the characteristics of the information in the data source 806 (as determined, e.g., by steps 1022 and 1024), the create database tables 1028 are functions provided by drivers 804 configured for the particular database being used for access. For example, when the "create database table" function 905 is written or configured for use in connection with an data source such as that depicted in Figs.
• the database tables which are created may include, e.g., an Account table, a employee table, a subsidiary table (which may be a roll up of Account), and a details table (as described more thoroughly below), whereas a "create database table" function written or provided in a driver configured for use in connection with the database depicted in Figs. 7A through 7D may include a location table, a date table, a time table, a unit table and a detail table.
• the tables created in the new databases 808 have a structure or architecture which is dynamic in the sense that it can accommodate virtually any type of data definition or structure which might be found in various data sources 806.
• the new databases 808 are intended to be used primarily for outputting information such as generating reports and analysis and thus are preferably configured, as described below, to provide superior output performance such as high flexibility in the types of outputs and data analysis available and relatively rapid execution of such analysis and/or output.
• a database is optimized for speed and/or flexibility of output if it provides speed or flexibility of output which is superior to the speed or flexibility of some other possible configuration.
• "optimized” does not necessarily require a mathematically precise optimization.
• three general types of tables are provided in step 1028: a plurality of category tables (including rollup tables where appropriate), at least one detail table and at least one entry table.
• a category table is provided for each manner in which a particular data point or record can be categorized.
• a category table is provided for each manner in which a particular data point or record can be categorized.
• the "create database table" function 905 is provided in a driver 804 which is configured for use with the database as depicted in Figs. 4A through 4F, the new database, as depicted in Fig.
• each record or item in each ofthe category tables 1202 is associated with an index for use in a detail table as described below.
• the detail table 1240 will be provided that will, once populated, have a record for each accounting entry or transaction in the data source or sources 806.
• fields 1244 namely a date field 1242b, description field 1242c, quantity field 1242d, and amount field 1242e are provided corresponding to a transaction in the data source (Fig. 4 A).
• An index field 1242a is provided for storing an identifier number or index number for each record.
• a separate field is provided for storing an indication of any appropriate information for each ofthe categories defined in the load general information step 1022, in the example of Fig.
• the structure ofthe database depicted in Fig. 12 may be contrasted with the structure ofthe data source depicted in Figs. 4 A through 4F and the structure of the data source depicted in Fig. 1.
• the manner in which a particular transaction (Fig. 4A) was associated with a particular account (Fig. 4B) was indicated in a separate link table (Fig. 4D), whereas in the embodiment of Fig. 12 the index for the appropriate account 1242f will be stored in its own field of the same record which contains the transaction information 1244.
• the database 808 having tables as depicted in Fig. 12 can store the information found in either an data source as depicted in Fig.
• the structure or architecture ofthe database in the example of Fig. 12 is different from that of either the data source depicted in Fig. 1 or the data source depicted in Figs. 4A through 4F.
• the relational database structure of Fig. 12 is different from the flat file structure depicted in Fig. 1 even though the type of information stored in the two organizations is similar.
• the main procedure 802 can be configured to save the general information 1030 (loaded in step 1022) and save the data definition 1032 (loaded in step 1024), e.g., in additional tables provided in the databases 808, e.g., for use in later steps of Fig. 10 and/or in updates performed at a later time.
• the main procedure 802 uses appropriate functions 908 of one or more drivers 804, to load into the new databases 808, data definition codes (e.g., field width, data type, etc.) for the various tables created in step 1028 (1033).
• data definition codes e.g., field width, data type, etc.
• rollup information is also loaded at this time.
• rollup information refers to information used for defining sub-categories of data, i.e., groups of items within one ofthe category tables.
• the employee category table may be associated with a location or region code, e.g., to identify the location or site or region where each company employee is located or has responsibility. For example, there may be numerous sales employee for a company, each of which is associated with a sales region.
• the various products of a company may be rolled up into or grouped into product lines.
• a field 1238 for a location rollup code the structure of Fig. 12 makes it possible to output a report which is grouped by sales region. Rollups may also be used for providing statistical analysis of group data such as averages, means, standard deviations, etc.
• the field for the location rollup code 1238 is shown as being a field ofthe employee category table 1216, a location field 1238 could also be provided in the detail table if desired.
• a function for loading a rollup code 908 operates will vary depending on the type of source data 806 with which it is configured to act, and thus it is provided as a function of various drivers 804 so that different programming instructions can be provided for use with different types of source data.
• a "define rollup code" function 908 may be provided in a driver 804 configured for use in connection with the database depicted in Figs. 4A through 4F.
• a location field is already defined in the employee table 416 which can be directly used for location code pu ⁇ oses.
• a location rollup may not be possible for data obtained from the data source in Fig. 1. If, however, there is, e.g., another file which provides the home address for each employee in the company, it may be possible to use, e.g., the state of residence of each salesperson to infer the sales region that person is responsible for and thus define a location rollup code inferential ly.
• a "define rollup code" function may include access to information which can be used for defining a rollup code. For example, in connection with the data source depicted in Fig.
• the "define data rollup" function 908 may contain, within that function, a table indicating, for each potential temperature station 612, whether that station is a northem hemisphere or a southern hemisphere station and thus could create a hemisphere rollup code on such basis. In some cases it may be desirable to provide for word recognition and/or searching for key words in the database in order to define additional rollups and/or structures. In some situations, the rollup code will be relative to information which was not used in the data source as a basis for analyzing or grouping data (e.g., the hemisphere rollup code for the data source of Fig. 6).
• providing a rollup code involves providing an enhancement ofthe data by automatically providing additional elements which were not available (or at least not used) for outputting or analyzing information in the data source 806.
• data is classified into categories and then grouped by an in-depth analysis ofthe data source's data.
• the system may be configured to recognize and add certain reporting relationships which can be determined with a degree of definiteness depending on the data in the information sources 806a - 806d or the structure of that data.
• sales information may have sales which are organized by sales districts (either expressly in the information sources 806a - 806d or inferred, e.g., using salesman addresses as described above). These districts may be further viewed or analyzed, according to an embodiment ofthe present invention, by other groupings such as by two or more vice presidents, each of which may be responsible for two or more sales districts, provided the relationship ofthe vice president to the sales districts can be found expressly in or inferred from the information sources 806a - 806d.
• such additional roll-ups or enhancements are defined only if the data indicates it may be meaningful to view the data according to the roll-up criteria.
• an enhancement or additional relationship is not automatically added unless there exists more than one value for a particular field or parameter but, preferably, fewer than the number of records containing that field or parameter. For example, if there are at least two different vice presidents responsible for different districts, then it may be meaningful to look at the sales according to the amount of sales corresponding to each vice president.
• the system will preferably examine the data in the source databases 806a - 806d to determine whether, for example, there is more than one salesperson, more than one region and/or more than one product. If there is more than one value in a field for the various records of a database, the system can be configured to automatically provide an option to view the data according to that field, e.g., grouped by the average or sum of values (such as average or sum of sales, expenses, etc.).
• this is achieved by creating a Microsoft Excel® "pivot" table.
• the table of Fig. 4D can be used to determine, for any transaction, which employee is associated with that transaction.
• Table 4C can be used to determine whether, for a given employee, that employee is associated with the Northwest region, Southeast region or the Central region. If all transactions are associated with employees in the Central region, there is no point in displaying transactions by region. However, if the transactions in table 4D are associated with employees in at least two different regions, the system can be configured to automatically generate a roll-up which displays the average or the sum of transactions (and/or transactions of a particular account type as determined from Fig. 4B) broken-down by the location or region associated with the employee who is associated with that transaction.
• optional reference fields refer to fields which will typically not be used for grouping data such as free-text fields (comments, memo fields, invoice numbers, etc.) but which may be desired for inclusion in reports, etc.
• step 1034 defines the categories (e.g. Account,
• Step 1028 creates the category tables defined in step 1024 (and any other tables defined in step 1022).
• Step 1032 saves the data definition, typically in one standard table.
• Steps 1033 and 1034 load and save data definitions and rollup codes.
• the main process 802 calls or activates a function 909 in the appropriate drivers 804 to load data 1036, save the data 1038, and repeat the process 1039 until all the desired data has been loaded and saved 1040.
• the new databases 808 will be populated with data from one or more data sources 806.
• the main process 802 can now build and, if desired, execute, data queries such as summarization queries 1042.
• data queries such as summarization queries 1042.
• the first type of query may be provided which will be common to all new databases 808 which are created using the procedure of Fig. 10, such as a query which provides the number of entries in the detail table or the number of entries for a given date range (e.g., per quarter).
• Other queries may be built at least partially depending on the general information and data definition obtained with respect to one or more of the data sources 806, including rollups which may have been provided and accordingly could, if desired, be provided as part of a driver 804, specific to a particular data source.
• a third type of query may be provided to replicate or include queries or reports which were used in the original data source (e.g., as depicted in Fig. 4E).
• the system will perform certain audits or checks ofthe data, automatically, to verify that the system operated properly.
• Various types of checks can be included. Samples ofthe data in the new database 808 (or, if desired, all of the data) can be compared with corresponding data in the source 806a - 806d to verify that data was not corrupted.
• the new database 808 can be checked to verify that necessary structures are present. For example, in the case of an accounting database, the new database 808 can be automatically checked to determine that the required account parts are present.
• the new database 808 can be checked for empty sets, such as by making sure that rollups, or other enhancements that have been defined, are non-empty.
• the main procedure 802 may close the tables and databases 1044 and the dynamic drivers 1046, e.g., to free up memory.
• a major contemplated use of the present invention is in connection with providing standardized and/or enhanced reporting and analysis of data in one or more data sources, it is also possible to use the present invention in connection with data entry and data storage by using a database management system (e.g., Microsoft Access ® , Excel ® , FoxPro ® , Btrieve ® , etc.) in connection with the new databases 808.
• a database management system e.g., Microsoft Access ® , Excel ® , FoxPro ® , Btrieve ® , etc.
• the present invention facilitates standardization of reporting and analysis despite a variety in the brands or types of data sources used.
• the present invention provides for a system which can be optimized or otherwise provide improved performance in outputting or reporting data.
• the present invention provides for data reporting and analysis capability which is enhanced compared to data reporting and analysis of the data sources. Via sophisticated interrogation ofthe source data, in the context of an accounting system the present invention is able to mirror the chart of accounts setup in the data source.
• the process extracts some or all defined rollups, optional reference fields, accounting period information.
• the present invention is less labor-intensive and less time-intensive than previous methods, in some cases making it possible to populate new databases 808 in a matter of a few minutes or hours in situations that would require several days or weeks under previous methods.
• the driver 804 can be configured to detect, analyze and maintain, in the new databases 808, any security, passwords, permissions, etc., that are used in the data source 806. In this way it is not necessary for a systems administrator to need to maintain a new and separate set of accounts, passwords, permissions, etc., for the new databases 808 in addition to that maintained with the original data source 806.
• the system can be configured to perform updates substantially at predetermined intervals such as daily, weekly, etc.
• a number of variations and modifications ofthe present invention can also be used. It is possible to use some aspects ofthe invention without using others. For example, it is possible to provide for populating a new database 808 without defining new or additional rollups.
• the various drivers 804 can be provided as separate DLL files and are dynamic in the sense that as many as desired can be added simply by storing additional DLL files in the proper directory, it would also be possible to make an operable version of the invention in which the function performed by the function modules are provided as portions of or subroutines called by the main procedure 802 rather than being separately stored modules.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Databases & Information Systems (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Software Systems (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Communication Control (AREA)
• Debugging And Monitoring (AREA)

Priority Applications (6)

Applications Claiming Priority (4)

Publications (1)

Family

ID=27078483

Family Applications (1)

Country Status (8)

Cited By (8)

Families Citing this family (9)

Citations (1)

Family Cites Families (4)

• 1996
  • 1996-12-20 AU AU13398/97A patent/AU729275B2/en not_active Ceased
  • 1996-12-20 KR KR1019980705075A patent/KR100538547B1/ko not_active IP Right Cessation
  • 1996-12-20 WO PCT/US1996/020366 patent/WO1997024658A1/en active Search and Examination
  • 1996-12-20 CA CA002240663A patent/CA2240663C/en not_active Expired - Fee Related
  • 1996-12-20 IL IL12512996A patent/IL125129A0/xx not_active IP Right Cessation
  • 1996-12-20 EP EP96944909A patent/EP0954777A4/en not_active Withdrawn
  • 1996-12-20 CN CNB961994754A patent/CN1163821C/zh not_active Expired - Fee Related
• 1999
  • 1999-07-13 HK HK99103011A patent/HK1018328A1/xx not_active IP Right Cessation

Patent Citations (1)

Also Published As

Similar Documents

Legal Events