US10922301B1 - Apparatus, computer program, and method for trigger-based tracking of database modifications - Google Patents
Apparatus, computer program, and method for trigger-based tracking of database modifications Download PDFInfo
- Publication number
- US10922301B1 US10922301B1 US15/220,286 US201615220286A US10922301B1 US 10922301 B1 US10922301 B1 US 10922301B1 US 201615220286 A US201615220286 A US 201615220286A US 10922301 B1 US10922301 B1 US 10922301B1
- Authority
- US
- United States
- Prior art keywords
- database
- package file
- triggers
- input tables
- log table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000012986 modification Methods 0.000 title claims abstract description 51
- 230000004048 modification Effects 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000004590 computer program Methods 0.000 title claims abstract description 16
- 230000001960 triggered effect Effects 0.000 claims abstract description 8
- 230000004044 response Effects 0.000 claims abstract description 6
- 239000000284 extract Substances 0.000 claims description 6
- 238000004519 manufacturing process Methods 0.000 claims description 6
- 238000011161 development Methods 0.000 claims description 3
- 238000009434 installation Methods 0.000 claims description 3
- 230000003213 activating effect Effects 0.000 claims description 2
- 238000012856 packing Methods 0.000 claims 1
- 238000003860 storage Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 10
- 238000004806 packaging method and process Methods 0.000 description 5
- 238000010200 validation analysis Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000008676 import Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000002596 correlated effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1451—Management of the data involved in backup or backup restore by selection of backup contents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
- G06F16/24565—Triggers; Constraints
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
Definitions
- the present invention relates to database systems, and more particularly to tracking database modifications.
- An apparatus, computer program, and method are provided for trigger-based tracking of database modifications.
- One or more input tables associated with a database are identified. Further, one or more triggers are generated, based on the one or more input tables. Such one or more triggers are then installed in connection with an application configured for modifying the database.
- one or more modifications to the database are tracked. Further, information is stored where such information is associated with the one or more modifications to the database.
- FIG. 1 illustrates a method for trigger-based tracking of database modifications, in accordance with one embodiment.
- FIG. 2 illustrates a system trigger-based tracking of database modifications, in accordance with one embodiment.
- FIG. 3 illustrates operation of a system for trigger-based tracking of database modifications, in accordance with one embodiment.
- FIG. 4 illustrates operation of a packager module method, in accordance with one embodiment.
- FIG. 5 illustrates operation of a deployer module method, in accordance with one embodiment.
- FIG. 6 illustrates a network architecture, in accordance with one possible embodiment.
- FIG. 7 illustrates an exemplary system, in accordance with one embodiment.
- FIG. 1 illustrates a method 100 for trigger-based tracking of database modifications, in accordance with one embodiment.
- the method 100 includes the identification of one or more input tables associated with a database, operation 102 .
- the aforementioned one or more input tables may each include any data structure that includes information on any changes (additions, deletions, modifications) to a database.
- the identification of such input tables may occur in any desired manner (e.g. a push or pull mechanism, a look-up operation in a predetermined folder, etc.).
- the input table may be stored in any of multiple locations when identified (e.g. as part of an application, the database, etc.).
- one or more triggers are generated, based on the one or more input tables.
- such triggers may include any data structure and/or code that is capable of being used to trigger subsequent processing.
- the trigger(s) may be generated in any desired manner that is based on the one or more input tables.
- the data structure and/or code may, in one embodiment, be generated using a template or automated script generator that processes the input table(s) in a manner that allows the trigger(s) to occur when any change occurs to the database.
- the one or more triggers are then installed in connection with an application configured for modifying the database. See operation 106 .
- the trigger(s) may be installed by itself and/or with other accompanying software/data (e.g. plug-in, script, interface, etc.) that, in turn, is installed in connection with the application (e.g. by becoming pan or an add-on of the application, by interfacing with the application, etc.).
- the application may include any software that is designed to make changes to a database.
- the triggers(s) may be triggered based on execution of database mark-up language (DML) operations.
- DML database mark-up language
- the triggers may monitor and/or otherwise “hook” such DML code/operations, for the purpose of triggering the trigger(s).
- the aforementioned application after the installation of the one or more triggers, is capable of switching between a first mode of operation for tracking the one or more modifications to the database and a second mode of operation for avoiding the tracking. Further, the switching between such different modes may be under the control of the application. More information regarding such feature will be set forth in greater detail during reference to other embodiments outlined in subsequent figures.
- the database modification(s) may be tracked in any desired manner that allows for the storage and possible subsequent use of such data involved in such database modification(s).
- the tracking may enable the database modification(s) (and any data associated therewith) to be stored as storage.
- storage may be a database separate from the aforementioned database (that was the subject of the database modification(s)).
- such storage may take the form of a log table.
- information associated with the database modification(s) is stored.
- information may include any description of the database modification(s), data subject of the database modification(s), and/or any other information that may be derived from tracking the database modification(s).
- the information may include a timestamp, a sequence identifier, data that is subject of the modification, etc.
- the information stored in operation 110 may be used in any desired manner.
- a report on the one or more modifications to the database may be generated, utilizing the information.
- a subset of the information associated with the one or more modifications to the database may be extracted, and packaged into a file.
- the file may be validated against one or more database schemas of one or more other separate database.
- a list of tables may be built for the file, for use in rolling back any failed deployment of the file.
- the file may be deployed to the one or more other separate database, based on the validation.
- the file may be deployed to the separate databases simultaneously.
- FIG. 2 illustrates a system 200 trigger-based tracking of database modifications, in accordance with one embodiment.
- the system 200 may be implemented in the context of any one or more of the embodiments set forth in any previous and/or subsequent figure(s) and/or description thereof. However, it is to be appreciated that the system 200 may be implemented in the context of any desired environment.
- one or more applications 202 are provided for accessing and making modifications to one or more first databases 204 .
- an apparatus 206 In communication with the application(s) 202 and/or the first database(s) 204 is an apparatus 206 including a plurality of components for tracking the aforementioned modifications to the first database(s) 204 carried out by the application(s) 202 .
- the apparatus 206 includes a tracker module 208 that has access to one or more second databases 210 for storing information, the second database(s) 210 being associated with the database modification(s) to the first database(s) 204 made by the application(s) 202 .
- the apparatus 206 further includes a packager module 212 in communication with the tracker module 208 for packaging at least a portion of the aforementioned information (collected by the tracker module 208 ) into a package file. Still yet, the apparatus 206 further includes a deployer module 214 that includes services to deploy the package file to one or more third databases 216 after some processing (e.g. validation, etc.).
- the system 200 is configured to: (i) perform entire data changes on a separate database associated within a development or master reference data environment, (ii) pack such changes and quickly deploy the same, and (iii) perform tests on staging environments. Once the package is tested successfully, it is ready to be deployed on a production system and/or any other equivalent target, while all the data changes are tracked and recorded. Decision exit points may also be provided to review the changes before the implementation into a desired system.
- the system 200 is a database trigger-based solution. It may be attached to a data development area and may track each data change done by multiples sources on a database level. It is thereby transparent with respect to a main application process and records changes implicitly in a pre-build tracking log table while tracking such changes sequentially with the right order. In use, each committed set of data changes is committed, packed, and deployed to pre-defined target system(s). More information will now be set forth regarding the operation of each of the foregoing components.
- FIG. 3 illustrates operation of a system 300 for trigger-based tracking of database modifications, in accordance with one embodiment.
- the system 200 may be implemented in the context of any one or more of the embodiments set forth in any previous and/or subsequent figure(s) and/or description thereof.
- the operation set forth in FIG. 3 may reflect an operation of the system 200 of FIG. 2 .
- the system 200 may be implemented in the context of any desired environment.
- the system 300 includes one or more applications 302 , a tracker module 304 , a packager module 306 , and a deployer module 308 .
- the tracker module 304 includes database trigger-based code that captures any changes that are done on a database table level and records the same in a predefined log table.
- the tracker module 304 is explicitly or implicitly used by the application(s) 302 in order to activate the tracker module 304 right before activating the code that is responsible for modifying the database (e.g. first database(s) 204 of FIG. 2 , etc.) and to deactivate the same once completed. As long as the triggers are activated, the data changes are accumulated in a log table.
- the data flow is driven by the application(s) 302 (and not necessarily by the tracker module 304 , in one possible embodiment), such that the application(s) 302 triggers the tracker module 304 on and off as needed.
- Any type of information may be recorded this way, including, but not limited to, applicative information, session information, system information, timestamp, transactions sequence, etc.
- the tracker module 304 module in operation 1 , receives input tables for the purpose of generating triggers in operation 2 . Upon receipt of such triggers, the application(s) 302 acknowledges the same in operation 3 . Under the control of the application(s) 302 , a tracking mode may be enabled in operation 4 and, in such mode, data changes may be tracked in operation 5 , until the application(s) 302 terminates the aforementioned tracking mode in operation 6 .
- the triggers are disabled in operation 7 and such disablement is acknowledged by both the application(s) 302 and the tracker module 304 in operations 8 - 9 . Still yet, as indicated in operation 10 , the data changes are stored in a log table.
- the tracker module 304 thus generates the core information for subsequent modules (e.g. packager module 306 , and a deployer module 308 , etc.). This is accomplished by the tracker module 304 being governed by external processes that perform DML (e.g. database transactions) operations on a database and that implicitly fire auto-generate triggers that record any changes. As mentioned earlier, a first step to accomplish this involves the generation of the triggers based on input tables list, where such triggers catch the foregoing DML operations and, accordingly, insert records into the log table, including a timestamp and continue transaction sequence number (e.g. a sequence tracking identifier, etc.).
- DML e.g. database transactions
- This transaction sequence number may be used to determine a latest operation on a record, during a subsequent process managed by the packager module 306 , as will soon become apparent.
- the tracker module 304 disables or drops the triggers and stops the accumulating. In one possible embodiment, the triggers may remain activated, which may allow the system to keep collecting data changes information. Once the tracker module 304 has completed its tasks (or based on other business-driven logic) or the business decides to extract the incremental changes currently in the system, the packager module 306 is engaged.
- the packager module 306 processes the data that is captured (in the log table) during the operation of the tracker module 304 . Further, the packager module 306 prepares a package which may be delivered and installed in an equivalent target system, as will soon become apparent. The packager module 306 further prepares all files required for data load processing. For instance. Table 1 illustrates various files that are prepared in such manner.
- the packager module 306 In the case of the database records content files, the packager module 306 generates a filtering mechanism that instructs the ETL (e.g.—Oracle DATAPUMP) to exact the records. Thus, only desired data is extracted and packed by the packager module 306 and later on to be deployed.
- the files in Table 1 may be generated for each sub-system/schema that is defined in a runtime profile of the packager module 306 , and the packager module 306 generates a single package file which contains the information above in a manner that is ready for deployment.
- such module in operation 11 , accesses the data changes in the log table and processes the same in operation 12 so that a subset of the data may be extracted in operation 13 .
- such extracted data may be packaged into a single package, for being communicated to the deployer module 308 in operation 15 . More information regarding such packager module operations will be set forth during the description of the method 400 of FIG. 4 .
- the deployer module 308 processes the input package file (or files) from the packager module 306 and deploys the same into predefined database schemas. With all information for deployment included in the package file, the deployer module 308 deploys the same package on several products as well as several database schemas per product. Also, the deployer module 308 has the ability to read multiple packages, to validate, and to deploy the same in the correct order.
- the deployer module 308 receives and opens the package file or several package files (e.g. TAR file(s), etc.), in order to deploy them, one by one, on a predefined database schemas list.
- the deployer module 308 unpacks the package file(s) for validation purposes. Specifically, the deployer module 308 obtains sequence tracking information from the package file and validates such number against all of the target database schemas. To validate, the deployer module 308 ensures that a sequence tracking identifier in the package matches a current tracking sequence on each schema. The deployer module 308 also checks and confirms a data loading configuration for all targets for the upcoming data loading. Further, when multiple packages are being validated, the deployer module 308 validates the sequential order of all packages.
- the deployer module 308 builds a table(s) lists based on the package information, and performs backups of the table(s) that are going to be changed during deployment. This may be done in order to preserve an ability to roll back to the current backup point in case of any failure during deployment.
- the deployer module 308 starts sub-processes in parallel, and starts the deployment. See operations 18 - 19 . More information regarding such deployer module operations will be set forth during the description of the method 500 of FIG. 5 .
- FIG. 4 illustrates operation of a packager module method 400 , in accordance with one embodiment.
- the method 400 may be implemented in the context of any one or more of the embodiments set forth in any previous and/or subsequent figure(s) and/or description thereof.
- the method 400 may reflect an operation of the packager module 212 of FIG. 2 and/or the packager module 306 of FIG. 3 .
- the method 400 may be implemented in the context of any desired environment.
- the method 400 begins by validating the log table (e.g. including determining whether it contains changes), per operation 402 .
- the log table e.g. including determining whether it contains changes
- the following information in Table 2 is generated.
- the method 400 continues by starting the actual extraction of data in operation 404 based on the configuration files that were generated in operation 402 . Further, sub-processes are incrementally extracted for all predefined database schemas/products. Thereafter, data content files are received from a local directory, and packed together in a single package file.
- a report is generated for all changes in operation 410 .
- such report may be sent to a predefined email/contact list.
- operations 402 - 410 may be repeated for each package to be created, per decision 408 . Thereafter, the package file(s) is ready for delivery and deployment, per operation 412 .
- FIG. 5 illustrates operation of a deployer module method 500 , in accordance with one embodiment.
- the method 500 may be implemented in the context of any one or more of the embodiments set forth in any previous and/or subsequent figure(s) and/or description thereof.
- the method 500 may reflect an operation of the deployer module 214 of FIG. 2 and/or the deployer module 308 of FIG. 3 .
- the method 500 may be implemented in the context of any desired environment.
- the method 500 begins in operation 502 by identifying a package file(s) (e.g. that is ready for deployment per operation 412 of FIG. 4 , etc.).
- decision 504 it is determined whether the input package sequential identifier and that of the target schema match, in order to ensure that a proper deployment order is adhered to (i.e. so it will be deployed sequentially). If not, deployment is aborted until there is a match per operation 506 .
- the ETL e.g.—Oracle DATAPUMP
- the ETL is checked to ensure that it is configured and can be activated appropriately. Specifically, it is confirmed that the identifier in the target database matches the identifier in the deployment package. It should be noted that these checks are activated during a pre-deployment phase in order to eliminate, in advance, potential error or failures (and to ensure smooth deployment).
- the target tables are backed up in operation 512 . This allows for automatic rollback in case of failure. Thereafter, deployment is completed per operation 514 after additional steps are performed, including, but not limited to, determining where each data contents file is located, in connection with its related server. Further, relevant records are deleted by primary-key, as defined in the package files. Such files are those that will be replaced with new records that exist in the package to be deployed. Further, before an actual import, a dynamic external operation may optionally be performed on the database.
- all imports may be done in parallel while the deployer module waits for all to finish successfully, before going to next step. Again, in case of failure, all database schemas may be rolled backed to the initial backup point. Once all imports are complete, the deployer module retrieves the next package, if any, and all operations (except the backup) are repeated. When all are complete, the deployer module updates the new package sequential identifier in the database and completes the process.
- the above techniques thus overcome challenges by capturing exact database changes that were made, with the ability to track and record each extraction in an intelligent and efficient way. Also, all sub-system databases are synchronized using a unique tracking identifier for each extraction toward the deployment in the equivalent production system. Further, any data change (e.g. new, delete, update, etc.) in connection with the database are tracked. Further, the modified records from the database are incrementally extracted to files, and the whole data is packed within a single package file for delivery. Still yet, the package is deployed into the multiples database targets in a manner that ensures data consistency while maintaining tracking identification for each logical data package (across distributed sub-systems).
- This design thus incrementally extracts and deploys data with an ability also to export character large objects (CLOB) and binary large objects (BLOB).
- CLOB character large objects
- BLOB binary large objects
- a row level extract process may also be used to dramatically reduce timing and storage consumption in connection with data extracts at the schema- or table-level. Further, full tracking and reporting abilities are included which provide information for a business on upcoming new data, before the actual deployment. Thus, in various embodiments, various possible features are available.
- data may be selected and transferred from one or more database sources in a manner that enables fast and smooth data transfer with full control on the transferred data.
- a fully automated tracking mechanism (with no manual processing/involvement, in one embodiment) may be provided, in order to identify the exact changed data and the log for packaging and deployment purposes via later stages.
- certain scenarios from an end user may be recorded for use in connection with an automation packaging and deployment tool for replaying those scenarios to any equivalent database targets. In one possible embodiment, this may be accomplished in a manner independent on any user interface being implemented.
- FIG. 6 illustrates a network architecture 600 , in accordance with one embodiment. As shown, at least one network 602 is provided. In various embodiments, any one or more components/features set forth during the description of any previous figure(s) may be implemented in connection with any one or more of the components of the at least one network 602 .
- the network 602 may take any form including, but not limited to a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, peer-to-peer network, cable network, etc. While only one network is shown, it should be understood that two or more similar or different networks 602 may be provided.
- LAN local area network
- WAN wide area network
- peer-to-peer network cable network
- Coupled to the network 602 is a plurality of devices.
- a server computer 612 and an end user computer 608 may be coupled to the network 602 for communication purposes.
- Such end user computer 608 may include a desktop computer, lap-top computer, and/or any other type of logic.
- various other devices may be coupled to the network 602 including a personal digital assistant (PDA) device 610 , a mobile phone device 606 , a television 604 , etc.
- PDA personal digital assistant
- FIG. 7 illustrates an exemplary system 700 , in accordance with one embodiment.
- the system 700 may be implemented in the context of any of the devices of the network architecture 600 of FIG. 6 .
- the system 700 may be implemented in any desired environment.
- a system 700 including at least one central processor 702 which is connected to a bus 712 .
- the system 700 also includes main memory 704 [e.g., hard disk drive, solid state drive, random access memory (RAM), etc.].
- main memory 704 e.g., hard disk drive, solid state drive, random access memory (RAM), etc.
- the system 700 also includes a graphics processor 708 and a display 710 .
- the system 700 may also include a secondary storage 706 .
- the secondary storage 706 includes, for example, a hard disk drive and/or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, etc.
- the removable storage drive reads from and/or writes to a removable storage unit in a well-known manner.
- Computer programs, or computer control logic algorithms may be stored in the main memory 704 , the secondary storage 706 , and/or any other memory, for that matter. Such computer programs, when executed, enable the system 700 to perform various functions (as set forth above, for example).
- Memory 704 , secondary storage 706 and/or any other storage are possible examples of non-transitory computer-readable media.
- a “computer-readable medium” includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer readable medium and execute the instructions for carrying out the described methods.
- Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format.
- a non-exhaustive list of conventional exemplary computer readable medium includes: a portable computer diskette; a RAM; a ROM; an erasable programmable read only memory (EPROM or flash memory); optical storage devices, including a portable compact disc (CD), a portable digital video disc (DVD), a high definition DVD (HD-DVDTM), a BLU-RAY disc; and the like.
- one or more of these system components may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described Figures.
- the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.
- At least one component defined by the claims is implemented at least partially as an electronic hardware component, such as an instruction execution machine (e.g., a processor-based or processor-containing machine) and/or as specialized circuits or circuitry (e.g., discreet logic gates interconnected to perform a specialized function).
- an instruction execution machine e.g., a processor-based or processor-containing machine
- specialized circuits or circuitry e.g., discreet logic gates interconnected to perform a specialized function.
- Other components may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other components may be combined, some may be omitted altogether, and additional components may be added while still achieving the functionality described herein.
- the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of what is claimed.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An apparatus, computer program, and method are provided for trigger-based tracking of database modifications. One or more input tables associated with a database are identified. Further, one or mom triggers are generated, based on the one or more input tables. Such one or more triggers are then installed in connection with an application configured for modifying the database. In response to at least one of the one or more triggers being triggered, one or more modifications to the database are tracked. Further, information is stored where such information is associated with the one or more modifications to the database.
Description
The present invention relates to database systems, and more particularly to tracking database modifications.
When working with software that accomplishes mission critical, high availability/demand services at a large scale, such as in the telecommunication space, it is important to ensure minimal deployment downtime, typically resulting from ongoing changes in reference data (e.g. business rules, pricing models etc.) performed by multiple sources (e.g. manual changes, application user interfaces, batch operations, internal/external systems, etc.). Since such changes are used by applications, there is a need to have an ability to track and collect the incremental data changes, pack such incremental data changes, and deploy the same in production systems. Further, these capabilities need to be executed with minimal resources and minimal downtime of live systems, while still validating data consistency across several sub-systems in pre- and post-deployment stages. Typically, this is achieved using finger pointing capabilities as to where the changes were made, as well as using the ability to collect such changes in a row level resolution for further deployment on various systems destinations.
Thus, efficient and accurate data transfer is vital in a production life cycle and requires frequent reference data changes to follow the dynamic changes of live production live systems. However, most of the data extraction technics are merely relying upon a full dump of data or pre-defined rules, in order to extract partial of data thereafter. Those solutions are time- and resource-consuming, and usually result in inaccuracies and, therefore, data inconsistency.
There is thus a need for addressing these and/or other issues associated with the prior art.
An apparatus, computer program, and method are provided for trigger-based tracking of database modifications. One or more input tables associated with a database are identified. Further, one or more triggers are generated, based on the one or more input tables. Such one or more triggers are then installed in connection with an application configured for modifying the database. In response to at least one of the one or more triggers being triggered, one or more modifications to the database are tracked. Further, information is stored where such information is associated with the one or more modifications to the database.
In operation 104, one or more triggers are generated, based on the one or more input tables. In the present description, such triggers may include any data structure and/or code that is capable of being used to trigger subsequent processing. Further, the trigger(s) may be generated in any desired manner that is based on the one or more input tables. For example, the data structure and/or code may, in one embodiment, be generated using a template or automated script generator that processes the input table(s) in a manner that allows the trigger(s) to occur when any change occurs to the database.
After generation, the one or more triggers are then installed in connection with an application configured for modifying the database. See operation 106. As an option, the trigger(s) may be installed by itself and/or with other accompanying software/data (e.g. plug-in, script, interface, etc.) that, in turn, is installed in connection with the application (e.g. by becoming pan or an add-on of the application, by interfacing with the application, etc.). Further, it should be noted that, in the context of the present description, the application may include any software that is designed to make changes to a database.
In response to at least one of the one or more triggers being triggered, one or more modifications to the database are tracked per operation 108. In one embodiment, the triggers(s) may be triggered based on execution of database mark-up language (DML) operations. For example, to the extent that DML code is executed to cause operations on the database in connection with the input table, the triggers may monitor and/or otherwise “hook” such DML code/operations, for the purpose of triggering the trigger(s).
As an option, in one embodiment, the aforementioned application, after the installation of the one or more triggers, is capable of switching between a first mode of operation for tracking the one or more modifications to the database and a second mode of operation for avoiding the tracking. Further, the switching between such different modes may be under the control of the application. More information regarding such feature will be set forth in greater detail during reference to other embodiments outlined in subsequent figures.
Once triggered, the database modification(s) may be tracked in any desired manner that allows for the storage and possible subsequent use of such data involved in such database modification(s). In one embodiment, the tracking may enable the database modification(s) (and any data associated therewith) to be stored as storage. For example, such storage may be a database separate from the aforementioned database (that was the subject of the database modification(s)). As an option, such storage may take the form of a log table.
To this end, per operation 110, information associated with the database modification(s) is stored. In various embodiments, such information may include any description of the database modification(s), data subject of the database modification(s), and/or any other information that may be derived from tracking the database modification(s). Just by way of example, in various embodiments, the information may include a timestamp, a sequence identifier, data that is subject of the modification, etc.
By this design, the information stored in operation 110 may be used in any desired manner. For example, as will be described later in the context of subsequent embodiments, a report on the one or more modifications to the database may be generated, utilizing the information. As another option, a subset of the information associated with the one or more modifications to the database may be extracted, and packaged into a file. Still yet, the file may be validated against one or more database schemas of one or more other separate database. Further, a list of tables may be built for the file, for use in rolling back any failed deployment of the file. To this end, the file may be deployed to the one or more other separate database, based on the validation. As a further option, the file may be deployed to the separate databases simultaneously.
More illustrative information will now be set forth regarding various optional architectures and uses in which the foregoing method may or may not be implemented, per the desires of the user. It should be noted that the following information is set forth for illustrative purposes and should not be construed as limiting in any manner. Any of the following features may be optionally incorporated with or without the exclusion of other features described.
As shown, one or more applications 202 are provided for accessing and making modifications to one or more first databases 204. In communication with the application(s) 202 and/or the first database(s) 204 is an apparatus 206 including a plurality of components for tracking the aforementioned modifications to the first database(s) 204 carried out by the application(s) 202. To accomplish this, the apparatus 206 includes a tracker module 208 that has access to one or more second databases 210 for storing information, the second database(s) 210 being associated with the database modification(s) to the first database(s) 204 made by the application(s) 202.
The apparatus 206 further includes a packager module 212 in communication with the tracker module 208 for packaging at least a portion of the aforementioned information (collected by the tracker module 208) into a package file. Still yet, the apparatus 206 further includes a deployer module 214 that includes services to deploy the package file to one or more third databases 216 after some processing (e.g. validation, etc.).
Thus, the system 200 is configured to: (i) perform entire data changes on a separate database associated within a development or master reference data environment, (ii) pack such changes and quickly deploy the same, and (iii) perform tests on staging environments. Once the package is tested successfully, it is ready to be deployed on a production system and/or any other equivalent target, while all the data changes are tracked and recorded. Decision exit points may also be provided to review the changes before the implementation into a desired system.
To this end, the system 200 is a database trigger-based solution. It may be attached to a data development area and may track each data change done by multiples sources on a database level. It is thereby transparent with respect to a main application process and records changes implicitly in a pre-build tracking log table while tracking such changes sequentially with the right order. In use, each committed set of data changes is committed, packed, and deployed to pre-defined target system(s). More information will now be set forth regarding the operation of each of the foregoing components.
As shown, similar to the system 200 of FIG. 2 , the system 300 includes one or more applications 302, a tracker module 304, a packager module 306, and a deployer module 308. The tracker module 304 includes database trigger-based code that captures any changes that are done on a database table level and records the same in a predefined log table. The tracker module 304 is explicitly or implicitly used by the application(s) 302 in order to activate the tracker module 304 right before activating the code that is responsible for modifying the database (e.g. first database(s) 204 of FIG. 2 , etc.) and to deactivate the same once completed. As long as the triggers are activated, the data changes are accumulated in a log table.
In use, the data flow is driven by the application(s) 302 (and not necessarily by the tracker module 304, in one possible embodiment), such that the application(s) 302 triggers the tracker module 304 on and off as needed. Any type of information may be recorded this way, including, but not limited to, applicative information, session information, system information, timestamp, transactions sequence, etc.
In accordance with one possible example of use in the context of the system 300 of FIG. 3 and, in particular, the application(s) 302 and tracker module 304, the tracker module 304 module, in operation 1, receives input tables for the purpose of generating triggers in operation 2. Upon receipt of such triggers, the application(s) 302 acknowledges the same in operation 3. Under the control of the application(s) 302, a tracking mode may be enabled in operation 4 and, in such mode, data changes may be tracked in operation 5, until the application(s) 302 terminates the aforementioned tracking mode in operation 6. Upon the termination of the tracking mode, the triggers are disabled in operation 7 and such disablement is acknowledged by both the application(s) 302 and the tracker module 304 in operations 8-9. Still yet, as indicated in operation 10, the data changes are stored in a log table.
The tracker module 304 thus generates the core information for subsequent modules (e.g. packager module 306, and a deployer module 308, etc.). This is accomplished by the tracker module 304 being governed by external processes that perform DML (e.g. database transactions) operations on a database and that implicitly fire auto-generate triggers that record any changes. As mentioned earlier, a first step to accomplish this involves the generation of the triggers based on input tables list, where such triggers catch the foregoing DML operations and, accordingly, insert records into the log table, including a timestamp and continue transaction sequence number (e.g. a sequence tracking identifier, etc.). This transaction sequence number may be used to determine a latest operation on a record, during a subsequent process managed by the packager module 306, as will soon become apparent. Once the scenario is done, the tracker module 304 disables or drops the triggers and stops the accumulating. In one possible embodiment, the triggers may remain activated, which may allow the system to keep collecting data changes information. Once the tracker module 304 has completed its tasks (or based on other business-driven logic) or the business decides to extract the incremental changes currently in the system, the packager module 306 is engaged.
The packager module 306 processes the data that is captured (in the log table) during the operation of the tracker module 304. Further, the packager module 306 prepares a package which may be delivered and installed in an equivalent target system, as will soon become apparent. The packager module 306 further prepares all files required for data load processing. For instance. Table 1 illustrates various files that are prepared in such manner.
TABLE 1 | ||
Database records content files, via extract transfer load (ETL), | ||
e.g. - Oracle DATAPUMP | ||
Deployment configuration files (including records identification, | ||
loading configuration setup, etc.) | ||
SQL DML file (correlated with the changes records in the | ||
contents files) | ||
Data changes report file (to present the package contents) | ||
Remark files | ||
In the case of the database records content files, the packager module 306 generates a filtering mechanism that instructs the ETL (e.g.—Oracle DATAPUMP) to exact the records. Thus, only desired data is extracted and packed by the packager module 306 and later on to be deployed. In one embodiment, the files in Table 1 may be generated for each sub-system/schema that is defined in a runtime profile of the packager module 306, and the packager module 306 generates a single package file which contains the information above in a manner that is ready for deployment.
Thus, in accordance with one possible example of use in the context of the system 300 of FIG. 3 and, in particular, the packager module 306, such module, in operation 11, accesses the data changes in the log table and processes the same in operation 12 so that a subset of the data may be extracted in operation 13. Thus, in operation 14, such extracted data may be packaged into a single package, for being communicated to the deployer module 308 in operation 15. More information regarding such packager module operations will be set forth during the description of the method 400 of FIG. 4 .
The deployer module 308 processes the input package file (or files) from the packager module 306 and deploys the same into predefined database schemas. With all information for deployment included in the package file, the deployer module 308 deploys the same package on several products as well as several database schemas per product. Also, the deployer module 308 has the ability to read multiple packages, to validate, and to deploy the same in the correct order.
Thus, in accordance with one possible example of use in the context of the system 300 of FIG. 3 and, in particular, the deployer module 308, such module, in operation 16, receives and opens the package file or several package files (e.g. TAR file(s), etc.), in order to deploy them, one by one, on a predefined database schemas list. In operation 17, the deployer module 308 unpacks the package file(s) for validation purposes. Specifically, the deployer module 308 obtains sequence tracking information from the package file and validates such number against all of the target database schemas. To validate, the deployer module 308 ensures that a sequence tracking identifier in the package matches a current tracking sequence on each schema. The deployer module 308 also checks and confirms a data loading configuration for all targets for the upcoming data loading. Further, when multiple packages are being validated, the deployer module 308 validates the sequential order of all packages.
Once all validation has passed, the deployer module 308 builds a table(s) lists based on the package information, and performs backups of the table(s) that are going to be changed during deployment. This may be done in order to preserve an ability to roll back to the current backup point in case of any failure during deployment. After backup, the deployer module 308 starts sub-processes in parallel, and starts the deployment. See operations 18-19. More information regarding such deployer module operations will be set forth during the description of the method 500 of FIG. 5 .
As shown, the method 400 begins by validating the log table (e.g. including determining whether it contains changes), per operation 402. In an environment involving a multi-source/database configuration, it may also be confirmed that the sequential identification on all source schemas are correlated. Based on the information in the logic table, the following information in Table 2 is generated.
TABLE 2 | ||||
Unique package identification | ||||
List of tables for extraction | ||||
Specific table query to perform an incremental data export | ||||
Default HTML detailed report for the data changes | ||||
(and provide interface to support any other formats) | ||||
Data loading runtime configuration parameters files | ||||
(for both packager/deployer modules). | ||||
Once the pre-packaging validation has passed, the method 400 continues by starting the actual extraction of data in operation 404 based on the configuration files that were generated in operation 402. Further, sub-processes are incrementally extracted for all predefined database schemas/products. Thereafter, data content files are received from a local directory, and packed together in a single package file.
In addition, a report is generated for all changes in operation 410. As an option, such report may be sent to a predefined email/contact list. As shown, operations 402-410 may be repeated for each package to be created, per decision 408. Thereafter, the package file(s) is ready for delivery and deployment, per operation 412.
As shown, the method 500 begins in operation 502 by identifying a package file(s) (e.g. that is ready for deployment per operation 412 of FIG. 4 , etc.). In decision 504, it is determined whether the input package sequential identifier and that of the target schema match, in order to ensure that a proper deployment order is adhered to (i.e. so it will be deployed sequentially). If not, deployment is aborted until there is a match per operation 506.
Further, in operation 508, it is confirmed that database connection details are valid and the database connection can be accomplished. Still yet, in operation 510, the ETL (e.g.—Oracle DATAPUMP) is checked to ensure that it is configured and can be activated appropriately. Specifically, it is confirmed that the identifier in the target database matches the identifier in the deployment package. It should be noted that these checks are activated during a pre-deployment phase in order to eliminate, in advance, potential error or failures (and to ensure smooth deployment).
With continuing reference to FIG. 5 , the target tables are backed up in operation 512. This allows for automatic rollback in case of failure. Thereafter, deployment is completed per operation 514 after additional steps are performed, including, but not limited to, determining where each data contents file is located, in connection with its related server. Further, relevant records are deleted by primary-key, as defined in the package files. Such files are those that will be replaced with new records that exist in the package to be deployed. Further, before an actual import, a dynamic external operation may optionally be performed on the database.
By this design, all imports may be done in parallel while the deployer module waits for all to finish successfully, before going to next step. Again, in case of failure, all database schemas may be rolled backed to the initial backup point. Once all imports are complete, the deployer module retrieves the next package, if any, and all operations (except the backup) are repeated. When all are complete, the deployer module updates the new package sequential identifier in the database and completes the process.
The above techniques thus overcome challenges by capturing exact database changes that were made, with the ability to track and record each extraction in an intelligent and efficient way. Also, all sub-system databases are synchronized using a unique tracking identifier for each extraction toward the deployment in the equivalent production system. Further, any data change (e.g. new, delete, update, etc.) in connection with the database are tracked. Further, the modified records from the database are incrementally extracted to files, and the whole data is packed within a single package file for delivery. Still yet, the package is deployed into the multiples database targets in a manner that ensures data consistency while maintaining tracking identification for each logical data package (across distributed sub-systems).
This design thus incrementally extracts and deploys data with an ability also to export character large objects (CLOB) and binary large objects (BLOB). A row level extract process may also be used to dramatically reduce timing and storage consumption in connection with data extracts at the schema- or table-level. Further, full tracking and reporting abilities are included which provide information for a business on upcoming new data, before the actual deployment. Thus, in various embodiments, various possible features are available.
Firstly, data may be selected and transferred from one or more database sources in a manner that enables fast and smooth data transfer with full control on the transferred data. Further, a fully automated tracking mechanism (with no manual processing/involvement, in one embodiment) may be provided, in order to identify the exact changed data and the log for packaging and deployment purposes via later stages.
Secondly, using tracking techniques by instrumenting the user interface of the application, certain scenarios from an end user may be recorded for use in connection with an automation packaging and deployment tool for replaying those scenarios to any equivalent database targets. In one possible embodiment, this may be accomplished in a manner independent on any user interface being implemented.
Thirdly, using tracking and packaging techniques, various scenarios may be accommodated where lots of changes are related to the same tables, and/or the same record is changed multiple times (which makes an analysis process more complicated and might affect data inconsistency in deployment stage). The various techniques disclosed herein are designed to handle such changes in an optimized way and further to pack and deploy only the most up-to-date data in a single record level.
In the context of the present network architecture 600, the network 602 may take any form including, but not limited to a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, peer-to-peer network, cable network, etc. While only one network is shown, it should be understood that two or more similar or different networks 602 may be provided.
Coupled to the network 602 is a plurality of devices. For example, a server computer 612 and an end user computer 608 may be coupled to the network 602 for communication purposes. Such end user computer 608 may include a desktop computer, lap-top computer, and/or any other type of logic. Still yet, various other devices may be coupled to the network 602 including a personal digital assistant (PDA) device 610, a mobile phone device 606, a television 604, etc.
As shown, a system 700 is provided including at least one central processor 702 which is connected to a bus 712. The system 700 also includes main memory 704 [e.g., hard disk drive, solid state drive, random access memory (RAM), etc.]. The system 700 also includes a graphics processor 708 and a display 710.
The system 700 may also include a secondary storage 706. The secondary storage 706 includes, for example, a hard disk drive and/or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, etc. The removable storage drive reads from and/or writes to a removable storage unit in a well-known manner.
Computer programs, or computer control logic algorithms, may be stored in the main memory 704, the secondary storage 706, and/or any other memory, for that matter. Such computer programs, when executed, enable the system 700 to perform various functions (as set forth above, for example). Memory 704, secondary storage 706 and/or any other storage are possible examples of non-transitory computer-readable media.
It is noted that the techniques described herein, in an aspect, are embodied in executable instructions stored in a computer readable medium for use by or in connection with an instruction execution machine, apparatus, or device, such as a computer-based or processor-containing machine, apparatus, or device. It will be appreciated by those skilled in the art that for some embodiments, other types of computer readable media are included which may store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memory (RAM), read-only memory (ROM), and the like.
As used here, a “computer-readable medium” includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer readable medium and execute the instructions for carrying out the described methods. Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format. A non-exhaustive list of conventional exemplary computer readable medium includes: a portable computer diskette; a RAM; a ROM; an erasable programmable read only memory (EPROM or flash memory); optical storage devices, including a portable compact disc (CD), a portable digital video disc (DVD), a high definition DVD (HD-DVD™), a BLU-RAY disc; and the like.
It should be understood that the arrangement of components illustrated in the Figures described are exemplary and that other arrangements are possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent logical components in some systems configured according to the subject matter disclosed herein.
For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described Figures. In addition, while at least one of these components are implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.
More particularly, at least one component defined by the claims is implemented at least partially as an electronic hardware component, such as an instruction execution machine (e.g., a processor-based or processor-containing machine) and/or as specialized circuits or circuitry (e.g., discreet logic gates interconnected to perform a specialized function). Other components may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other components may be combined, some may be omitted altogether, and additional components may be added while still achieving the functionality described herein. Thus, the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of what is claimed.
In the description above, the subject matter is described with reference to acts and symbolic representations of operations that are performed by one or more devices, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processor of data in a structured form. This manipulation transforms the data or maintains it at locations in the memory system of the computer, which reconfigures or otherwise alters the operation of the device in a manner well understood by those skilled in the art. The data is maintained at physical locations of the memory as data structures that have particular properties defined by the format of the data. However, while the subject matter is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that various of the acts and operations described hereinafter may also be implemented in hardware.
To facilitate an understanding of the subject matter described herein, many aspects are described in terms of sequences of actions. At least one of these aspects defined by the claims is performed by an electronic hardware component. For example, it will be recognized that the various actions may be performed by specialized circuits or circuitry, by program instructions being executed by one or more processors, or by a combination of both. The description herein of any sequence of actions is not intended to imply that the specific order described for performing that sequence must be followed. All methods described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.
The use of the terms “a” and “an” and “the” and similar referents in the context of describing the subject matter (particularly in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. Furthermore, the foregoing description is for the purpose of illustration only, and not for the purpose of limitation, as the scope of protection sought is defined by the claims as set forth hereinafter together with any equivalents thereof entitled to. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illustrate the subject matter and does not pose a limitation on the scope of the subject matter unless otherwise claimed. The use of the term “based on” and other like phrases indicating a condition for bringing about a result, both in the claims and in the written description, is not intended to foreclose any other conditions that bring about that result. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention as claimed.
The embodiments described herein include the one or more modes known to the inventor for carrying out the claimed subject matter. It is to be appreciated that variations of those embodiments will become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventor expects skilled artisans to employ such variations as appropriate, and the inventor intends for the claimed subject matter to be practiced otherwise than as specifically described herein. Accordingly, this claimed subject matter includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed unless otherwise indicated herein or otherwise clearly contradicted by context.
Claims (12)
1. A computer program product comprising computer executable instructions stored on a non-transitory computer readable medium that when executed by a processor instruct the processor to:
identify, by a system, one or more input tables of a database;
generate, by the system, one or more triggers for the one or more input tables, the one or more triggers executable to cause the system to catch operations that modify the one or more input tables of the database;
install, by the system, the one or more triggers in connection with an application configured for modifying the database;
in response to at least one of the one or more triggers being triggered by the application, track, by the system, one or more modifications made to the one or more input tables of the database by the application, including for each of the one or more modifications made to the one or more input tables of the database by the application:
insert, into a log table by the system, a record of the modification that includes a timestamp and a transaction sequence number;
extract, by the system from the log table, a subset of the records in the log table that reflect the latest data in the database, using the transaction sequence number stored in each record of the log table;
pack, by system into a package file, the subset of the records extracted from the log table;
validate, by the system, the package file against a database schema of each of a plurality of additional databases to which the package file is to be deployed;
responsive to validating the package file, simultaneously deploy, by the system, the package file to the plurality of additional databases for making the modifications recorded in the package file to each of the plurality of additional databases;
wherein the application, after the installation of the one or more triggers, is capable of switching the system between a first mode of operation for tracking the one or more modifications made to the one or more input tables of the database and a second mode of operation for avoiding the tracking.
2. The computer program product of claim 1 , wherein the operations are database mark-up language (DML) operations.
3. A method, comprising:
identifying, by a system, one or more input tables of a database;
generating, by the system, one or more triggers for the one or more input tables, the one or more triggers executable to cause the system to catch operations that modify the one or more input tables of the database;
installing, by the system, the one or more triggers in connection with an application configured for modifying the database;
in response to at least one of the one or more triggers being triggered by the application, tracking, by the system, one or more modifications made to the one or more input tables of the database by the application, including for each of the one or more modifications made to the one or more input tables of the database by the application:
insert, into a log table by the system, a record of the modification that includes a timestamp and a transaction sequence number;
extracting, by the system from the log table, a subset of the records in the log table that reflect the latest data in the database, using the transaction sequence number stored in each record of the log table;
packing, by the system into a package file, the subset of the records extracted from the log table;
validating, by the system, the package file against a database schema of each of a plurality of additional databases to which the package file is to be deployed;
responsive to validating the package file, simultaneously deploying, by the system, the package file to the plurality of additional databases for making the modifications recorded in the package file to each of the plurality of additional databases;
wherein the application, after the installation of the one or more triggers, is capable of switching the system between a first mode of operation for tracking the one or more modifications made to the one or more input tables of the database and a second mode of operation for avoiding the tracking.
4. A system, comprising:
at least one processor configured to:
identify, by the system, one or more input tables of a database;
generate, by the system, one or more triggers for the one or more input tables, the one or more triggers executable to cause the system to catch database mark-up language (DML) operations that modify the one or more input tables of the database;
install, by the system, the one or more triggers in connection with an application configured for modifying the database;
in response to at least one of the one or more triggers being triggered by the application, track, by the system, one or more modifications made to the one or more input tables of the database by the application, including for each of the one or more modifications made to the one or more input tables of the database by the application:
insert, into a log table by the system, a record of the modification that includes a timestamp and a transaction sequence number;
extract, by the system from the log table, a subset of the records in the log table that reflect the latest data in the database, using the transaction sequence number stored in each record of the log table;
pack, by the system into a package file, the subset of the records extracted from the log table;
validate, by the system, the package file against a database schema of each of a plurality of additional databases to which the package file is to be deployed;
responsive to validating the package file, simultaneously deploy, by the system, the package file to the plurality of additional databases for making the modifications recorded in the package file to each of the plurality of additional databases.
5. The computer program product of claim 1 , wherein the system validates the package file by:
unpacking the package file,
obtaining the transaction sequence numbers from the unpacked package file, and
validating the transaction sequence numbers against the database schema of the additional database.
6. The computer program product of claim 1 , wherein responsive to validating the package file, the system further builds a table list based on the package file and, based on the table list, performs backups of one or more additional tables in the additional database that will be modified via deployment of the package file to preserve an ability to roll the additional database back in case of a failure during the deployment of the package file.
7. The computer program product of claim 1 , wherein the database is associated with a development environment, and wherein the plurality of additional databases are associated with a production system.
8. The computer program product of claim 1 , wherein the one or more triggers are further installed by the system in connection with at least one additional application configured for modifying the one or more input tables of the database, wherein additional modifications made to the one or more input tables of the database by the at least one additional application are tracked to insert into the log table a record of the additional modifications.
9. The computer program product of claim 1 , wherein the log table tracks the modifications sequentially with a correct order.
10. The computer program product of claim 1 , wherein the one or more triggers are activated right before activating code in the application that is responsible for making the one or more modifications to the one or more input tables of the database, and wherein the one or more triggers are deactivated once the one or more modifications to the one or more input tables of the database are completed.
11. The computer program product of claim 1 , wherein the one or more input tables of the database are identified from an input tables list.
12. The computer program product of claim 1 , wherein the transaction sequence number stored in each record of the log table is used to determine a latest operation on each record of the database which in turn is the basis used for extracting the subset of the records in the log table that reflect the latest data in the database.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/220,286 US10922301B1 (en) | 2016-07-26 | 2016-07-26 | Apparatus, computer program, and method for trigger-based tracking of database modifications |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/220,286 US10922301B1 (en) | 2016-07-26 | 2016-07-26 | Apparatus, computer program, and method for trigger-based tracking of database modifications |
Publications (1)
Publication Number | Publication Date |
---|---|
US10922301B1 true US10922301B1 (en) | 2021-02-16 |
Family
ID=74569988
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/220,286 Active 2037-07-16 US10922301B1 (en) | 2016-07-26 | 2016-07-26 | Apparatus, computer program, and method for trigger-based tracking of database modifications |
Country Status (1)
Country | Link |
---|---|
US (1) | US10922301B1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230185789A1 (en) * | 2021-12-09 | 2023-06-15 | International Business Machines Corporation | Gaining insights on database system table changes |
Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6999977B1 (en) * | 2002-05-09 | 2006-02-14 | Oracle International Corp | Method and apparatus for change data capture in a database system |
US7058615B2 (en) | 2003-04-24 | 2006-06-06 | International Business Machines Corporation | Scheduling for data warehouse ETL processing and data mining execution |
US20060265706A1 (en) * | 2005-05-19 | 2006-11-23 | Isaacson Scott A | System for creating a customized software installation on demand |
US20070067357A1 (en) * | 2005-09-20 | 2007-03-22 | Nicholas Clark | Methods and apparatus to provide a database version control system |
US20080301168A1 (en) * | 2007-05-29 | 2008-12-04 | International Business Machines Corporation | Generating database schemas for relational and markup language data from a conceptual model |
US20090013162A1 (en) * | 2007-07-02 | 2009-01-08 | Microsoft Corporation | Managing a deployment of a computing architecture |
US20100088286A1 (en) * | 2008-10-06 | 2010-04-08 | Teradata Us, Inc. | Deployment manager for master data management |
US7849049B2 (en) | 2005-07-05 | 2010-12-07 | Clarabridge, Inc. | Schema and ETL tools for structured and unstructured data |
US20110010344A1 (en) * | 2009-07-09 | 2011-01-13 | Norsync Technology As | Methods, systems and devices for performing incremental updates of partial databases |
US20130086576A1 (en) * | 2011-09-29 | 2013-04-04 | Oracle International Corporation | System and method for supporting automatically deploying/undeploying application components in a transactional middleware machine environment |
US20150154233A1 (en) * | 2013-12-02 | 2015-06-04 | Qbase, LLC | Dependency manager for databases |
US9824128B1 (en) * | 2012-08-01 | 2017-11-21 | The United States Of America As Represented By The Administrator Of Nasa | System for performing single query searches of heterogeneous and dispersed databases |
-
2016
- 2016-07-26 US US15/220,286 patent/US10922301B1/en active Active
Patent Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6999977B1 (en) * | 2002-05-09 | 2006-02-14 | Oracle International Corp | Method and apparatus for change data capture in a database system |
US7058615B2 (en) | 2003-04-24 | 2006-06-06 | International Business Machines Corporation | Scheduling for data warehouse ETL processing and data mining execution |
US20060265706A1 (en) * | 2005-05-19 | 2006-11-23 | Isaacson Scott A | System for creating a customized software installation on demand |
US7849049B2 (en) | 2005-07-05 | 2010-12-07 | Clarabridge, Inc. | Schema and ETL tools for structured and unstructured data |
US20070067357A1 (en) * | 2005-09-20 | 2007-03-22 | Nicholas Clark | Methods and apparatus to provide a database version control system |
US20080301168A1 (en) * | 2007-05-29 | 2008-12-04 | International Business Machines Corporation | Generating database schemas for relational and markup language data from a conceptual model |
US20090013162A1 (en) * | 2007-07-02 | 2009-01-08 | Microsoft Corporation | Managing a deployment of a computing architecture |
US20100088286A1 (en) * | 2008-10-06 | 2010-04-08 | Teradata Us, Inc. | Deployment manager for master data management |
US20110010344A1 (en) * | 2009-07-09 | 2011-01-13 | Norsync Technology As | Methods, systems and devices for performing incremental updates of partial databases |
US20130086576A1 (en) * | 2011-09-29 | 2013-04-04 | Oracle International Corporation | System and method for supporting automatically deploying/undeploying application components in a transactional middleware machine environment |
US9824128B1 (en) * | 2012-08-01 | 2017-11-21 | The United States Of America As Represented By The Administrator Of Nasa | System for performing single query searches of heterogeneous and dispersed databases |
US20150154233A1 (en) * | 2013-12-02 | 2015-06-04 | Qbase, LLC | Dependency manager for databases |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230185789A1 (en) * | 2021-12-09 | 2023-06-15 | International Business Machines Corporation | Gaining insights on database system table changes |
US11853282B2 (en) * | 2021-12-09 | 2023-12-26 | International Business Machines Corporation | Gaining insights on database system table changes |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8078588B2 (en) | Recoverable execution | |
CN106933703B (en) | Database data backup method and device and electronic equipment | |
CN105989059B (en) | Data record checking method and device | |
CN104937556A (en) | Recovering pages of database | |
CN112487083B (en) | Data verification method and device | |
CN106991104B (en) | Database script deployment device and database script deployment method | |
US10817382B2 (en) | Concurrent upgrade of primary and standby databases | |
CN109840194B (en) | Method and system for detecting configuration file | |
US10459804B2 (en) | Database rollback using WAL | |
CN107479881B (en) | Method for synchronizing difference codes, storage medium, electronic device and system | |
US10922307B2 (en) | Automated transaction engine | |
CN111367994A (en) | Method and system for synchronously backing up incremental data of database | |
CN110244951B (en) | Application publishing method and device | |
EP3131041A1 (en) | Testing of enterprise resource planning systems | |
US8738569B1 (en) | Systematic verification of database metadata upgrade | |
CN111078719A (en) | Data recovery method and device, storage medium and processor | |
CN106990974B (en) | APP updating method and device and electronic equipment | |
US10922301B1 (en) | Apparatus, computer program, and method for trigger-based tracking of database modifications | |
US11144346B2 (en) | Systems and methods for batch job execution in clustered environments using execution timestamp granularity to execute or refrain from executing subsequent jobs | |
CN109597706B (en) | Detection method, device and system for checking difference data | |
CN114490570A (en) | Production data synchronization method and device, data synchronization system and server | |
CN112699129A (en) | Data processing system, method and device | |
CN110147355B (en) | Data synchronization method, device and server | |
CN110297881A (en) | For realizing the method and computer-readable medium of secondary index | |
CN111079199B (en) | Enterprise credit data screenshot tamper-proofing method based on block chain technology |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |