JP2022543992A - Methods, systems and programs for managing operational data in distributed processing systems - Google Patents

Abstract

A method of managing operational data in a distributed processing system comprising monitoring system workload to establish a current assessment of operational data movement between data sources and data targets; receive historical information about previous data movements, including previous instances of moves that have resulted in compromised service quality standards of the company, and current assessment and historical information that the next operational data action will not meet specified service quality standards and responsively applying a data management optimization infrastructure (data backplane services) adapted to improve specific service quality standards as defined by data sources and data targets manages operational data in distributed processing systems. Predict operational outcomes using a cognitive system trained using historical information containing past operational factors that correlate with past operational outcomes in relation to service quality criteria.

Description

The present invention relates generally to computer systems, and more particularly to methods for maintaining quality of service standards in distributed computing systems.

Computing systems have become significantly more complex over the years. In early computing, there was a single computer that handled all the tasks associated with a project. With the advent of more ancillary systems and the advent of networked computing (particularly the Internet), much of the computing world is transforming into distributed computing. A distributed computing or distributed system is a system that includes components implemented in different locations, such as different network computers. Examples include peer-to-peer networks, online gaming, telephony, and data management.

Data management is generally the art of procuring, maintaining, and using data (ie, information). The data itself can be as simple as customer details such as names and addresses, or larger as financial services (e.g. financial crime investigation solutions) . Data management in such systems is very complex. This challenge is especially true in systems that use distributed functional processing architectures such as microservices. Microservices is a software development technique that allows structuring applications as a collection of loosely coupled services. One advantage of decomposing an application into smaller, distinct services is that it improves modularity, makes the application easier to understand, develop, test, and more resilient to architectural collapse. Microservices are discrete processes that communicate over a network to achieve arbitrary goals, especially using technology-neutral protocols such as the hypertext transfer protocol (HTTP). can be considered to exist.

The nature of the particular microservices used is highly application dependent. In a financial services fraud detection application, for example, a microservice could be a receiving service that queues a transaction, checks if an attachment exists, and if so, passes that attachment to an optical character recognition service, etc. An attachment processor service that sends to another microservice in the Create Context service that analyzes the current transaction and associates it with any past transactions related to it, set by the client to identify violations A decision execution engine that executes rules, an analysis engine that reviews transactions and flags outliers, and a case manager that determines whether to create a case for human follow-up based on identified issues. It may include a service and notification manager that returns updates to the client's expense/procurement system as each transaction is processed.

As with all computing systems, the ability to monitor distributed computing systems and ensure that they meet quality-of-service (QoS) requirements. is important. QoS is a measure of the overall performance of a service, such as a telephony or computer network, or a cloud computing service, particularly the performance seen by the users of the network. In order to measure quality of service quantitatively, a number of relevant characteristics of network services are often considered. As with microservices, the specific nature of QoS requirements depends on the specific applications involved. QoS criteria may be defined, for example, in service level agreements that identify response time requirements based on data type and contextual requirements including time and data requirements.

The present invention, in at least one embodiment, receives definitions of data sources and data targets and quality of service metrics for the data sources and data targets, and monitors the operational workload of a distributed processing system. , establishing a current assessment of operational data movement between data sources and data targets, and previous instances of operational data movement that have resulted in compromised one or more of the quality of service criteria; , receiving historical information about previous operational data movements within the distributed processing system, and determining from the current evaluation and historical information that an upcoming operational data action will not meet a particular one of the quality of service criteria; and, in response to said determination, automatically applying a data management optimization infrastructure adapted to improve specified quality of service standards as defined; are generally directed to methods of managing operational data in distributed processing systems. In an exemplary implementation, the service level agreement model provides quality of service criteria, the service level agreement model includes data type, context requirements, time/date requirements, and response time requirements, and data movement requirements. The data movement execution history model provides historical information, the data movement execution history model includes historical service statistics, service resource consumption, and service type execution forecasts, and the current system load model (current system load model) provides the aforementioned monitoring of the operational workload, the current system load model includes resource utilization, current cluster size, and capacity assessment; The data type and QoS requirements model provides definitions and characteristics for types, includes data type definitions and data quality of service definitions, and the process control mechanism implements process control functions as needed to meet quality of service criteria. Controls the data management optimization infrastructure to spawn worker threads at locations in the network remote from the mechanism, and the process control mechanism controls data-optimized selection, data services, service feedback, and data Includes move dispatch. Response time requirements are based on data type, context requirements, and time/data requirements. Determining can use a cognitive system trained using historical information, the cognitive system predicts operational outcomes based on current evaluations, and operational outcomes where certain service quality criteria are not met. provide instructions for. Current ratings may include resource utilization of resources of the distributed processing system, resource capabilities, and resource response times. The data management optimization infrastructure includes multiple extensible data backplane services. In an exemplary application, the distributed processing system provides a fraud detection solution, where the data sources and data targets are large amounts of customer information such as names, addresses, phone numbers, social security numbers, or tax registration numbers; have data types, including transaction information such as transaction volume and type (wire, ACH, credit, debt), and case management data for tracking the aforementioned collections of customer or financial information, and quality of service criteria; contains resource allocations, data integrity specifications, and service uptime, and the data backplane service contains messaging interfaces, APIs, streams, or other communication means for data communication.

In addition to the above, additional objects, features, and advantages of various embodiments of the present invention will become apparent in the following detailed written description.

The present invention may be better understood, and numerous objects, features, and advantages of its various embodiments made apparent to those skilled in the art by referencing the accompanying drawings.

1 is a block diagram of a computer system programmed to perform automated operational data management dictated by quality of service criteria, according to one implementation of the present invention; FIG. 1 is a pictorial representation of a cloud computing environment, according to one implementation of the present invention; 1 is a block diagram illustrating functional modules of an operational data management system, according to one implementation of the present invention; FIG. 1 illustrates one solution for automated operational data management according to one implementation of the present invention using a service level agreement model, a data movement execution history model, a current system load model, and a data type and QoS requirements model; It is a model diagram. 4 is a block diagram of a cognitive system used to predict operational outcomes of the operational data management system of FIG. 3, according to one implementation of the invention; FIG. FIG. 4 is a chart showing the logic flow of a data management process, according to one implementation of the invention; FIG.

The use of the same reference symbols in different drawings indicates similar or identical items.

While the use of distributed computing and microservices offers multiple advantages, the method also presents new challenges to system designers. Historically, monolithic applications could be executed as a single large functional unit to access data and optimized to minimize data movement and duplication. Data can easily reside in a single common data store, can be distributed and accessed by integrated mechanisms, and exist in multiple types of data structures (databases, file-based, etc.) could have done, but still enabled the data access API layer. In each of those cases, there has been a consistent effort to not rely on data movement and redundancy, which introduces opportunities for data integrity, inconsistencies, and problems with distribution.

All of these assumptions change with distributed computing systems. The difficulty is that in many cases the data must be replicated to take advantage of segmented services. For example, in financial services applications, it is desirable to leverage multiple technology elements to support financial crime investigation solutions. Leveraging multiple technology elements means, for example, using elemental technologies to associate bank transaction data as belonging to the same individual, understanding that individual's peer network, or It may involve performing machine learning analysis to identify potentially fraudulent behavioral patterns. Leveraging existing services for these functions is important, but designers often find that those services require access to similar data records (customer information, transaction records, etc.) Faced with the problem of expecting to retrieve data records in a specific format, using a specific data store access interface to a data service that resides in a specific schema or is expected. ing. In such cases, the application writer has no control over how the data is expected to be used and how to minimize data movement and duplication. This problem has become an issue that should be managed by the solution provider as a whole, and now needs to be managed on a project-by-project basis as the components may change but the problem remains.

Therefore, it is desirable to devise improved methods of managing data in such distributed systems. It would be further advantageous if the method could be automated to meet the quality of service (QoS) requirements specific to the particular system involved. Various embodiments of the present invention provide solutions for meeting data movement, replication, and distribution needs using managed QoS methods that define how the system manages data movement throughout the system. These and other benefits are realized by providing solutions. This system allows the application developer to define the sources and targets of specific data elements, how fast the data needs to move (distribution), what the consistency goals are (e.g. guaranteed consistency). , eventual consistency, etc.), how data is expected to be deleted, and where duplicate data is created. The system can use a scalable infrastructure for data movement (including updates and deletions), and flexible scaling of this infrastructure allows QoS goals to be met.

Referring now to the figures, and more particularly to FIG. 1, there is shown one embodiment 10 of a computer system in which automated operational data management can be implemented in accordance with the present invention. Computer system 10 is a symmetric multiprocessor (SMP) system including a plurality of processors 12a, 12b connected to system bus 14; System bus 14 is further connected to and communicates with a coupled memory controller/host bridge (MC/HB) 16 that provides an interface to system memory 18 . System memory 18 may be a local memory device or, alternatively, may include multiple distributed memory devices, including dynamic random-access memory (DRAM). preferable. There may be additional structures within the memory hierarchy not shown, such as on-board (L1) and second level (L2) or third level (L3) caches. The system memory 18, in accordance with the present invention, identifies distributed systems, data definitions, QoS criteria, system monitors, data optimization, various backplane services, and cognitive systems used to predict operational outcomes. It loads one or more applications or software modules, including the operating programs necessary to perform the functions of, all of which are described in further detail below. Although FIG. 1 shows these various components in a single memory 18, some of these components may be similar to computer system 10, or may be different from computer system 10. It is understood that it may reside in a networked computer system (located remotely). In particular, backplane services can be implemented at multiple network locations far from data optimization.

The MC/HB 16 also includes an interface with PCI (peripheral component interconnect) Express links 20a, 20b, 20c. Each PCI Express link 20a, 20b is connected to a respective PCIe adapter 22a, 22b, and each PCIe adapter 22a, 22b is connected to a respective input/output (I/O) device 24a, 24b. MC/HB 16 may further include an interface with I/O bus 26 connected to switch (I/O fabric) 28 . Switch 28 provides I/O bus fanouts to multiple PCI links 20d, 20e, 20f. These PCI links are further connected to PCIe adapters 22c, 22d, 22e, which in turn support further I/O devices 24c, 24d, 24e. I/O devices include keyboards, graphical pointing devices (mouse), microphones, display devices, speakers, persistent storage devices (hard disk drives) or arrays of such storage devices, CDs or DVDs. (one example of a computer-readable storage medium), and network cards. Each PCIe adapter provides an interface between a PCI link and each I/O device. MC/HB 16 provides a low latency path through which processors 12a, 12b can access PCI devices mapped anywhere in bus memory or I/O address space. can be done. MC/HB 16 also provides a high bandwidth path for PCI devices to access memory 18 . Switch 28 may provide peer-to-peer communication between different endpoints, and this data traffic must be forwarded to MC/HB 16 if it does not involve cache-coherent memory transfers. do not have. Switch 28 is shown as a separate logical component, but can be integrated into MC/HB 16 .

In this embodiment, PCI link 20c connects MC/HB 16 to service processor interface 30 and allows communication between I/O device 24a and service processor 32. FIG. Service processor 32 is connected to processors 12a, 12b via JTAG interface 34 and uses attention lines 36 to interrupt the operation of processors 12a, 12b. Service processor 32 may include its own local memory 38 and is connected to read-only memory (ROM) 40 that stores various program instructions for booting the system. Service processor 32 may be able to access hardware operator panel 42 to provide system status and diagnostic information.

In alternate embodiments, computer system 10 may include modifications of these hardware components or their interconnections, or additional components, so that the example shown may be any architecture with respect to the present invention. should not be construed to imply a limitation of The present invention is further implemented within equivalent cloud computing networks.

Service processor 32 uses JTAG interface 34 to interrogate system (host) processors 12a, 12b and MC/HB 16 when computer system 10 is first powered on. After completing the query, service processor 32 obtains the inventory and topology of computer system 10 . Service processor 32 then subjects the components of computer system 10 to tests such as built-in-self-tests (BIST), basic verification tests (BAT), and memory tests. , run various tests. Error information regarding faults detected during testing is reported by service processor 32 to operator panel 42 . After retrieving a component that was detected to be faulty during testing, computer system 10 is allowed to continue if a valid configuration of system resources is still possible. Executable code is loaded into memory 18 and service processor 32 processes program code (e.g., an operating system (OS) used to start applications and, in particular, automated operation of the present invention). host processor 12a, 12b for execution of a data management application), the results of which are stored on the system's hard disk drive (I/O device 24). While host processors 12a, 12b are executing program code, service processor 32 monitors cooling fan speed and operation, thermal sensors, power control units, and processors 12a, 12b, memory 18, and MC/ A mode may be entered to monitor and report any operating parameter or error, including recoverable and non-recoverable errors reported by any of the HBs 16 . Service processor 32 may further take action based on the type of error or defined thresholds.

The invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include a computer readable storage medium containing computer readable program instructions for causing a processor to carry out aspects of the present invention.

A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. not. A non-exhaustive list of more specific examples of computer readable storage media include portable floppy disks, hard disks, random access memory (RAM), read only memory (ROM). -onlymemory), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD - ROM (compact disc read-only memory), digital versatile disk (DVD), memory stick, floppy (R) disk, punch card or raised structure in a groove where instructions are recorded and any suitable combination thereof. As used herein, a computer-readable storage medium is itself a radio wave or other freely propagating electromagnetic wave, or an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., passing through a fiber optic cable). It should not be interpreted as a transient signal such as a light pulse) or an electrical signal transmitted over a wire.

Computer readable program instructions described herein can be transferred from a computer readable storage medium to each computing device/processing device or over a network (e.g., the Internet, local area network, wide area network, or wireless network, or the like). combination) to an external computer or external storage device. The network may comprise copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing device/processing device receives computer readable program instructions from the network and translates those computer readable program instructions into a computer readable network within each computing device/processing device. Transfer for storage on a storage medium.

Computer readable program instructions for performing the operations of the present invention include assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or written in any combination of one or more programming languages, including object-oriented programming languages such as Java(R), Smalltalk, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages; It may be source code or object code. The computer-readable program instructions may execute wholly on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on the remote computer, respectively. It can run on a remote computer or run entirely on a server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN); Or a connection may be made to an external computer (eg, over the Internet using an Internet service provider). In some embodiments, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), for example, are used to implement aspects of the present invention. ) may execute computer readable program instructions to customize the electronic circuit by utilizing the state information of the computer readable program instructions.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions are the means by which instructions executed via a processor of a computer or other programmable data processing apparatus perform the functions/acts specified in the flowchart illustrations and/or block diagrams. For production, it may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. These computer readable program instructions are such that the computer readable storage medium on which the instructions are stored comprises instructions for implementing aspects of the functions/operations specified in the flowchart and/or block diagram blocks. It may be stored on a computer readable storage medium and capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

Computer readable program instructions are instructions that are executed on a computer or other programmable apparatus or other device to perform the functions/acts specified in the flowcharts and/or block diagrams. , computer, or other programmable data processing apparatus, or other device, thereby executing a series of operable steps on the computer, other programmable apparatus, or computer-implemented process. Make it run on other devices you generate.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing the specified logical function. . In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession will in fact be executed substantially concurrently, or possibly in the reverse order, depending on the functionality involved. Each block in the block diagrams and/or flowchart illustrations, and combinations of blocks included in the block diagrams and/or flowchart illustrations, perform the specified function or operation, or implement a combination of dedicated hardware and computer instructions. Note also that it may be implemented by a dedicated hardware-based system for execution.

Computer system 10 executes program instructions for an automated operational data management process that manages data in distributed systems using novel optimization techniques. Thus, programs embodying the invention may further include conventional features of various data management tools, the details of which will be apparent to those skilled in the art upon reviewing this disclosure. Some of those tools may be related to cloud computing. Although this disclosure includes detailed discussion regarding cloud computing, it should be understood that the implementation of the material presented herein is not limited to cloud computing environments. Embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

Cloud computing provides convenient, on-demand network access to shared pools of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services). It is a service delivery model to enable rapid provisioning and release of those resources with minimal administrative effort or interaction with service providers. A cloud model can include different characteristics, service models, and deployment models.

Characteristics can include, but are not limited to, on-demand services, broad network access, resource pools, rapid adaptability, and metered services. On-demand self-service is a cloud-based service that automatically provisions server time and computing power, such as network storage, unilaterally, as needed, without the need for human interaction with the service provider. Refers to the ability of the user. Broad network access refers to the ability to be accessed using standard mechanisms that are available over a network and can be accessed by heterogeneous thin or thick client platforms (e.g. mobile phones, laptops, and personal digital assistants). Resource pooling occurs when a provider's computational resources are pooled and served to multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated according to demand. and reassigned. There is a sense of location independence, where consumers typically have no control or knowledge as to the exact location of the resources provided, but at a higher level of abstraction, location (e.g., country, state, or data center) ) may be specified. Rapid adaptability means that cloud capacity can be provisioned quickly and flexibly, sometimes automatically, scaled out quickly, released quickly and scaled in quickly. The capacity available for provisioning often appears to the consumer as unlimited purchases of any amount at any time. Metered services automatically control and control resource usage at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) by leveraging metering capabilities. It is the ability of the cloud system to optimize. Resource usage can be monitored, controlled and reported, providing transparency to both providers and consumers of the services utilized.

Service models can include, but are not limited to, SaaS (Software as a Service), PaaS (Platform as a Service), and IaaS (Infrastructure as a Service). Software as a Service (SaaS) refers to the ability provided to consumers to consume a provider's applications running on a cloud infrastructure. These applications can be accessed from a variety of client devices through thin client interfaces such as web browsers. Customers may also manage the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application functions, except for the possibility of limited user-specific application configuration settings. can't even control. PaaS (Platform as a Service) is the ability provided to the customer to deploy applications created or acquired by the customer, written using programming languages and tools supported by the provider, onto a cloud infrastructure. refers to Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but configure deployed applications and, in some cases, application hosting environments can be controlled. Infrastructure as a Service (IaaS) refers to the ability provided to consumers to provision processing, storage, networking, and other basic computing resources, including operating systems and applications can deploy and run any software that can Consumers do not manage or control the underlying cloud infrastructure, but they do have control over operating systems, storage, deployed applications, and in some cases, selected network components. (e.g. host firewall) can be controlled in a limited manner.

Deployment models can include, but are not limited to, private cloud, community cloud, public cloud, and hybrid cloud. A private cloud is a cloud infrastructure that is operated solely for an organization. A private cloud can be managed by this organization or a third party and can exist on-premises or off-premises. A community cloud includes cloud infrastructure that is shared by multiple organizations and supports a particular community that has shared concerns (eg, missions, security requirements, policies, and compliance considerations). Community clouds can be managed by these organizations or a third party and can exist on-premises or off-premises. In public clouds, the cloud infrastructure is made available to the general public or large industry associations and is owned by an organization that sells cloud services. The cloud infrastructure of a hybrid cloud remains a unique entity with standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds). A composite of two or more clouds (private, community, or public) that remain coupled together.

A cloud computing environment can be a service-oriented environment with an emphasis on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes. An exemplary cloud computing environment 50 is shown in FIG. As shown, cloud computing environment 50 includes local computing devices (e.g., personal digital assistants (PDAs) or mobile phones 54a, desktops, etc.) used by cloud subscribers. • including one or more cloud computing nodes 52 in network 56 with which computers 54b, laptop computers 54c, or automotive computer systems 54d, or combinations thereof, can communicate; Nodes 52 may communicate with each other. Nodes 52 may be physically or virtually grouped within one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds as described above, or combinations thereof. (not shown). This allows cloud computing environment 50 to provide an infrastructure, platform, and/or SaaS that does not require cloud customers to maintain resources on local computing devices. The types of computing devices 754a-d shown in FIG. 2 are intended to be exemplary only, and computing nodes 52 and cloud computing environment 50 may be any type of network or network-addressable. It is understood that any type of computerized device can be communicated via a connection (eg, using a web browser) or both.

Referring now to Figure 3, one embodiment of an operational data management system 60 constructed in accordance with the present invention is shown. Operational data management system 60 typically includes operational functions or modules 62 , system monitoring functions or modules 64 , and data movement optimization functions or modules 66 . The operations module 62 includes data engineering (information engineering) 68, service level agreements 70, and a set of data type/object properties and addresses 72. FIG. Data Engineering 68 comprises the primary functionality of the Operations Module 62 and includes architectural methods for planning, analyzing, designing and implementing applications. Its specific function will vary according to the specific application. A service level agreement 70 is a contract between a service provider and one or more clients. Certain characteristics of the service such as quality, availability and liability are agreed between the service provider and the client. A service level agreement may include a number of service-performance metrics along with corresponding service level objectives. For the financial services example, service level agreement metrics may include service availability, cost trade-offs, and support response times. The service level agreement 70 as embodied in the operations module 62 reflects the quantitative values stated in the contract. Data type/object properties and addresses are used to describe relevant data characteristics of applications that apply to either data engineering 68 or service level agreements 70 . Properties may include more advanced features (data structures, classes, etc.) in addition to the basic features of the data type. For the financial services example, specific characteristics can include large amounts of information requiring structured and unstructured data, graph data, and big data implementations, and data sources and data • Targets include data types such as names, addresses, phone numbers, social security or tax registration numbers, or other identifying numbers. An address is a network location (physical or virtual) where data is to be stored and retrieved, and can be of various forms depending on the protocol used (HTTP, TCP/IP, etc.) .

System monitor module 64 tracks current and historical operational performance of components of the distributed data system. System monitor module 64 stores current information in a central processing unit (CPU), disk drive or other persistent (non-volatile) memory, volatile memory (i.e., RAM), these or other resources from various hardware tools 76, such as clusters of resources. This information can include any parameters associated with the device, such as performance usage, allocation, power consumption, resource availability, and the like. This information is used to build current system capabilities 78 . System monitor 64 provides historical operational performance information 80 (i.e., device usage and capabilities) correlated to various parameters such as time periods (e.g., peak usage hours), specific clients, or specific services. history).

The data movement optimization module 66 tracks data movement within the distributed data system and other data performance factors such as consistency and distribution, particularly against specifications provided by the system designer. It includes a separate monitor 82 that This information may also be provided to the historical operational performance 80 of the system monitor module 64 . The data movement optimization module can then call the data synchronization service 84 based on current data performance factors, as needed, as further described below. These services may include, for example, file system copying, messaging, database access, transfer protocols, and the like. Therefore, service workers 86 are optimized as needed.

In an exemplary implementation, the present invention uses various models to provide input used to optimize data management. As shown in FIG. 4, process control mechanism 90 receives inputs from service level agreement model 92 , data movement execution history model 94 , current system load model 96 , and data type and QoS requirements model 98 . Any or all of these features may be embodied in computer system 10 . A service level agreement model 92 provides for storing definitions of data sources, targets, and associated QoS criteria for each. This model defines how data is arranged and managed throughout the system. Typically, application developers create only this definition and then at run time the data is automatically moved and managed by the system, so solution developers need to focus on that part of system operation. Instead, they can focus on domain value and have the flexibility to adopt data architecture changes without requiring infrastructure coding investments in that domain. In an exemplary implementation, the service level agreement model 92 includes a list of data types, context requirements, time/data requirements, and response time requirements.

A data movement execution history model 94 reflects past operations of data movement requests, which are learned from past operations and used by the processing control mechanism 90 to predict current operational outcomes. and can be used to make an informed assessment of the amount of resources that need to be applied to work effort in order to meet (compliance with) criteria. As further described below in conjunction with FIG. 5, the data movement performance history model 94 can use machine learning and predictive techniques to ensure that the system operates at optimal thresholds. In an exemplary implementation, the data movement performance history model 94 includes history of service statistics, service resource consumption, and service type performance predictions. Current system load model 96 tracks the existing workload of the system. This information is needed to understand the current capabilities of the system and its ability to meet the QoS criteria for the next data movement action. A heavily loaded system may require more backplane operation threads to be started than a lightly used system with available resources to complete work. Of course, this can change in real time, requiring active monitoring and adaptation. In an exemplary implementation, current system load model 96 includes resource utilization, current cluster size, and capacity rating. The data type and QoS requirements model 98 represents the characteristics for various data types and how QoS definitions can be implemented. For example, targeting real-time consistency along with low-latency distribution requirements differ in implementation between relational databases, distributed file systems, block storage, and the like. In an exemplary implementation, data type and QoS requirements model 98 includes data type definitions and data QoS definitions.

The process control mechanism 90 directs the flexible microservices 100 (data backplane) to invoke worker threads to meet the need to move data. Flexible microservices 100 constitute an extensible infrastructure to actually move (update or delete) data across disparate system components and technologies. These systems are conventional and have a variety of known and predictable behavioral characteristics. A data backplane service is a mechanism for communication of data in a data architecture. An example of a data backplane service is Apache's Kafka. Kafka is an open source stream processing software platform that provides a unified, high-throughput, low-latency communication mechanism for processing real-time data feeds. Data backplane services for fraud detection solutions may include such messaging interfaces, application program interfaces (APIs), streams, or other communication means for data communication. . Elastic scaling is the ability to dynamically grow or shrink your infrastructure on demand, i.e. resources such as physical disk space, memory, and CPU, depending on your application's needs at a given point in time. It refers to the ability to increase or decrease.

In an exemplary implementation, the process control mechanisms 90 include data optimized selection, data services, service feedback, and data movement dispatch. The data-optimized selection receives response time requirements from the service level agreement model 92 and selects the order of services to be processed by the data service. A data service receives data type and QoS definitions from the data type and QoS requirements model 98 and determines which backplane service is suitable for a particular data type. The data service can then sequence data movement dispatches to initiate the necessary data backplane services. The data backplane service provides feedback to the process control mechanism 90 service feedback, and the process control mechanism 90 can also update the history of service statistics in the data movement execution history model 94 .

In a preferred implementation, the predictive functionality of the data management system is embodied in the new cognitive system. Cognitive systems (sometimes called deep learning, deep sorting, or deep question answering) are forms of artificial intelligence that use machine learning and problem solving. Cognitive systems often employ neural networks, but alternative designs exist. Neural networks can be of various types. A feedforward neural network is an artificial neural network in which the connections between units do not form a cycle. Feedforward neural networks were the first and simplest kind of artificial neural network to be devised. In this network, information travels in only one direction (forward) from input nodes through hidden nodes (if any) to output nodes. There are no cycles or loops in this network. This network is therefore different from recurrent neural networks. A recurrent neural network is a type of artificial neural network in which the connections between units form a directed cycle. A recurrent neural network creates an internal state of the network that allows it to exhibit dynamic temporal behavior. Unlike feedforward neural networks, recurrent neural networks can use internal memory to process inputs in any order. Convolutional neural networks are particularly useful for processing image data because they are a particular type of feedforward neural network based on animal vision. Convolutional neural networks are similar to regular neural networks, but are composed of neurons with learnable weights and biases.

There are many alternatives to using neural networks for machine learning, such as support vector machines (SVMs). An SVM essentially constructs a multi-dimensional mathematical space based on training examples, provides boundaries within that space, and performs a binary classification of inputs (e.g., “good” and “bad” answers). to enable. Another method involves Bayesian networks that represent a set of variables in a directed acyclic graph. This network is used to compute probabilistic relationships between variables. A cognitive system is not limited to using a single method, ie it can incorporate any number of these machine learning algorithms.

The latest implementation of artificial intelligence is IBM Watson™ Cognitive Technology, which applies advanced natural language processing, information retrieval, knowledge representation, automated reasoning, and machine learning techniques to the field of open-domain question answering. Apply. Such cognitive systems rely on pre-existing documents (corpora) to extract answers relevant to queries, such as people, locations, organizations, and specific objects, or to identify positive and negative emotions. To do so, these documents can be analyzed in a variety of ways. Various techniques can be used to analyze natural language, identify sources, find and generate hypotheses, find and score evidence, and merge and rank hypotheses . A model for answer scoring and ranking can be trained based on a large set of question (input) and answer (output) pairs. The more algorithms that independently find the same answer, the more likely the answer is correct, resulting in a higher overall score or confidence level.

FIG. 5 illustrates how a new cognitive system 120 can be trained and applied according to one implementation of the invention. The predictive capabilities of cognitive system 120 are based on historical information used as training data 122 . In this example, the cognitive system is used to provide results of ongoing operations of a financial services application that provides fraud detection solutions. Training data 122 thus constitutes examples of previous operational factors in a variety of situations with practical consequences for QoS requirements. For example, training data 122 may include temporal information (such as the time of day, day of the week, date of the month, or other calendar date) that reflects peak usage times or temporary quiescence in activity, snapshots of resource availability, service provision the specific customer (or simply, the number of customers) being served, the transaction load (i.e., the number of transactions recently requested or currently being processed), and the network traffic on the communication lines used by the operational system. can include Each data point in training data 122 may contain this information and other information that correlates with data movement parameters (for all data types) compared to QoS requirements. In other words, the data points provide the input factors that lead to a particular data management state (past operational results) where some QoS requirements are met and others are not. This training allows the cognitive system 120 to learn the likelihood that certain QoS requirements will not be met for certain operational situations. Past operating factors can be updated with service feedback from the data backplane service.

After being so trained, the cognitive system 120 can be used by an operational data management system to predict likely behavior based on current factors. Current system operational factors 124 are provided to cognitive system 120, and these factors include the same types of inputs (temporal, resource, etc.) as training data. With specific machine learning algorithms implemented in the perception system 120, predicted operational results can be forwarded to the data movement optimization 126 of the data management system. In addition to fraud detection examples, cognitive systems provide indications that any or all of the computation, resource, data allocation, or service uptime requirements may be compromised now or in the near future. be able to. Data movement optimization 126 can then prioritize the necessary data backplane services to more effectively handle these deficiencies based on the specific QoS requirements that are compromised. In one implementation, the predicted operational outcome includes different values that indicate in quantitative form the probability of QoS failure (based on the confidence generated by the cognitive system that certain QoS criteria are not met). and data movement optimization 126 may prioritize services associated with requirements indicated as most likely to fail, i.e., call those services first, or even more , can be offered to QoS criteria with a higher probability of failure, or both.

The present invention can be further understood with reference to the chart of FIG. 6, which illustrates the logic flow of data management process 150 according to one implementation. Process 150, which is performed on computer system 10 or any computer system, including distributed systems, begins by receiving source and target data definitions and quality of service criteria for those sources and targets (152). . Definitions can be provided by the application developer according to the specific coding and variables used. The data management system continuously monitors the operational workload (154). The monitored factors can include, for example, resource usage, capacity, and response time. A current valuation is thereby established for ongoing operational data flows (156). Current valuations are used to predict immediate operational outcomes (158). These results can be identified 160 by the cognitive system using historical information about past operational data movements. The predicted results allow data movement optimization to identify which QoS criteria are in jeopardy (162). Some QoS criteria are more at risk and are given priority in resource allocation accordingly. Data movement optimization can then apply the appropriate optimization infrastructure to improve the identified QoS criteria (164). The optimization infrastructure (data backplane services) spawns worker threads as needed to meet the QoS goals to be achieved. As long as operation continues (166), the process iterates back to box 154 to continue monitoring.

The present invention thereby provides workload monitoring and subsequent predictive scaling to meet QoS specifications so that the data backplane can dynamically adapt to the needs of the application at any given time. Together, they provide superior flexibility for data backplanes. Although the invention has been described with reference to particular embodiments, this description is not meant to be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention, will become apparent to persons skilled in the art upon reference to the description of the invention. It is therefore contemplated that such modifications may be made without departing from the spirit or scope of the invention as defined in the appended claims.

Claims (20)

A method of managing operational data in a distributed processing system, comprising:
receiving definitions of data sources and data targets and quality of service criteria for said data sources and said data targets;
monitoring the operational workload of the distributed processing system to establish a current assessment of operational data movement between the data sources and the data targets;
receiving historical information regarding previous operational data movements within the distributed processing system, including previous instances of operational data movements that have resulted in compromising one or more of the quality of service criteria;
determining from the current evaluation and historical information that an upcoming operational data action will not meet a particular one of the quality of service criteria;
and responsive to said determination, automatically applying a data management optimization infrastructure adapted to improve said specified quality of service criteria according to said definition. a service level agreement model provides the quality of service criteria, the service level agreement model includes data type, context requirements, time/date requirements, and response time requirements;
a data movement execution history model provides the history information, the data movement execution history model includes history of service statistics, service resource consumption, and service type execution prediction;
a current system load model provides said monitoring of said operational workload, said current system load model including resource utilization, current cluster size, and capacity assessment;
a data type and QoS requirements model providing said definitions and characteristics for said data type, said data type and QoS requirements model comprising a data type definition and a data quality of service definition;
a process control mechanism controlling said data management optimization infrastructure to spawn worker threads at network locations remote from said process control mechanism as needed to meet said quality of service criteria; 2. The method of claim 1, wherein the process control mechanisms include data optimized selection, data services, service feedback, and data movement dispatch. 3. The method of claim 2, wherein the response time requirement is based on the data type, the context requirement, and the time/data requirement. said determining using a cognitive system trained using said historical information, said cognitive system predicting an operational outcome based on said current evaluation, said operational outcome predicting said specified quality of service; 3. The method of claim 1, providing an indication that the criteria are not met. 2. The method of claim 1, wherein the current ratings include resource utilization of resources of the distributed processing system, capabilities of the resources, and response times of the resources. 2. The method of claim 1, wherein the data management optimization infrastructure includes multiple extensible data backplane services. the distributed processing system provides a fraud detection solution;
for said data sources and data targets to track transaction information including customer name, customer address, customer phone number, customer identification number, transaction volume and transaction type, and said collection of operational data; has a data type that contains case management data for
wherein the quality of service criteria include resource allocations, data integrity specifications, and service uptime;
7. The method of claim 6, wherein the data backplane services include messaging interfaces, application program interfaces, and streams. A computer system,
one or more processors that process program instructions;
a memory device connected to the one or more processors;
receiving definitions of data sources and data targets and quality of service criteria for said data sources and said data targets; monitoring an operational workload of a distributed processing system to monitor said data sources and said data targets; within said distributed processing system, including establishing a current assessment of operational data movement between targets and previous instances of operational data movement that have resulted in compromised one or more of said quality of service criteria; and determining from said current evaluation and historical information that an upcoming operational data action will not meet a particular one of said quality of service criteria. and automatically applying a data management optimization infrastructure adapted to improve said specified quality of service standards in accordance with said definition, to manage operational data within said distributed processing system. and program instructions residing in said memory device. a service level agreement model provides the quality of service criteria, the service level agreement model includes data type, context requirements, time/date requirements, and response time requirements;
a data movement execution history model provides the history information, the data movement execution history model includes history of service statistics, service resource consumption, and service type execution prediction;
a current system load model provides said monitoring of said operational workload, said current system load model including resource utilization, current cluster size, and capacity assessment;
a data type and QoS requirements model providing said definitions and characteristics for said data type, said data type and QoS requirements model comprising a data type definition and a data quality of service definition;
a process control mechanism controlling said data management optimization infrastructure to spawn worker threads at network locations remote from said process control mechanism as needed to meet said quality of service criteria; 9. The computer system of claim 8, wherein the process control mechanisms include data-optimized selection, data services, service feedback, and data movement dispatch. 9. The computer system of claim 8, wherein the response time requirement is based on the data type, the context requirement, and the time/data requirement. said determining using a cognitive system trained using said historical information, said cognitive system predicting an operational outcome based on said current evaluation, said operational outcome predicting said specified quality of service; 9. The computer system of claim 8, providing an indication that the criteria are not met. 9. The computer system of claim 8, wherein said current ratings include resource utilization of resources of said distributed processing system, capabilities of said resources, and response times of said resources. 9. The computer system of claim 8, wherein said data management optimization infrastructure includes multiple extensible data backplane services. the distributed processing system provides a fraud detection solution;
for said data sources and data targets to track transaction information including customer name, customer address, customer phone number, customer identification number, transaction volume and transaction type, and said collection of operational data; has a data type that contains case management data for
wherein the quality of service criteria include resource allocations, data integrity specifications, and service uptime;
9. The computer system of claim 8, wherein said data backplane services include a messaging interface, an application program interface, and streams. A computer program product,
a computer readable storage medium;
receiving definitions of data sources and data targets and quality of service criteria for said data sources and said data targets; monitoring an operational workload of a distributed processing system to monitor said data sources and said data targets; within said distributed processing system, including establishing a current assessment of operational data movement between targets and previous instances of operational data movement that have resulted in compromised one or more of said quality of service criteria; and determining from said current evaluation and historical information that an upcoming operational data action will not meet a particular one of said quality of service criteria. and automatically applying a data management optimization infrastructure adapted to improve said specified quality of service standards in accordance with said definition, to manage operational data within said distributed processing system. program instructions residing on said storage medium. a service level agreement model provides the quality of service criteria, the service level agreement model includes data type, context requirements, time/date requirements, and response time requirements;
a data movement execution history model provides the history information, the data movement execution history model includes history of service statistics, service resource consumption, and service type execution prediction;
a current system load model provides said monitoring of said operational workload, said current system load model including resource utilization, current cluster size, and capacity assessment;
a data type and QoS requirements model providing said definitions and characteristics for said data type, said data type and QoS requirements model comprising a data type definition and a data quality of service definition;
a process control mechanism controlling said data management optimization infrastructure to spawn worker threads at network locations remote from said process control mechanism as needed to meet said quality of service criteria; 16. The computer program product of claim 15, wherein the process control mechanisms include data optimized selection, data services, service feedback, and data movement dispatch. 16. The computer program product of claim 15, wherein the response time requirement is based on the data type, the context requirement, and the time/data requirement. said determining using a cognitive system trained using said historical information, said cognitive system predicting an operational outcome based on said current evaluation, said operational outcome predicting said specified quality of service; 16. The computer program product of claim 15, providing an indication that a criterion has not been met. 16. The computer program product of claim 15, wherein the current ratings include resource usage of resources of the distributed processing system, capabilities of the resources, and response times of the resources. 16. The computer program product of claim 15, wherein the data management optimization infrastructure includes multiple extensible data backplane services.

Images