WO2006121448A1

WO2006121448A1 - A variable architecture distributed data processing and management system

Info

Publication number: WO2006121448A1
Application number: PCT/US2005/020780
Authority: WO
Inventors: William M. Ghetti; Adam M. Ghetti
Original assignee: Wells, Richard B.
Priority date: 2005-05-06
Filing date: 2005-06-13
Publication date: 2006-11-16

Abstract

A variable architecture distributed data processing and management system (10) and methods for a networked computer system (1) having server (20) and network hardware device resources (18) and software management layers (22, 24, 26, 28, 30, 32) for controlling communication and processing of data and instructions between clients (14) and the hardware device resources (20). The software management layers (22, 24, 26, 28, 30, 32) may be distributed in any combination across clients (14), server (20) and network hardware device resources (18), and any portion of the management layers (22, 24, 26, 28, 30, 32) are activated only when necessary to free up system resources. The system (1) also allows for the breaking of processor instructions into the smallest executable portions that are then distributed to all system resources. As the management layers (22, 24, 26, 28, 30, 32) continuously poll the system resources, if any resource processing those distributed instructions falls below a predetermined threshold of performance, an alternative system resource will start processing the same previously distributed smallest executable portion.

Description

A VARIABLE ARCHITECTURE DISTRIBUTED DATA PROCESSING

AND MANAGEMENT SYSTEM

TECHNICAL FIELD The present invention relates to a variable architecture networked computer system and methods that provide distributed processing and management over server and network hardware devices in a networked computer system environment.

BACKGROUND ART

Currently, the computer networking industry uses a multi-tiered networked environment with little or no active distributed processing instructions or distributed management policies in distributed environment tiers. The lack of distributed processing instructions causes several points of failure, among those being network bottleneck at entrance to system, over utilization of servers resulting in server unresponsiveness, poor replication resulting in heterogeneous systems having unreliable failover characteristics, and poor load distribution resulting in the inefficient use of system and hardware resources causing general system unresponsiveness. The current network industry standard employs software architectures and methodologies from the early nineties that have become outdated due to new hardware performance and current system design requirements.

Currently, such networked distributed management systems are designed and developed as an integral part of the networked computer system's initial design, with every desired option or action predicted and engineered as part of the total system development. These designs result in costly and complex systems that are difficult and time-consuming to implement, and particularly complicated, once implemented, to adapt or expand to changing design requirements. Currently designed networked distributed management systems decrease the ability of the system to perform dynamically load balancing. Dynamic load balancing is the act of distributing processing and communications activity evenly across a computer network so no single device is overwhelmed by processing demand. Load balancing has been previously dismissed due to the high CPU resource demand of the complex algorithms necessary to perform dynamic load balancing. But, with the improvement in CPU processor speed and the increase in size of RAM, front side bus memory caching and bus delivery speeds, the industry is able to overcome this previous limitation. However, there is still difficulty in predicting the number of server requests that will be issued to a server since load balancing is especially important for computer networks. If a server becomes overburdened with processing requests, requests are forwarded to another server with more capacity. This is accomplished through the use of a distributed architecture employing algorithms specifically modeled to take advantage of the underutilized processors in the server configuration where only the operating system manages the server.

Additionally, current distributed networked computer systems have an inability to respond without disruption to an unexpected hardware or software failure, i.e. , poor fault tolerance characteristics. There are many levels of fault tolerance, the lowest being the ability to continue operation in the event of a power failure. Many fault tolerant computer systems mirror all operations, i.e. , every operation is performed on two or more duplicate systems, so if one system fails the other can take over.

Furthermore, current networked computer systems fail to have the flexibility to respond adequately to system or processor availability. System availability enables processors to be brought online during a load event in order to supplement and thereby distribute the processing load by making more processing power available only when necessary.

And furthermore, most current networked computer systems struggle in scaling to adapt the system software to handle a mathematically limitless number of connections, restricted only by the actual hardware deployed at the time of the connection demand.

Additionally, current networked computer systems architecture compromises system reliability, or the ability of the system to be always is ready and available to meet the task obligations demanded by a client. And finally, system performance is compromised due to combinations of hardware and software configurations. Typically a system is a sum of its parts, however if the parts are determined to be incompatible, or at best, inefficient in processing and transmission between parts, then the sum tends to be less than the individual parts.

Thus a variable architecture distributed data processing and management system solving the aforementioned problems is desired. DISCLOSURE OF THE INVENTION

This disclosure is directed to a variable architecture distributed data processing and management system for a networked computer system. The management system includes a hardware device layer that has at least one server hardware device resource and at least one network hardware device resource. The hardware device layer receives job process and requests for communication, connection and data from at least one client. The management system further includes the software management layers described in 1-6 below.

1) A governor layer for managing the replication of the client requested job processes for all hardware device resources, redundancy of the client requested job processes on all hardware device resources, and maintenance or termination of all active client requested job processes. The governor layer also manages the connection of clients to the networked computer system, and determines the most efficient client requested job process routing through the networked computer system.

2) A density manager layer for managing the compression and decompression of data received and transmitted through the networked computer system.

3) A security manager layer for managing encryption and decryption of data, the integrity of data transmitted from at least one client, and validation of at least one client in the networked computer system.

4) An application manager layer for managing the breaking of client requested job processes into the smallest executable instructions, and publication of client requested job processes to the hardware device resource processors predetermined to most efficiently process the published client requested job processes. The application manager also manages the identification of published and completed client requested job processes, and the continuous monitoring of each client requested job process running on each system resource, thereby determining the networked computer system service capability and communication of information on hardware device resource capabilities to all hardware device resources in the networked computer system;

5) A data manager layer for managing the integrity of all data processing in and data transmitted through the networked computer system. 6) A resource manager layer for managing the monitoring and reporting the status of all hardware device resources in or connected to the networked computer system. The software management layers described above are computer middleware sets of components, objects, applications and processes that manage all communications and client requested job processes between the at least one client and the hardware device layer.

The present invention further comprises a method of operating a variable architecture distributed data processing and management system for a networked computer 5 system in which a job process request is received from at least one client. The management system includes a hardware device layer that has at least one server hardware device resource and at least one network hardware device resource. The method includes the steps described in 1-6 below.

1) Managing, under control of a software governor layer, replication of the client 0 requested job processes to all hardware device resources, monitoring of redundancy of the client requested job processes on all hardware device resources, maintenance or termination of all active client requested job processes, the connecting of clients to the networked computer system, and the most efficient client requested job process routing through the

I networked computer system.

5 2) Managing, under control of a software density manager layer, compression and decompression of data.

3) Managing, under control of a software security manager layer, the encryption and decryption of data, the integrity of data transmitted from the at least one client, and client

I validation in the networked computer system.

!0 4) Managing, under control of a software application manager layer, the breaking of client requested job processes into the smallest executable instructions, publication of client requested job processes to hardware device resource processors predetermined to most efficiently process the published client requested job processes, the identification of

; published and completed client requested job processes, and continuous monitoring of each

IS client requested job process running on each system resource thereby determining the networked computer system service capability and communication of information on hardware device resource capabilities to all hardware device resources in the networked computer system.

5) Managing, under control of a software data manager layer, the integrity of all data 30 processing and data transmitted through the networked computer system.

6) Managing, under control of a software resource manager layer, the monitoring and reporting the status of all hardware device resources in or connected to the networked computer system.

The software layers are computer middleware sets of components, objects, applications and processes that manage all communications and client requested job processes between at least one client and the hardware device resource layer. The current invention also includes a method of operating a distributed data processing and management system for a networked computer system. The management system includes system resources that have at least one server hardware device resource and at least one network hardware device resource. The method includes receiving a job process request from at least one client and determining the smallest portions of executable instructions that can be broken down from the client requested job process. The client requested job process is then broken down into the smallest portions of executable instructions. System feedback is continuously analyzed from all system resources in the networked computer system capable of processing the smallest portions of executable instructions. The most efficient systems are for processing the smallest portions of the executable instructions and the most efficient path for transmitting the smallest portions of executable instructions to the most efficient system resources are then designated. The designation of the most efficient system resources is based on the continuous analysis of system feedback. The smallest portions of executable instructions are then transmitted on the most efficient path for transmitting to the designated most efficient system resources for processing based on the previous designation.

The current invention also comprises a method of operating a variable architecture distributed data processing and management system for a networked computer system. The management system includes hardware device resources as well as server and network hardware device resources, and software management layers to manage communications and job processes in the networked computer system. The software management layers include a governor layer, a density manager layer, a security manager layer, an application manager layer, a data manager layer, and a resource manager layer. The method of operating the system includes a multi-step process. During the process, a connection request is received from at least one client and processed in the software governor layer. The information is then transmitted to at least one client from the software governor layer to a software client analyzer sub-layer in the software security manager layer. The software client analyzer sub-layer determines if a specific client is valid and authorized to access to the management system. Information is transmitted to the software governor layer based on the determination regarding whether the client is valid and authorized. A request for a new client connection is sent to the management system from the software governor layer to a software proxy monitor sub-layer in the software governor layer. Under control of the software proxy monitor sub-layer communicating with a software session monitor sub-layer in the software governor layer, a determination is made regarding whether the client has an existing session and the last machine that the client was connected to; a server hardware device resource is assigned for the connection. A new connection is then created between the client and the server hardware device resource that is under control of a software proxy manager sub-layer in the software governor layer based on a determination regarding whether the client has an existing session. A software decompression manager sub-layer is activated in the software density manager layer that is able to decompress data sent from the software proxy manager sub-layer in the software governor layer and a determination is made regarding whether the transmitted data sent from the software proxy manager sub-layer needs decompression under control of a software decompression agent sub-layer in the software density manager layer. A decompression algorithm is retrieved from a decompression dictionary in the software density manager layer. The transmitted data is decompressed based on the steps of determining if the data needs decompression and retrieving a decompression algorithm under control of the software decompression agent sub-layer and notifying software decompression manager sub-layer upon completion of decompression. The decompressed data is then transmitted to a software decryption manager sub-layer in the software security manager layer. A software decryption manager sub-layer is activated in the software security manager layer. The software decryption manager is able to decrypt the decompressed data sent from the software decryption manager sub-layer. A determination is made regarding whether the decompressed data sent from the software decompression manager sub-layer needs decryption under control of a software decryption agent sub-layer in the software security manager layer. The decompressed data is decrypted based on the steps of determining if the decompressed data needs decryption and retrieving a decryption algorithm under control of the software decryption agent sub-layer and notifying software decryption manager sub-layer upon completion of decryption. The decrypted data is transmitted from the step of decrypting to a software queue manager sub-layer in the software application manager layer. The decrypted date is queued from the previous step of transmitting in a software message monitor sub-layer in the software queue manager sublayer. A software job process analyzer sub-layer in the software application manager layer under control of the software message monitor sub-layer is notified that the decrypted data is ready for processing as a job process. A determination is made, under control of the software job process analyzer sub-layer, regarding how the job processes should be processed, whether the job process can be divided into smaller independent job processes, and the type of job process that should be used. A determination is also made, under control of the software job process analyzer sub-layer querying a software service manager sub-layer in the software application manager layer, regarding which server hardware device resources in the networked computer system contain necessary services to process the job, and regarding the designation of a specific server hardware device resource having the necessary services to process the job process. The job process is transmitted to a software job process publisher sub-layer in the software application manager layer. The job process is published with an "active" job process status designation from the software job process publisher sub-layer to the designated specific server hardware device resource, and to a software replication manager sub-layer in the software governor layer. The job process is transmitted with a "passive" job process status designation from the software replication manager sub-layer to all remaining server hardware device resources, other than the specific server hardware device resource, based on the step of determining which server hardware device resources contain necessary services to process the job process. The processing of all server hardware device resources having the job are managed through the process described above.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a block diagram illustrating a variable system framework of a preferred embodiment of the present invention.

Figs. 2a and 2b are a block diagrams illustrating the present invention's variable architecture system identifying software layers and software sub-layers of the present invention.

Fig. 3a is a logic flowchart illustrating a distributed processing aspect of a preferred embodiment of the present invention. Fig. 3b is a block diagram illustrating a distributed processing aspect of a preferred embodiment of the present invention.

Fig. 4a is a logic flowchart illustrating an aspect of a specific embodiment of the present invention. Fig. 4b is a block diagram of communication paths of information illustrating an aspect of a specific embodiment of the present invention.

Similar reference characters denote corresponding features consistently throughout the attached drawings.

BEST MODE FOR CARRYING OUT THE INVENTION The present invention provides a networked computer system that outperforms current systems due to a fully distributed framework, fully distributed algorithms and fully distributed rules that achieves a fully distributed methodology and architecture. This allows each element within the networked computer system to achieve the highest efficiency and productivity within system limits. The invention manages processes and the execution of processes across a system or multiple networked systems to increase the load balancing capability, fault tolerance capability, processing availability, scalability, reliability, and processing performance. The invention also manages data and the transmission of data across a system or multiple networked systems to increase the load balancing capability, fault tolerance capability, processing availability, scalability, reliability, and processing performance.

The current invention also provides for a variable architecture distributed data processing and management system for a networked computer system with a hardware device layer having at least one server hardware device resource and at least one network hardware device resource. The hardware device layer receiving job process requests for communication, connection and data from clients and further includes the following software management layers: a governor layer, a density manager layer, a security manager layer, an application manager layer, a data manager layer, and a resource manager layer. The governor layer manages replication of client requested job processes to all hardware device resources, the redundancy of the client requested job processes on all hardware device resources, the maintenance or termination of all active client requested job processes, the connection of clients to the networked computer system, and determines the most efficient client requested job process routing through the networked computer system. The density manager layer manages the compression and decompression of data received and transmitted through the networked computer system.

The security manager layer manages encryption and decryption of data, the integrity of data transmitted from clients, and the validation of clients in the networked computer system.

The application manager layer manages the breaking of client requested job processes into the smallest executable instructions, the publication of client requested job processes to hardware device resource processors predetermined to most efficiently process the published client requested job processes, the identification of published and completed client requested job processes, and the continuous monitoring of each client requested job process running on each system resource thereby determining the networked computer system service capability and communicating this information of hardware device resource capabilities to all hardware device resources in the networked computer system.

The data manager layer manages the integrity of all data processing in and data transmitted through the networked computer system.

And finally, the resource manager layer manages the monitoring and reporting the status of all hardware device resources in or connected to the networked computer system. Each of the above set of software management layers are computer middleware sets of components, objects, applications and processes that manage all communications and client requested job processes between any client and the hardware device layer in the management system of the present invention.

The invention also includes a method of operating a distributed data processing and management system for a networked computer system, wherein the management system includes system resources of server and network hardware device resources. In this embodiment, the management system receives a job process request from a client and determines the smallest portions of executable instructions that can be broken down from the client requested job process. The management system then breaks down the client requested job process into the smallest portions of executable instructions.

The management system continuously analyzes system feedback from all system resources in the networked computer system capable of processing these newly broken down smallest portions of executable instructions. Once the feedback is collected and analyzed, the management system then designates the most efficient system resources to process the smallest portions of executable instructions and the most efficient path for transmitting the smallest portions of executable instructions to those system resources. Then, based on this designation, the management system transmits the smallest portions of executable instructions on the most efficient path to the designated system resources. At the same time, the management system transmits the same smallest portions of executable instructions to all other non-designated system resources capable of processing the instructions and identifies the status of these transmitted executable instructions at the non- designated system resources as "inactive" for processing.

The management system through its continuous monitoring the system resource feedback determines when a previously designated system resource is no longer efficient at processing a specifically transmitted smallest portion of executable instructions. When this situation occurs, the management system designates the next most efficient system resource of the previously non-designated system resources for processing the smallest portion of executable instructions. Once the new system resource is designated, the previously transmitted smallest portion of executable code in it is identified and the status is set to "active" for processing, thereby causing those instructions to be executed.

Another specific embodiment of the present invention is a method of operating a variable architecture distributed data processing and management system for a networked computer system including hardware device resources and software management layers including a governor layer, a density manager layer, a security manager layer, an application manager layer, a data manager layer, and a resource manager layer. In this embodiment, the management system receives a connection request from a client and processes the connection request in the software governor layer. Information is transmitted about the client from the software governor layer to a software client analyzer sub-layer in the software security manager layer. The software client analyzer sub-layer determines if the client is valid and authorized to access to the management system, and thereby transmits information to the software governor layer based on this determination.

The software governor layer requests a new client connection to the management system to a software proxy monitor sub-layer in the software governor layer, and the software proxy monitor sub-layer determines if the client has any existing sessions and the last machine the client was connected to. If the client has an existing session, a new connection is made between the client and a server hardware device resource under control of a software proxy manager sub-layer in the software governor layer.

A software decompression manager sub-layer in the software density manager layer is activated to decompress data sent from the software proxy manager sub-layer in the software governor layer. A software decompression agent sub-layer in the software density manager layer then determines if the transmitted data sent from the software proxy manager sub-layer needs decompression, and retrieves a decompression algorithm from a decompression dictionary in the software density manager layer to decompress the transmitted data.

The decompressed data is then transmitted to a software decryption manager sublayer in the software security manager layer that is activated to determine if the decompressed data sent from the software decompression manager sub-layer needs decryption under control of a software decryption agent sub-layer in the software security manager layer. Decrypting the decompressed data is based on determining if the decompressed data needs decryption and retrieving a decryption algorithm under control of the software decryption agent sub-layer.

The decrypted data is transmitted to a software queue manager sub-layer in the software application manager layer where it is queued in the software queue manager sub- layer before publication for processing. A software job process analyzer sub-layer in the software application manager layer is notified that the decrypted data is ready for processing as a job process.

The job process analyzer sub-layer then determines how the job process should be processed, whether the job process can be divided into smaller independent job processes, and the type of the job process. The software job process analyzer sub-layer queries a software service manager sub-layer in the software application manager layer to determine which server hardware device resources in the networked computer system contain the necessary services to process the job, and then designates a specific server hardware device resource having those necessary services. The job process is transmitted to a software job process publisher sub-layer in the software application manager layer and the job process is publishing with an "active" job process status designation from the software job process publisher sub-layer to the designated specific server hardware device resource.

The job process is additionally transmitted to a software replication manager sublayer in the software governor layer with a "passive" job process status designation and is then transmitted to all remaining server hardware device resources, other than the previously designated hardware device resource.

The management system then manages the processing of the job process on all server hardware device resources containing the job.

Fig. 1 generally illustrates the variable architecture distributed data processing and management system 10 in accordance with one general embodiment of the present invention. A networked computer system 1 includes a management system 10 connected via networked communication 12 to a number of clients 14. The management system 10 includes a hardware layer 16 having network hardware device resources 18 and server hardware device resources 20.

Additionally, the present invention includes a number of software management layers distributed on any number of system hardware device resources in the hardware layer 16 as well as in the networked computer system 1. These software management layers include a governor layer 22, a density manager layer 24, a security manager layer 26, an application manager layer 28, a data manager layer 30, and a resource manager layer 32. The following sections focus on the function of each of the software management layers acting as middleware sets of components, objects, applications and processes that manage all data communications and processes between clients 14 and hardware device resources in the hardware layer 16.

Figs. 2a and 2b illustrate the system framework, which is the management system environment less any physical devices such as hardware and the system hardware layer 16 of Fig. 1.

A. GOVERNOR LAYER

The governor layer 22 generally manages: a) the replication of client requested job processes to all hardware device resources, b) the state of redundancy of client requested job processes on all hardware device resources, c) the maintenance or termination of all active client requested job processes, d) the connection of clients to the networked computer system, and e) the determination of the most efficient client requested job process routing through the networked computer system. A.I. GHOST MANAGER SUB-LAYER

The first sub-layer of the governor layer 22 is the ghost manager sub-layer 100 that contains a replication manager sub-layer 101 for determining how many hardware device resources are in or connected to the networked computer system and the status of each connected hardware device resource. The replication manager 101 replicates and transmits client requested job processes to a plurality of hardware device resources, and a replication log 102 stores records in a memory or a physical location generated under control of the replication manager sub-layer. These records are regarding the identification, status and replication of client requested job processes to hardware device resources in or connected to the networked computer system.

The replication manager sub-layer additionally determines the specifically number, presence and status of server hardware device resources in or connected to the networked computer system and transmits replicated data to processors with a processing flag of "inactive" so as not to be processed immediately when received by the processors. A.2. INTEGRITY MANAGER SUB-LAYER

The second sub-layer of the governor layer 22 is the integrity manager sub-layer 200 that contains an integrity scout dispatch sub-layer 201 for sending scouts, or active software processes, to all hardware device resources in or connected to the networked computer system 1. Also included is an integrity analyzer sub-layer 202 for determining the current state of client requested job process redundancy on each hardware device resource in or connected to the networked computer system based on collected information from the integrity scout dispatch sub-layer 201. Additionally, an integrity log 203 stores records in a memory or a physical location generated under control of the integrity analyzer sub-layer 202 regarding the current state of redundancy of all hardware device resources in or connected to the networked computer system 1.

The integrity scout dispatch sub-layer 201 also continuously determines the number of scouts to be kept active and the number of scouts to be terminated in the networked computer system 1. A.3. GARBAGE MANGER SUB-LAYER The third sub-layer of the governor layer 22 includes a garbage manager sub-layer

300 that includes a garbage scout dispatch sub-layer 301 for sending scouts, or active software processes, to all hardware device resources in or connected to the networked computer system 1 to determine all active and inactive software processes on all of the hardware device resources. A garbage monitor sub-layer 302 then determines whether to maintain or terminate a client requested job process on a hardware device resource in or connected to the networked computer system to free hardware device resources and virtual resources. Additionally, a garbage log 303 stores records in a memory or a physical location generated under control of the garbage monitor sub-layer 302 regarding maintenance or termination of client requested job processes on all of the hardware device resources in or connected to the networked computer system 1.

Additionally, the software garbage scout dispatch sub-layer 302 determines the number of scouts to be kept active and the number of scouts to be terminated on all hardware service device resources in the networked computer system 1. A.4. STATE MANAGER SUB-LAYER

The fourth sub-layer in the governor layer 22 a state manager sub-layer 400 that includes a session regulator sub-layer 401 that collects information about the status of each client in or connected to the networked computer system, and a session monitor sub-layer 402 for receiving information from a plurality of session regulator sub-layers 401 across the management system 1 and determines the current status and number of clients in or connected to the networked computer system 1. The session monitor sub-layer 402 additionally determines the overall capacity of the networked computer system 1 regarding potential connections of new clients, as well as the throughput of data through the networked computer system. From these determinations, a data transmission routing table is created for future data transmission routing paths. Additionally, a session log 403 stores records in a memory or a physical location generated by the information collected and the determinations made of the session monitor sub-layer 402. A.5. PROXY MANAGER SUB-LAYER

The fifth sub-layer of the governor layer 22 is a proxy manager sub-layer 500 that includes a proxy monitor sub-layer 501 that continuously collects information of all currently available communication ports, the port type, communication protocols, communication routings and the current status of each communication port in the networked computer system 1. Additionally included is a proxy regulator sub-layer 502 for determining the most efficient data routing for the next incoming data from a client based on the information collected by the proxy monitor sub-layer 501, and a proxy log 503 for storing records in a memory or a physical location of information generated under control of the proxy regulator sub-layer 502.

B. DENSITY MANAGER LAYER

The management system 10 further includes a density manager layer 24 that manages the compression and decompression of data received and transmitted through the networked computer system 1. B.I. COMPRESSION MANAGER SUB-LAYER

The first sub-layer of the compression manager layer 24 is a compression manager sub-layer 600 that comprises a compression agent sub-layer 601 for determining if compression of a message stream data is necessary or feasible, and if so, determining what compression algorithm is most efficient in compressing the message stream data. A compression dictionary 602 lists different types of compression algorithms for use in compression under control of the compression agent sub-layer 601 , and a compression log 603 stores records in a memory or a physical location of data related to compression, the type of compression algorithm used under control of the compression agent sub-layer 601 and data related to the efficiency obtained in compression. B.2. DECOMPRESSION MANANGER SUB-LAYER

The second sub-layer in the density manager layer 24 is the decompression manager sub-layer 700 that includes a decompression agent sub-layer 701 for determining if decompression of a message stream data is necessary, and if so, determining what decompression algorithm is necessary to decompress the message stream data. A decompression dictionary 702 lists different types of decompression algorithms for use in decompression under control of the decompression agent sub-layer 701, and a decompression log 703 stores records in a memory or a physical location comprising data related to decompression, the type of decompression algorithm used under control of the decompression agent sub-layer 702 and data related to the verification of decompressed data.

C. SECURITY MANAGER LAYER

The management system 10 further includes a security manager layer 26 for managing encryption and decryption of data, the integrity of data transmitted from a client, and the validation of a client in the networked computer system. Cl ENCRYPTION MANAGER SUB-LAYER

The first sub-layer of the security manger layer 26 is the encryption manager sublayer 800 that includes an encryption agent sub-layer 801 that determines if message stream data needs encrypting and, if so, then determines what encryption algorithm is necessary to encrypt the message stream data and an encryption dictionary 802 that lists different types of encryption algorithms for use in encryption under control of the encryption agent sublayer 801. An encryption log 803 stores records in a memory or a physical location of data related to encryption and the type of encryption algorithm used under control of the encryption agent sub-layer. C.2. DECRYPTION MANAGER SUB-LAYER

The second sub-layer of the security manager layer 26 is the decryption manager sub-layer 900 that includes a decryption agent sub-layer 901 for determining if message stream data needs decryption and, if so, then determining what decryption algorithm is necessary to decrypt the message stream data from a decryption dictionary 902 that contains different types of decryption algorithms for use in decryption under control of the decryption agent sub-layer 901. Additionally, a decryption log 903 for storing records in a memory or a physical location comprising data related to decryption and the type of decryption algorithm used under control of the decryption agent sub-layer 901. C.3. ACCESS OFFICER SUB-LAYER The third sub-layer of the security manager layer 26 is the access officer sub-layer

1000 that includes a packet analyzer sub-layer 1001 for determining the format of received data, correcting the format of transmitted data, and verifying the security of data using embedded unique identification security codes. A client analyzer sub-layer 1002 determines the access rights of a client to system resources and a security dictionary 1003 that contains records of acceptable and unacceptable clients, types of data formats, and rules for client access to system resources. A security delegate sub-layer 1004 manages information received from multiple client analyzer sub-layers 1002 in the networked computer system 1 to determine which of the client analyzers 1002 are in use, and what is the most efficient route in transmitting data to specific client analyzer sub-layers 1002 to enable client analysis. Additionally, a security log 1005 stores records in a memory or a physical location of all analyzed client transactions and all functions executed by the access officer sub-layer 1000. D. APPLICATION MANAGER LAYER

The management system 10 further includes an application manager layer 28 for managing the breaking of client requested job processes into the smallest executable instructions, the publication of client requested job processes to hardware device resource processors predetermined to most efficiently process the published client requested job processes, the identification of published and completed client requested job processes, and the continuous monitoring of each client requested job process running on each system resource thereby determining the networked computer system service capability and communication of information on hardware device resource capabilities to all hardware device resources in the networked computer system. D.I. JOB PROCESS ANALYZER SUB-LAYER

The first sub-layer of the application manager layer 28 is the job process analyzer sub-layer 1100 that includes a job process analysis agent sub-layer 1101 that determines the most efficient means to break a client requested job process into the smallest executable instruction and then breaks the client requested job process into the smallest executable instruction. A job process analysis log 1102 stores records in a memory or a physical location of the client requested job process broken by the job process analysis agent sublayer 1101 and the locations of the smallest executable instruction. D.2. JOB BROKER SUB-LAYER The second sub-layer of the application manager layer 28 is the job process broker sub-layer 1200 that includes a job process monitor sub-layer 1201 for monitoring all processor activities across the networked computer system 1 , and a job process publisher sub-layer 1202 that determines which processors in the networked computer system 1 are most efficient to process each client requested job process and then publishing the client requested job processes to those most efficient processors. A job process broker log 1203 stores records in a memory or a physical location of processor activities from the job process monitor sub-layer 1201 and the publication of client requested job processes from the job process publisher sub-layer 1202.

Additionally, the job process broker sub-layer 1200 can further communicate with a graphical user interface for providing graphical feedback of the processor monitoring activities, and a user input device for receiving additional user input to determine which processors in the networked computer system are most efficient to process each client requested job process.

D.3. QUEUE MANAGER SUB-LAYER

The third sub-layer of the application manager layer 28 is the queue manager sublayer 1300 that includes a message manager sub-layer 1301 that determines when published client requested job processes have reached their completion of processing and then identifying those completed client requested job processes as having completed processing. A message log 1302 stores records in a memory or a physical location of the completed and identified client requested job processes from the message manager sub-layer 1301. D.4. SERVICE MANAGER SUB-LAYER The fourth sub-layer of the application manager layer 28 is the service manager sublayer 1400 that includes a service monitor sub-layer 1401 that continuously monitors the status of each client requested job process running on each hardware device resource, and continuously catalogs the networked computer system service capability to perform client requested job processes on each hardware device resource in the networked computer system 1. Additionally, a service broadcaster sub-layer 1402 determines what additional client requested job processes are capable of being processed on the hardware device resources in the networked computer system 1 , and then communicates information of the hardware device resource capabilities to all hardware device resources in the networked computer system. The service broadcaster sub-layer 1401 continuously monitors each hardware device resource in the networked computer system 1 and communicates this monitoring information on all job processes to all hardware device resources in the networked computer system 1. Further included is a service log 1403 for storing records in a memory or a physical location on the status of each running client requested job process, the networked computer system service capability, the additional client requested job processes capable of being processed, the hardware device resources capabilities, and the information on all client requested job processes on each hardware device resource. E. DATA MANAGER LAYER

The management system 10 further includes a data manager layer 30 for managing the integrity of all data processing in and data transmitted through the networked computer system 1. The data manager layer 30 synchronizes all data being processed on all hardware device resources within the networked computer system 1 and prevents interruption of that processing data on those hardware device resources. F. RESOURCE MANAGER LAYER

The management system 10 further includes a resource manager layer 32 for managing the monitoring and reporting the status of all hardware device resources in or connected to the networked computer system 1. The first sub-layer of the resource manager layer 32 is a resource monitor sub-layer 1500 that collects, organizes, creates records and transmits information of all vital statistics of each hardware device resource in the networked computer system 1. A resource profiler sub-layer 1600 stores records in a memory or a physical location on the vital statistics on all of the hardware device resources in the networked computer system 1. Vital statistics may include hard drive usage and capacity, processor type, usage and instantaneous loads, memory usage, and client requested job processes.

VARIABLE DISTRIBUTION OF SOFTWARE LAYERS AND SUB-LAYERS ON NETWORK RESOURCES

Referring back again to Fig.l, the software management layers and sub-layers, as illustrated in Figs. 2a and 2b, may all reside at different or similar resource locations in the networked computer system 1. The software management layers and sub-layers may reside on one server hardware device resource 20 or on more than one server hardware device resources 20 in the management system 1. They may also reside on one network hardware device resource 18 or on more than one network hardware device resources 18 in the management system 1.

Additionally, the software management layers and sub-layers may reside on all server hardware device resources 20 in the management system 1 or on all network hardware device resources 18 in the management system 1. Furthermore, they may reside on all servers 20 and network 18 hardware device resources in the management system 1. These aforementioned embodiments of the distribution of the software management layers and sub-layers create multiple instances of the software management layers and sub-layers within the management system 1 environment when the same software layer or sub-layer is distributed to more than one hardware device resource at the same time. An additionally embodiment allows a software management layer or sub-layer to be activated on a hardware device resource, and another software management layer or sublayer to be deactivated on the same hardware device resource where both software management layers or sub-layers reside. This managed distribution and activation of software layers and sub-layers provides the variable architecture of the software management layers and sub-layers as described below.

VARIABLE ARCHITECTURE OF SOFTWARE MANAGEMENT LAYERS AND SUB-LAYERS The system architecture is a "variable" architecture meaning that not all software management layers and sub-layers are necessary to be activated in order for the certain functions in system. Each software layer and sub-layer may have multiple configurations, including the order of processing and communication of layers and sub-layers, and the throughput of data through the system environment depending on the function called on to perform a specific task.

The management system 10 determines which software management layers and sublayers are necessary to be activated for execution client requested job processes, and then determines the order of instructions for the software management layers and sub-layers in processing client requested job processes. Additionally, the management system 10 determines a communication route within the software management layers and sub-layers for communicating the client requested job process, and for communicating a route of processed data from the software management layers and sub-layers through the management system 10 or through the networked computer system 1. Clients in the networked computer system may take on different identities as observed from the management system 10 processing environment. Clients may be remote machines physically separate from the management system 10 in the networked computer system 1, or a machine in another remote networked computer system.

Additionally, clients may take the form of an application or process residing on the at least one server 20 or network 18 hardware device resource in the management system 10, or an application or process residing on a remote hardware device resource connected to but not present in the management system 10.

Figs. 3a and 3b illustrate a principle of distributed management and processing in a specific embodiment of the present invention that further clarifies the variable distribution of data and variable activation of processing in the networked computer system 1. The management system 10 of the present invention receives 50 a job process request 62 from a client, and then determines 51 the smallest portions of executable instructions 63 that can be broken down from the client requested job process 62, and then breaks down 52 the client requested job process 62 into the smallest portions of executable instructions 64. The management system 10 continuously analyzes 53 system feedback 65 from by polling 66 all system resources 67 in the networked computer system 1 capable of processing the smallest portions of executable instructions 64. The management system then designates 54 the most efficient system resources 69, (herein shown for simplicity of illustration purposes as only a single resource), for processing the smallest portions of executable instructions 64 and the most efficient path (not shown) for transmitting the smallest portions of executable instructions 64 to the most efficient system resources 69.

The designation of the most efficient system resources is based on the step of continuously analyzing the system feedback of resources which can include: the processor type, status and availability; the hardware type, status and availability; the communication port type, status and availability; the network communication throughput; the total number of clients in the networked computer system; and, the type and number of client requested job processes in the management system 10. Next, the management system 10 transmits 55 the smallest portions of executable instructions 70, (herein shown for simplicity of illustration purposes as only a single portion of instructions), on the most efficient path (not shown) for transmitting to the designated most efficient system resources 69 for processing. The management system 10 immediately thereafter transmits 56 the smallest portions of executable instructions 70, (shown in like manner, as above, as only a single portion of instructions), to all other non-designated system resources 71, and identifies 57 the status of the smallest portions of executable instructions 70 at the non-designated system resources as inactive 72 for processing. The management system 10 continuously determines 58 whether the designated system resource 69 is no longer efficient at processing a specific one of the smallest portions of executable instructions 70 based on polling 73 and continuously analyzing the system feedback 74 of all system resources. When a system resource is determined as failing to meet a predetermined level of processing efficiency, that system resource processing is terminated 75, and the management system 10 designates 59 by choosing 76 the next most efficient system resource 77 of the non-designated system resources 71 having the inactive smallest portions of executable instructions 72. Management system 10 then identifies 60 the status of the formerly inactive smallest portions of executable instructions as active 78 for processing, wherein the system resource then begins executing 61 the newly activated smallest portions of executable instructions. Figs. 4a and 4b illustrate a principle of the variable architecture of the software management layers and sub-layers as they function in another specific embodiment of the present invention.

The management system 10 receives a connection request 80 from a client 14 and the software governor layer 22 processes the client connection request. Information is then transmitted about the client 14 from the software governor layer 22 to a software client analyzer sub-layer 1002 in the software security manager layer 26.

The software client analyzer sub-layer 1002 determines 81 if the client 14 is valid and authorized to access to the management system 10, and then transmits authorization information to the software governor layer 22 based on determining if the client is valid and authorized. If the client 14 is authorized, then the software governor layer 22 requests a new client connection to the management system 10 to a software proxy monitor sub-layer 501, (not shown in Figs. 4a and 4b), in the software governor layer 22, and determining, under control of the software proxy monitor sub-layer 501 and communicating with a software session monitor sub-layer 402, (not shown in Figs. 4a and 4b), in the software governor layer 22, if the client 14 has an existing session and the last machine the client 14 was connected to, and assigning 82 a server hardware device resource for a connection. A new connection is created 83 between the client and the server hardware device resource under control of a software proxy manager sub-layer 500 in the software governor layer 22 based on determining if the client 14 has an existing session. The management system 10 then activates a software decompression manager sublayer 700 in the software density manager layer 24 able to decompress data sent from the software proxy manager sub-layer 500 in the software governor layer 22, and determines if the transmitted data sent from the software proxy manager sub-layer 500 needs decompression under control of a software decompression agent sub-layer 701 in the software density manager layer 700. If so, a decompression algorithm is retrieved from a decompression dictionary 702 in the software density manager layer 24, and decompression 84 of the transmitted data based on determining if the data needs decompression which then notifies the software decompression manager sub-layer 700 upon completion of decompression.

The decompressed data is transmitted to a software decryption manager sub-layer

900 in the software security manager layer 26, whereby, a determination is made if the decompressed data sent from the software decompression manager sub-layer 700 needs decryption under control of a software decryption agent sub-layer 701 in the software security manager layer 26. Decryption 85 of the decompressed data based on determining if the decompressed data needs decryption and retrieving a decryption algorithm under control of the software decryption agent sub-layer 901 and notifying software decryption manager sub-layer 900 upon completion of decryption.

The decrypted data is transmitted 86 from the step of decrypting to a software queue manager sub-layer 1300 in the software application manager layer 28, and is queued in a software message monitor sub-layer 1301 in the software queue manager sub-layer.

A software job process analyzer sub-layer 1100 in the software application manager layer 28 is notified 87 that the decrypted data is ready for processing as a job process, and determines 88 how the job processes should be processed, whether the job process can be divided into smaller independent job processes, and the type of job process.

The software job process analyzer sub-layer 1100 queries 89 a software service manager sub-layer 1400 in the software application manager layer 28, as to which server hardware device resources in the networked computer system 1 contain the necessary services to process the job, and then designates a specific server hardware device resource having the necessary services to process the job process.

The job process is then transmitted to a software job process publisher sub-layer

1202 in the software application manager layer 28, and the job process is published 90 with an "active" job process status designation to the designated specific server hardware device resource for processing 91. The job process is also published to a software replication manager sub-layer 101 in the software governor layer 22 with a "passive" job process status designation, and the software replication manager sub-layer 101 to all remaining server hardware device resources. The management system 10 then manages the processing of all server hardware device resources containing the both the "active" and "inactive" status jobs.

Therefore, the present invention, as demonstrated above, results in an improvement dynamic load balancing accomplished through the use of a distributed software architecture employing algorithms specifically modeled to take advantage of the processors in the server configurations and the underlying operating system that manages the server. Operating system protocols are based open the open source Linux operating system but may be modified to incorporate a Unix or Windows Server environments, or any other operating system environment.

The present invention has the ability in a distributed network system to reproduce virtual copies of software, so in the event of a processor or server failure, there is seamless reallocation of resources utilizing the resources in a network environment. A preferred network environment for the present invention utilizes the IBM e-server BladeCenter™ servers, but any network environment can utilize the architecture and management system of the present invention. Furthermore, by manipulating a data stream into multiple cells, wherein each cell is transmitted throughout the system to replicate the original data, the data stream information is distributed to each system resource during the load balancing process enabling the network system to deploy a virtual backup copy of all data at the time of a failure or fault identification.

Additionally, the software management layers and sub-layers can be scaled to meet virtually any requirement for connection handling given the proper hardware and bandwidth connections and availability. Through the use of virtual replication and load balancing algorithms, the management system increases reliability based upon less congestion of connections and improved processing of data through the cell replication process due to distributing the workload within the distributed architecture network.

By combining software algorithms and hardware to enable faster performance of the functions on processors within the system, a larger gateway for connectivity of multiple client requests is achieved.

It is to be understood that the present invention is not limited to the embodiments described above, but encompasses any and all embodiments within the scope of the following claims. It is to be understood that the present invention is not limited to the embodiments described above, but encompasses any and all embodiments within the scope of the following claims.

Claims

CLAIMSWe claim:

1. A variable architecture distributed data processing and management system for a networked computer system, said management system including a hardware device layer having at least one server hardware device resource and at least one network hardware device resource, said hardware device layer receiving job process requests for communication, connection and data from at least one client, said management system further including software management layers comprising:

1) a governor layer for managing replication of said client requested job processes to all hardware device resources, redundancy of said client requested job processes on all hardware device resources, maintenance or termination of all active client requested job processes, connection of clients to said networked computer system, and the most efficient client requested job process routing through the networked computer system;

2) a density manager layer for managing the compression and decompression of data received and transmitted through said networked computer system;

3) a security manager layer for managing encryption and decryption of data, the integrity of data transmitted from said at least one client, and validation of said at least one client in said networked computer system;

4) an application manager layer for managing the breaking of client requested job processes into the smallest executable instructions, publication of client requested job processes to hardware device resource processors predetermined to most efficiently process said published client requested job processes, the identification of published and completed client requested job processes, and continuous monitoring of each client requested job process running on each system resource thereby determining said networked computer system service capability and communication of information on hardware device resource capabilities to all hardware device resources in said networked computer system;

5) a data manager layer for managing the integrity of all data processing in and data transmitted through said networked computer system; and

6) a resource manager layer for managing the monitoring and reporting the status of all hardware device resources in or connected to said networked computer system; wherein said software management layers are computer middleware sets of components, objects, applications and processes that manage all communications and client requested job processes between said at least one client and said hardware device layer.

2. The variable architecture distributed data processing and management system according to claim 1 , wherein said governor layer further comprises a ghost manager sublayer that comprises: a replication manager sub-layer for determining how many hardware device resources are in or connected to said networked computer system and the status of each connected hardware device resource, and replicating client requested job processes to a plurality of hardware device resources; and a replication log for storing records in a memory or a physical location generated under control of said replication manager sub-layer, said records regarding the identification, status and replication of client requested job processes to hardware device resources in or connected to said networked computer system.

3. The variable architecture distributed data processing and management system according to claim 1 , wherein said governor layer further comprises an integrity manager sub-layer that comprises: an integrity scout dispatch sub-layer for sending scouts, or active software processes, to all hardware device resources in or connected to said networked computer system; an integrity analyzer sub-layer for determining the current state of said client requested job process redundancy for each hardware device resource in or connected to said networked computer system based on collected information from said integrity scout dispatch sub-layer; and, an integrity log for storing records in a memory or a physical location generated under control of said integrity analyzer sub-layer regarding the current state of redundancy of all hardware device resources in or connected to said networked computer system.

4. The variable architecture distributed data processing and management system according to claim 1 , wherein said governor layer further comprises a garbage manager sub-layer that comprises: a garbage scout dispatch sub-layer for sending scouts, or active software processes, to all hardware device resources in or connected to said networked computer system to determine all active and inactive software processes on all of said hardware device resources; a garbage monitor sub-layer for determining whether to maintain or terminate a client requested job process on a hardware device resource in or connected to said networked computer system to free hardware device resources and virtual resources; and a garbage log for storing records in a memory or a physical location generated under control of said garbage monitor sub-layer regarding maintenance or termination of client requested job processes on all of said hardware device resources in or connected to said networked computer system.

5. The variable architecture distributed data processing and management system according to claim 1 , wherein said governor layer further comprises a state manager sublayer that comprises: a session regulator sub-layer collecting information about the status of each client in or connected to said networked computer system; a session monitor sub-layer for receiving information from a plurality of session regulator sub-layers across said management system and determining the current status and number of clients in or connected to said networked computer system, for determining the capacity of said networked computer system regarding potential connections of new clients, for determining the throughput of data through said networked computer system, and for creating a data transmission routing table; and a session log for storing records in a memory or a physical location generated under control of said session monitor sub-layer.

6. The variable architecture distributed data processing and management system according to claim 1 , wherein said governor layer further comprises a proxy manager sublayer that comprises: a proxy monitor sub-layer for collecting information regarding all currently available communication ports, and the type, communication protocol, routing and current status of each communication port in said networked computer system; a proxy regulator sub-layer for determining the most efficient data routing for the next incoming data from a client; and a proxy log for storing records in a memory or a physical location generated under control of said proxy regulator sub-layer.

7. The variable architecture distributed data processing and management system according to claim 1 , wherein said density manager layer further comprises a compression manager sub-layer that comprises: a compression agent sub-layer for compressing message stream data and determining what compression algorithm is most efficient to compress said message stream data; a compression dictionary that lists types of compression algorithms for use in compression under control of said compression agent sub-layer; and a compression log for storing records in a memory or a physical location comprising data related to compression, the type of compression algorithm used under control of said compression agent sub-layer and data related to the efficiency of compression obtained.

8. The variable architecture distributed data processing and management system according to claim 1 , wherein said density manager layer further comprises a decompression manager sub-layer that comprises: a decompression agent sub-layer for decompressing message stream data and determining what decompression algorithm is necessary to decompress said message stream data; a decompression dictionary that lists types of decompression algorithms for use in decompression under control of said decompression agent sub-layer; and a decompression log for storing records in a memory or a physical location comprising data related to decompression, the type of decompression algorithm used under control of said decompression agent sub-layer and data related to the verification of decompressed data.

9. The variable architecture distributed data processing and management system according to claim 1 , wherein said security manager layer further comprises an encryption manager sub-layer that comprises: an encryption agent sub-layer for encrypting message stream data and determining what encryption algorithm is necessary to encrypt said message stream data; an encryption dictionary that lists types of encryption algorithms for use in encryption under control of said encryption agent sub-layer; and an encryption log for storing records in a memory or a physical location comprising data related to encryption and the type of encryption algorithm used under control of said encryption agent sub-layer.

10. The variable architecture distributed data processing and management system according to claim 1 , wherein said security manager layer further comprises a decryption manager sub-layer that comprises: a decryption agent sub-layer for decrypting message stream data and determining what decryption algorithm is necessary to decrypt said message stream data; a decryption dictionary that contains types of decryption algorithms for use in decryption under control of said decryption agent sub-layer; and a decryption log for storing records in a memory or a physical location comprising data related to decryption and the type of decryption algorithm used under control of said decryption agent sub-layer.

11. The variable architecture distributed data processing and management system according to claim 1 , wherein said security manager layer further comprises an access officer sub-layer that comprises: a packet analyzer sub-layer for determining the format of received data, for correcting the format of transmitted data, and for verifying the security of data using embedded unique identification security codes; a client analyzer sub-layer for determining the access rights of said at least one client to system resources; a security dictionary that contains records of acceptable and unacceptable clients, types of data formats, and rules for client access to system resources; a security delegate sub-layer for managing at least one of a plurality of client analyzer sub-layers in said networked computer system to determine which of said plurality of client analyzers are in use, and what is the most efficient route in transmitting data to said plurality of client analyzer sub-layers to enable client analysis; and a security log for storing records in a memory or a physical location of all analyzed client transactions and all functions executed by said access officer sub-layer.

12. The variable architecture distributed data processing and management system according to claim 1, wherein said application manager layer further comprises a job process analyzer sub-layer that comprises: a job process analysis agent sub-layer for determining the most efficient means to break a client requested job process into the smallest executable instruction and for breaking a client requested job process into said smallest executable instruction; and a job process analysis log for storing records in a memory or a physical location of said client requested job process broken by said job process analysis agent sub-layer and the locations of said smallest executable instruction.

13. The variable architecture distributed data processing and management system according to claim 1 , wherein said application manager layer further comprises a job process broker sub-layer that comprises: a job process monitor sub-layer for monitoring all processor activities in said networked computer system; a job process publisher sub-layer for determining which processors in said networked computer system are most efficient to process each client requested job process and for publishing client requested job processes to said most efficient determined processors; and a job process broker log for storing records in a memory or a physical location of processor activities from said job process monitor sub-layer and publication of client requested job processes from said job process publisher sub-layer.

14. The variable architecture distributed data processing and management system according to claim 13, wherein said job process broker sub-layer further communicates with: a graphical user interface for providing graphical feedback of said processor monitoring activities; and a user input device for receiving additional user input for determining which processors in said networked computer system are most efficient to process each client requested job process and for publishing client requested job processes to said most efficient determined processors.

15. The variable architecture distributed data processing and management system according to claim 1 , wherein said application manager layer further comprises a queue manager sub-layer that comprises: a message manager sub-layer for determining processor completion of published client requested job processes and for identifying said completed client requested job processes as having completed processing; and a message log for storing records in a memory or a physical location of said completed and identified client requested job processes from said message manager sublayer.

16. The variable architecture distributed data processing and management system according to claim 1 , wherein said application manager layer further comprises a service manager sub-layer that comprises: a service monitor sub-layer for continuously monitoring the status of each client requested job process running on each hardware device resource, and for continuously cataloging the networked computer system service capability to perform client requested job processes on each hardware device resource in said networked computer system; a service broadcaster sub-layer for determining what additional client requested job processes are capable of being processed on said hardware device resources in said networked computer system, communicating information of said hardware device resource capabilities to all hardware device resources in said networked computer system, and continuously monitoring each hardware device resource in said networked computer system and communicating information on all job processes on each hardware device resource in said networked computer system to all hardware device resources in said networked computer system; and a service log for storing records in a memory or a physical location of said status of each running client requested job process, said networked computer system service capability, said additional client requested job processes capable of being processed, said hardware device resources capabilities, and said information on all client requested job processes on each hardware device resource.

17. The variable architecture distributed data processing and management system according to claim 1 , wherein said data manager layer manages synchronization of all data processing on all hardware device resources within said networked computer system and prevents interruption of data being processed on said hardware device resources.

18. The variable architecture distributed data processing and management system according to claim 1, wherein said resource manager layer further comprises: a resource monitor sub-layer for collecting, organizing, creating records and transmitting information of all vital statistics of each hardware device resource in said networked computer system; and a resource profiler sub-layer for storing records in a memory or a physical location of said vital statistics on all of said hardware device resources in said networked computer system; wherein said vital statistics include hard drive usage and capacity, processor type, usage and instantaneous loads, memory usage, and client requested job processes.

19. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on said at least one server hardware device resource.

20. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on a plurality of server hardware device resources in said management system resulting in multiple instances of said software management layers within said management system environment.

21. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on said at least one network hardware device resource.

22. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on a plurality of network hardware device resources in said management system resulting in multiple instances of said software management layers within said management system environment.

23. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on all server hardware device resources in said management system resulting in multiple instances of said software management layers within said management system environment.

24. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on all network hardware device resources in said management system resulting in multiple instances of said software management layers within said management system environment.

25. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on at least two of a plurality of hardware device resources in said management system resulting in multiple instances of said software management layers within said management system environment.

26. The variable architecture distributed data processing and management system of claim 1 , wherein all of said software management layers reside on all server and network hardware device resources in said management system resulting in multiple instances of said software management layers within said management system environment.

27. The variable architecture distributed data processing and management system of claim 1 , wherein at least one software management layer is activated and at least one software management layer is not activated on a hardware device resource where all of said software management layers reside.

28. The variable architecture distributed data processing and management system of claim 1 , wherein said at least one client is a remote machine physically separate from said management system in said networked computer system.

29. The variable architecture distributed data processing and management system of claim 1 , wherein said at least one client is a machine in another remote networked computer system.

30. The variable architecture distributed data processing and management system of claim 1 , wherein said at least one client is an application or process residing on said at least one server or network hardware device resource in said management system.

31. The variable architecture distributed data processing and management system of claim 1 , wherein said at least one client is an application or process residing on a remote hardware device resource connected to but not present in said management system.

32. A method of operating a variable architecture distributed data processing and management system for a networked computer system, said management system including a hardware device layer having at least one server hardware device resource and at least one network hardware device resource, said method comprising the steps of: receiving a job process request from at least one client;

1) managing, under control of a software governor layer, replication of said client requested job processes to all hardware device resources, monitoring of redundancy of said client requested job processes on all hardware device resources, maintenance or termination of all active client requested job processes, connecting of clients to said networked computer system, and the most efficient client requested job process routing through the networked computer system;

2) managing, under control of a software density manager layer, compression and decompression of data;

3) managing, under control of a software security manager layer, encryption and decryption of data, integrity of data transmitted from said at least one client, and client validation in said networked computer system;

4) managing, under control of a software application manager layer, the breaking of client requested job processes into the smallest executable instructions, publication of client requested job processes to hardware device resource processors predetermined to most efficiently process said published client requested job processes, the identification of published and completed client requested job processes, and continuous monitoring of each client requested job process running on each system resource thereby determining said networked computer system service capability and communication of information on hardware device resource capabilities to all hardware device resources in said networked computer system;

5) managing, under control of a software data manager layer, the integrity of all data processing and data transmitted through said networked computer system; and

6) managing, under control of a software resource manager layer, the monitoring and reporting the status of all hardware device resources in or connected to said networked computer system; wherein said software layers are computer middleware sets of components, objects, applications and processes that manage all communications and client requested job processes between said at least one client and said hardware device resource layer.

33. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software governor layer further comprises the steps of: determining the number, presence and status of server hardware device resources in or connected to said networked computer system, said step of determining under control of a software replication manager sub-layer in a software ghost manager sub-layer; replicating data under control of said software replication manager sub-layer to hardware device resources in said networked computer system, wherein said replicated data is identified to processors as "inactive"; and storing the status of each server hardware device resource as a record in a replication log based on said step of determining, said step of storing under control of said software ghost manager sub-layer.

34. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software governor layer further comprises the steps of: sending scouts, or active software processes, to all hardware device resources in or connected to said networked computer system, said step of sending under control of a software integrity scout dispatch sub-layer in a software integrity manager sub-layer; collecting information from said scouts about the state of all hardware device resources in or connected to said networked computer system, said step of collecting under control of said software integrity scout dispatch sub-layer in said software integrity manager sub-layer; determining the number of scouts to be kept active and determining the number of scouts to be terminated, said step of determining the number of scouts under control of said software integrity scout dispatch sub-layer in said software integrity manager sub-layer; determining the current state of client requested job process redundancy for each hardware device resource based on said step of collecting, said step of determining the current state of redundancy under control of a software integrity analyzer sub-layer in said software integrity manager sub-layer; and storing the current state of redundancy as a record in an integrity log based on said step of determining the current state of redundancy, said step of storing under control of said software integrity manager sub-layer.

35. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software governor layer further comprises the steps of: sending scouts, or active software processes, to all hardware device resources in or connected to said networked computer system, said step of sending under control of a software garbage scout dispatch sub-layer in a software garbage manager sub-layer; collecting information from said scouts about all active and inactive software processes on all hardware device resources in or connected to said networked computer system, said step of collecting under control of said software garbage scout dispatch sublayer in said software garbage manager sub-layer; determining the number of scouts to be kept active and determining the number of scouts to be terminated, said step of determining the number of scouts under control of said software garbage scout dispatch sub-layer in said software garbage manager sub-layer; determining to maintain or terminate software processes on each hardware device resource based on said step of collecting, said step of determining to maintain or terminate under control of a software garbage monitor sub-layer in said software garbage manager sub-layer; and storing active and terminated software processes as a record in a garbage log based on said step of determining to maintain or terminate, said step of storing under control of said software garbage manager layer.

36. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software governor layer further comprises the steps of: collecting information about the status of said at least one client in or connected to said networked computer system, said step of collecting under control of a software session regulator sub-layer in a software state manager sub-layer; transmitting said collected information to any software session monitors in said networked computer system, said step of transmitting under control of said software session regulator sub-layer; determining, by receiving, organizing and analyzing information from said software session regulator sub-layer:

1) the current status and number of clients in or connected to said networked computer system;

2) the capacity of said networked computer system regarding potential connections of new clients; or

3) the throughput of data through said networked computer system; said step of determining under control of a software session monitor sublayer in said software state manager sub-layer; creating a routing table based on the step of determining, said step of creating under control of said software session monitor sub-layer in said software state manager sub-layer; and storing all information from said steps of determining and creating as a record in a session log, said step of storing under control of said software state manager sub-layer.

37. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software governor layer further comprises the steps of: collecting information on at least one of the group consisting of:

1) all currently available communication ports;

2) the type of each communication port;

3) the communication protocol of each communication port;

4) the routing of each communication port; or,

5) the current status of each communication port; said step of collecting under control of a software proxy monitor sub-layer in a software proxy manager sub-layer; determining, by receiving, organizing and analyzing information from said software proxy manager sub-layer, the most efficient data routing for the next incoming data from a client, said step of determining under control of a software session regulator sub-layer in said software proxy manager sub-layer; and storing all information from said step of determining as a record in a proxy log, said step of storing is under control of said software proxy manager sub-layer.

38. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software density manager layer further comprising the steps of: receiving message stream data; determining:

1) if compression of said message stream data is necessary; and

2) what compression algorithm is most efficient to compress said message stream data; retrieving a compression algorithm from a compression dictionary based on said step of determining; compressing said message stream data based on said steps of determining and retrieving; outputting said compressed message stream data; creating a compression record based on said steps of determining and compressing that includes data related to the type of compression algorithm used in said step of compressing and data related to the efficiency of compression obtained; wherein said steps of receiving, determining, retrieving, compressing, outputting, and creating are under control of a software compression agent sub-layer in a software compression manager sub-layer; storing all information based on said step of creating as a record in a compression log, said step of storing under control of said software compression manager sub-layer.

39. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software density manager layer further comprising the steps of: receiving message stream data; determining:

1) if decompression of said message stream data is necessary; and

2) what decompression algorithm is necessary to decompress said message stream data; retrieving a decompression algorithm from a decompression dictionary based on said step of determining; decompressing said message stream data based on said steps of determining and retrieving; outputting said decompressed message stream data; creating a decompression record based on said steps of determining and decompressing that includes data related to the type of decompression algorithm used in said step of decompressing and data related to the verification of decompressed data; wherein said steps of receiving, determining, retrieving, decompressing, outputting, and creating are under control of a software decompression agent sub-layer in a software decompression manager sub-layer; storing all information based on said step of creating as a record in a decompression log, said step of storing is under control of said software decompression manager sub-layer.

40. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software security manager layer further comprising the steps of: receiving message stream data; determining:

1) if encryption of said message stream data is necessary; and

2) what encryption algorithm is most efficient to encrypt said message stream data; retrieving an encryption algorithm from an encryption dictionary based on said step of determining; encrypting said message stream data based on said steps of determining and retrieving; outputting said encrypted message stream data; creating an encryption record based on said steps of determining and encrypting that includes data related to the type of encryption algorithm used in said step of encrypting; wherein said steps of receiving, determining, retrieving, encrypting, outputting, and creating are under control of a software encryption agent sub-layer in a software encryption manager sub-layer; storing all information based on said step of creating as a record in an encryption log, said step of storing under control of said software encryption manager sub-layer.

41. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software security manager layer further comprising the steps of: receiving message stream data; determining:

1) if decryption of said message stream data is necessary; and

2) what decryption algorithm is necessary to decrypt said message stream data; retrieving a decryption algorithm from a decryption dictionary based on said step of determining; decrypting said message stream data based on said steps of determining and retrieving; outputting said decrypted message stream data; creating a decryption record based on said steps of determining and decrypting that includes data related to the type of decryption algorithm used in said step of decryption and data related to the verification of decrypted data; wherein said steps of receiving, determining, retrieving, decrypting, outputting, and creating are under control of a software decryption agent sub-layer in a software decryption manager sub-layer; storing all information based on said step of creating as a record in a decryption log, said step of storing under control of said software decryption manager sub-layer.

42. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said software security manager layer further comprising the steps of: determining a format of received data, correcting said format of received data when determined if it is necessary, and verifying the security of said received data by embedded unique identification codes controlled by a software packet analyzer sub-layer in a software access officer sub-layer; determining access rights of said at least one client to system resources controlled by a software client analyzer sub-layer in said software access officer sub-layer; retrieving records from a security dictionary of acceptable and unacceptable clients, types of data formats, and rules for client access to system resources; determining what is the most efficient route in transmitting data to enable client analysis controlled by a software security delegate sub-layer in said software access officer sub-layer; and storing records of all analyzed client transactions and all functions executed by said access officer sub-layer under control of said software access officer sub-layer.

43. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said application manager layer further comprises the steps of: determining the most efficient means to break a client requested job process into the smallest executable instruction and breaking a client requested job process into said smallest executable instruction, controlled by a software job process analysis agent sub-layer in a software job process analyzer sub-layer; and storing records in a memory or a physical location of said client requested job process broken by said job process analysis agent sub-layer and the locations of said smallest executable instruction, controlled by said software job process analyzer sub-layer.

44. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said application manager layer further comprises the steps of: monitoring all processor activities in said networked computer system under control of a software job process monitor sub-layer in a software job process broker sub-layer; determining which processors in said networked computer system are most efficient to process each client requested job process and for publishing client requested job processes to said most efficient determined processors, controlled by a software job process publisher sub-layer in said software job process broker sub-layer; storing records of processor activities from said job process monitor sub-layer and publication of client requested job processes from said job process publisher sub-layer, controlled by said software job process broker sub-layer.

45. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 44, wherein said step of managing under control of said application manager layer further comprises the steps of: displaying at a graphical user interface data from said step of monitoring; receiving user input at a user input device based on said step of displaying; and determining which processors in said networked computer system are most efficient to process each client requested job process and for publishing client requested job processes to said most efficient determined processors based on said step of receiving user input.

46. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said application manager layer further comprises the steps of: determining processor completion of published client requested job processes and identifying said completed client requested job processes as having completed processing, controlled by a software message manager sub-layer in a software queue manager sublayer; storing records of said completed and identified client requested job processes based on said step of determining, under control of said software queue manager sub-layer.

47. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said application manager layer further comprises the steps of: continuously monitoring the status of each client requested job process running on each hardware device resource, and continuously cataloging the networked computer system service capability to perform client requested job processes on each hardware device resource in said networked computer system, controlled by a software service monitor sublayer in a software service manager sub-layer; determining what additional client requested job processes are capable of being processed on said hardware device resources in said networked computer system, communicating information of said hardware device resource capabilities to all hardware device resources in said networked computer system, and continuously monitoring each hardware device resource in said networked computer system and communicating information on all job processes on each hardware device resource in said networked computer system to all hardware device resources in said networked computer system, controlled by a software service broadcaster sub-layer in said software service manager sub-layer; and storing records of said status of each running client requested job process, said networked computer system service capability, said additional client requested job processes capable of being processed, said hardware device resources capabilities, and said information on all client requested job processes on each hardware device resource, controlled by said software service manager sub-layer.

48. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said data manager layer further comprises the steps of: managing the synchronization of all data processing on all hardware device resources within said networked computer system; and preventing interruption of data being processed on said hardware device resources until all running job processes are completed.

49. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said step of managing under control of said resource manager layer further comprises the steps of: collecting, organizing, creating records and transmitting information of all vital statistics of each hardware device resource in said networked computer system under control of a software resource monitor sub-layer; and storing records of said vital statistics on all of said hardware device resources in said networked computer system under control of a software resource profiler sub-layer; wherein said vital statistics include at least one from the group consisting of hard drive usage and capacity, processor type, usage and instantaneous loads, memory usage, and client requested job processes.

50. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, further comprising the steps of: determining which of said software management layers are necessary to be activated for execution of at least one client requested job process.

51. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said software management layers include a plurality of software management sub-layers, the method of operating further comprising the steps of: determining which of said software management sub-layers are necessary to be activated for execution of at least one client requested job process.

52. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said software management layers include a plurality of software management sub-layers, the ^N method of operating further comprising the steps of: determining an order of instructions for said software management layers and sublayers in processing said client requested job process.

53. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said software management layers include a plurality of software management sub-layers, the method of operating further comprising the steps of: determining a communication route within said software management layers and sub-layers for communicating said client requested job process.

54. The method of operating a variable architecture distributed data processing and management system for a networked computer system according to claim 32, wherein said software management layers include a plurality of software management sub-layers, the method of operating further comprising the steps of: determining a communication route of processed data from said software management layers and sub-layers through said management system or through said networked computer system.

55. A method of operating a distributed data processing and management system for a networked computer system, said management system including system resources having at least one server hardware device resource and at least one network hardware device resource, said method comprising the steps of: receiving a job process request from at least one client; determining the smallest portions of executable instructions that can be broken down from said client requested job process; breaking down said client requested job process into said smallest portions of executable instructions; continuously analyzing system feedback from all system resources in said networked computer system capable of processing said smallest portions of executable instructions; designating the most efficient system resources for processing said smallest portions of executable instructions and the most efficient path for transmitting said smallest portions of executable instructions to said most efficient system resources, said step of designating the most efficient system resources is based on said step of continuously analyzing system feedback; transmitting said smallest portions of executable instructions on said most efficient path for transmitting to said designated most efficient system resources for processing based on said step of designating.

56. The method of operating the variable architecture distributed data processing and management system of claim 55, wherein said step of continuously analyzing system feedback further comprises the steps of analyzing system feedback related to at least one of the group consisting of: the processor type, status and availability; the hardware type, status and availability; the communication port type, status and availability; the network communication throughput; the total number of clients in the networked computer system; and the type and number of client requested job processes in said management system.

57. The method of operating the variable architecture distributed data processing and management system of claim 55, further comprising the steps of: transmitting said smallest portions of executable instructions to all other non- designated system resources; and identifying the status of said smallest portions of executable instructions at said non- designated system resources as inactive for processing.

58. The method of operating the variable architecture distributed data processing and management system of claim 57, further comprising the steps of: determining when a specific system resource is no longer efficient at processing a specific one of said smallest portions of executable instructions based on said step of continuously analyzing system feedback; designating the next most efficient system resource of said non-designated system resources for processing said specific one of said smallest portions of executable instructions; identifying said status of said specific one of said smallest portions of executable instructions as active for processing based on said step of designating said next most efficient system resource; executing said identified active specific one of said smallest portions of executable instructions on said designated next most efficient system resource for processing.

59. A method of operating a variable architecture distributed data processing and management system for a networked computer system, said management system including hardware device resources including server and network hardware device resources, and software management layers to manage communications and job processes in said networked computer system, said software management layers including a governor layer, a density manager layer, a security manager layer, an application manager layer, a data manager layer, and a resource manager layer, said method of operating comprising the steps of: receiving a connection request from at least one client; processing said connection request in said software governor layer; transmitting information about said at least one client from said software governor layer to a software client analyzer sub-layer in said software security manager layer; determining, at said software client analyzer sub-layer, if said at least one client is valid and authorized to access to said management system; transmitting information to said software governor layer based on said step of determining if said at least one client is valid and authorized; requesting a new client connection to the management system from said software governor layer to a software proxy monitor sub-layer in said software governor layer; determining, under control of said software proxy monitor sub-layer communicating with a software session monitor sub-layer in said software governor layer, if said at least one client has an existing session and the last machine the client was connected to, and assigning a server hardware device resource for a connection; creating a new connection between said at least one client and said at least one server hardware device resource under control of a software proxy manager sub-layer in said software governor layer based on said step of determining if said at least one client has an existing session; activating a software decompression manager sub-layer in said software density manager layer able to decompress data sent from said software proxy manager sub-layer in said software governor layer; determining if said transmitted data sent from said software proxy manager sublayer needs decompression under control of a software decompression agent sub-layer in said software density manager layer; retrieving a decompression algorithm from a decompression dictionary in said software density manager layer; decompressing said transmitted data based on said steps of determining if said data needs decompression and retrieving a decompression algorithm under control of said software decompression agent sub-layer and notifying software decompression manager sub-layer upon completion of decompression; transmitting the decompressed data to a software decryption manager sub-layer in said software security manager layer; activating a software decryption manager sub-layer in said software security manager layer able to decrypt said decompressed data sent from said software decryption manager sub-layer; determining if said decompressed data sent from said software decompression manager sub-layer needs decryption under control of a software decryption agent sub-layer in said software security manager layer; decrypting said decompressed data based on said steps of determining if said decompressed data needs decryption and retrieving a decryption algorithm under control of said software decryption agent sub-layer and notifying software decryption manager sublayer upon completion of decryption; transmitting the decrypted data from said step of decrypting to a software queue manager sub-layer in said software application manager layer; queuing said decrypted data from said previous step of transmitting in a software message monitor sub-layer in said software queue manager sub-layer; notifying a software job process analyzer sub-layer in said software application manager layer under control of said software message monitor sub-layer that said decrypted data is ready for processing as a job process; determining, under control of said software job process analyzer sub-layer, how said job processes should be processed, whether said job process can be divided into smaller independent job processes, and the type of job process; determining, under control of said software job process analyzer sub-layer querying a software service manager sub-layer in said software application manager layer, which server hardware device resources in said networked computer system contain necessary services to process said job, and designating a specific server hardware device resource having said necessary services to process said job process; transmitting said job process to a software job process publisher sub-layer in said software application manager layer; publishing said job process with an "active" job process status designation from said software job process publisher sub-layer to said designated specific server hardware device resource, and to a software replication manager sub-layer in said software governor layer; transmitting said job process with a "passive" job process status designation from said software replication manager sub-layer to all remaining server hardware device resources, other than said specific server hardware device resource, based on said step of determining which server hardware device resources contain necessary services to process said job process; and managing the processing of all server hardware device resources having said job.