JP2016537746A - Distributed data system with document management and access control - Google Patents

Abstract

The data management system and method includes a cloud-based platform connected to an agent or folder system hosted on a client device. The platform does not store the actual data; instead, the metadata supplied by the agent tracks the location of all the data in the system, manages the distributed storage, and the movement and processing of the actual data between agents. To use. At that time, the platform pools the network-connected storage in a “virtual cluster” that uses local storage at the agent. As described in detail herein, the agent collectively monitors, stores, transfers, or moves data, and performs data processing operations as directed by the platform. Agents are hosted on or connected to processor-based devices, agents hosted on local area network devices, agents hosted on wide area network devices, and hosted on mobile devices. And agents hosted on cloud-based devices.

Description

RELATED APPLICATION This application is a priority application of US Patent Application No. 61 / 877,585, filed September 13, 2013.

  This application is a continuation-in-part of US Patent Application No. 13 / 738,796, filed January 10, 2013.

  The embodiments described herein relate to data processing, and more specifically to distributed storage and processing.

  Cloud-based services are expanding rapidly, and customers also have local customer premises equipment (CPE) to include cloud-based services to increase productivity, reduce support costs, and reduce upfront investments. Extending the solution. Hybrid solutions combining both CPE and cloud solutions are rapidly adopted because the local / cloud approach takes full advantage of existing investments and is less risky than a full migration to the cloud. Yes. Traditional cloud services include personal synchronization and sharing services, online backup, and large file transfer services. While these traditional cloud services have focused on providing online storage, the number of connected computers, devices and websites is constantly increasing, so that users of these services can spread across multiple devices or across the web Through this, I continued to experience the difficulty of synchronizing and accessing my latest files. In addition, enterprise users require a relatively high level of security to protect their business data, and security breaches and cloud storage providers' long-term continuity that are notable as issues related to moving storage and services to the cloud. Is added. Therefore, there is a need for a cloud service that has an efficient approach for file synchronization and access across multiple devices, and that focuses on local data storage and peer-to-peer transfers, as well as a relatively high level of security. Exists.

INCORPORATION BY REFERENCE Patents, patent applications, and / or published patents referred to herein are shown to express their individual patents, patent applications, and / or published patents, specifically and individually, incorporated by reference. In its entirety to the same extent as described herein.

FIG. 1 is a block diagram of a system including a platform and an agent according to the embodiment. FIG. 2 is a block diagram of a system including a platform connected to a group of independent agents and a group of network agents, according to an embodiment. FIG. 3 is a block diagram of a system including a platform and a number of agents included in an organization according to an embodiment. FIG. 4A is a block diagram of an exemplary system including a platform and agents, according to an embodiment. FIG. 4B is another block diagram of an exemplary system including a platform and agents, according to an embodiment. FIG. 5 is a block diagram of an example involving file synchronization between the platform and the agent according to the embodiment. FIG. 6 is an exemplary flow diagram including agents, platforms, and databases, according to an embodiment. FIG. 7 is an exemplary flow diagram that includes the use of a synchronization goal to scan and synchronize a library, according to an embodiment. FIG. 8 is a block diagram of a platform including a policy database and a rule engine according to the embodiment. FIG. 9 is a block diagram illustrating metadata separated from another platform component, according to an embodiment. FIG. 10 is a diagram illustrating platform connection types according to the embodiment. FIG. 11 is a diagram illustrating the P2P connection type of the platform according to the embodiment. FIG. 12 is a diagram illustrating a screen example of the management console according to the embodiment. FIG. 13 is a diagram illustrating a screen example including file synchronization control according to the embodiment. FIG. 14 is a diagram illustrating an example of a screen including security activity monitoring and notification according to the embodiment. FIG. 15 is a diagram illustrating an example of a screen including a document audit trail according to the embodiment. FIG. 16 is a diagram showing an example of a screen including file search and classification according to the embodiment. FIG. 17 is a diagram illustrating an example of a screen including controlled user sharing according to the embodiment. FIG. 18 is a diagram showing an example of a screen including remote device management and geolocation service according to the embodiment. FIG. 19 is a diagram illustrating an example of a screen including selection of a storage location according to the embodiment.

  Data management systems and methods are described that include a cloud-based platform or engine connected to a system of agents or folders hosted on a client device. The platform of an embodiment does not store the actual data, but instead tracks the location of all the data in the system and manages the movement and processing of the actual data between distributed storage and agents. , Use the metadata supplied by the agent. In that case, the system of an embodiment pools network-connected storage in a “virtual cluster” using the agent's local storage. As described in detail herein, the agent collectively monitors, stores, transfers, or moves data, and performs data processing operations as directed by the platform. Agents in certain embodiments are hosted or connected to processor-based devices (eg personal computers (PCs), tablet computers, server computers, network attached storage (NAS) devices, Apple computers, mobile devices, iOS devices, Android devices, etc.) Agents hosted on local area network (LAN) devices, agents hosted on wide area network (WAN) devices, agents hosted on mobile devices, and cloud-based devices (eg, servers, storage devices) Including, but not limited to, hosted agents.

  Although the detailed description herein includes many specific descriptions for purposes of illustration, those skilled in the art will appreciate that many derivatives and modifications to the following details are described herein. It will be understood that it is within the scope of the described embodiments. Accordingly, the following description of the exemplary embodiments does not reduce the versatility of the claimed invention or impose any limitation on the invention.

  FIG. 1 is a block diagram of a system including a platform 110 and an agent 120 according to an embodiment. As described in detail herein, in this embodiment, the platform is connected to and / or includes many databases 130. Multiple client devices 140, each containing or hosting an agent, are connected to the platform and database via a network connection, and to receive and execute work tasks as instructed by the platform Use a database.

  FIG. 2 is a block diagram of a system including an independent agent group and a peer agent group according to the embodiment. The platform of this embodiment is connected to and / or includes multiple databases, as will be described in detail below. The client device is connected to the platform by a network connection. The first group of client devices 210 includes a plurality of client devices, each of which includes or hosts an agent and is separately connected to the platform by a network connection. The second group of client devices 220 includes a plurality of client devices, each of which includes an agent and is separately connected to the platform and forms a peer network (eg, TAN, WAN, etc.). Peer-to-peer communication 230 between agents is supported within the peer network.

  FIG. 3 is a block diagram of a system including a platform and multiple peer agents included in an organization according to an embodiment. As described in detail below, the platform is connected to and / or includes multiple databases. An organization includes a plurality of groups (eg, group 1 to group M (M is an arbitrary number)), and each group includes some number of users (eg, user 1 to user N (N is an arbitrary number)). Yes. Each user of the organization includes some device (eg, device 1 to device P, where P is any number), and each device includes or hosts one agent connected to the platform by a network connection. And includes an administrator console that allows an administrator to access the device according to the hierarchy, the platform includes controls or rules that control access to data according to the organization, and the console to the controls or rules Providing access to the administrator allows the administrator to set and maintain (eg, add, delete, modify, etc.) rules.

  In this example organization, the agent is managed by and executes instructions received from the platform. Each agent is hosted on a device registered with the user. Each user is a subset of a group, and each group is a subset of an organization. The agent on each device indexes the contents of the memory included in the corresponding device (or accessible by the corresponding device), and supplies the memory contents metadata to the platform. The console provides administrator control over the rules that control access to data according to the organization. For example, an administrator can create a rule that specifies that an organization cannot contain files larger than 10 MB. The administrator can create another rule that specifies that group 1 cannot contain music and video files. Yet another example rule specifies that group 2 cannot contain files marked as secret.

  FIG. 4A is a block diagram of an exemplary system including a platform and agents, according to an embodiment. FIG. 4B is another block diagram of an exemplary system including a platform and agents, according to an embodiment. Here, FIGS. 4A and 4B are collectively referred to as “FIG. 4”. The platform uses knowledge of data accessible by each agent acquired by metadata to create or generate a work or assign a work to an agent. Then, the agent executes the work assigned to itself by the platform, and supplies its own data and information of the executed work to the platform. In some embodiments, the agent performs work as directed by the platform, so it is neither an application nor a user.

  The platform at least includes or is connected to various databases. For example, the platform includes an agent database that is a list of all agents available to the platform and associated information for each agent. In addition to the agent database, the platform has other databases for use by agents in tracking the location of all data in the system, managing distributed storage and exchanging information in support of managing real data movement and processing between agents. . Databases include, but are not limited to, the agent database, library database, goal database, and work database as described above. Each agent may have one or more libraries representing its own local disk or storage and a library database containing a list of storage available to the corresponding agent. Storage includes, but is not limited to, any device (eg, disk or disk drive, network mount point, hard drive, flash drive, storage service, etc.) that the agent is authorized for read and / or write access. The platform and each agent have a library database that includes a library of systems and a list of each corresponding agent of the system that can access each library.

  The goal database includes a list of goals, which is a collection of libraries in which a particular task is performed (eg, synchronized). The goal is used by the agent to execute the task. A goal corresponds to one or more libraries and is used to perform a task or work on the contents of the library. In addition, the platform creates a work database for each agent and registers the work tasks to be achieved by the corresponding agent in the work database. The work database contains a description of each task to be performed by the agent, along with information needed by the agent to perform the work. The database will be described in detail.

  The platform of an embodiment is a universal sync engine that forms a grid through coupling between the platform and multiple agents, where the platform acts as a master controller for the agent. The platform includes or is connected to one or more databases in the cloud. The database is information received from the agent, and includes information including data (for example, metadata) regarding the position and state of all data of the grid handled by the platform. An agent is an independently functioning entity that can be hosted or installed on a specific device or computer, but can itself communicate and work with other agents that form the platform and grid. is there. It should be noted that the components of the grid work in concert to accomplish specific tasks assigned by the platform.

  In operation, the agent is hosted or installed on the device, and the agent functions to perform work or tasks assigned by the platform. The platform knows the identity (identity) and location of all agents with which it is associated, and the data that each agent can access. An agent is not an application, but a platform agent that communicates with both the platform and other agents over an encrypted channel to perform work as directed by the platform. In general, a platform generates and includes goals that it is responsible for, and the platform assigns work or tasks to agents and manages agents to achieve the goals. Furthermore, the agent does not require knowledge of the purpose corresponding to the tasks assigned to the individual agent or any other agent.

  The platform of an embodiment does not store actual data in the database of the embodiment, but instead stores metadata corresponding to the data it serves. Therefore, since the platform stores only metadata, the agent functions as a distributed data storage in which actual data is stored. The platform uses metadata to understand the data stored at the agent at the bit level, thereby providing a better understanding of the data than at the file level. The metadata is received from each agent and is generated by the agent from data handled by each agent. In generating metadata, the agent scans the file as requested by the platform. Scanning for each file involves generating data fragments or components from the data forming the file by dividing the file into variable-sized fragments called blobs. As described in more detail, blobs are generated using a data fingerprinting algorithm that generates a data portion having a variable size between a predetermined minimum length and maximum length. The term “file” and “blob” are used interchangeably in the description herein because the embodiments described herein generate blobs from the contents of a file, and blobs represent the contents of a file.

  When generating a blob, the hash value of the location where the blob is split or separated from the rest of the file, the hash of the complete content of the blob, the offset value based on the location of the blob, and the size of the blob (e.g. The blob's complete content hash (blob hash or “bash”) from which the blob description containing the length is generated serves as a unique identifier for the blob. Furthermore, to generate a unique identifier for the entire file content, the complete list of blobs that make up the content of the file is hashed as a hash of a combination of a content hash and a name hash (fash) (cash )). The hash of an embodiment further includes a hash of the file name and file path (pash), and a hash of file metadata (eg, file name, size, date, location, etc.) (mash).

  These various hashes, which will be described in detail here, are generated by each agent for each data file handled by each agent and distributed to the platform. The platform generates a record for each file to include these various hash information. Platforms are reported by agents and the information or status of the data contained in those records can be used to determine the status of the data, to determine which agents have a version of data that has become outdated due to data changes, Used to match file versions in. The platform then issues work to the agent that needs to update one or more files to the latest version, as described in detail herein.

  Thus, using metadata, the platform maintains a master index of data in the cloud, determines the location of various parts of the data associated with individual goals, identifies data inconsistencies and inconsistencies, and identifies identified inconsistencies Goals are used to read the agent's master index or library to publish or generate work to eliminate or conflicts. In some embodiments, the agent is not known anonymously and is directed to the platform, and in addition to exchanging content only with other agents controlled by the platform and having an ID established on the platform, the peer-to-peer system In contrast to

  The platform achieves efficiency by transferring as little data as possible between peers as directly as possible. In addition, in this embodiment, in order to move only the data that needs to be moved and process the data at the position of the data (for example, the agent), the knowledge gathered by the platform for the position of all the data is obtained between the agents. When used in conjunction with cooperative processing, it provides a relatively larger bandwidth than one agent has when operating independently. As such, the platform of an embodiment provides data ubiquity by providing efficient storage, distribution, and global data mobility.

  The system provides hashing and block level transfer between devices through the use of blobs. Since only changed data blocks need to be updated between devices, block level transfer supports efficient file synchronization during data changes. When the data is hashed to form a blob, the blob is reassembled after being moved, so there is no need for sequential file transfers, and block level transfers further support unordered file transfers. Similarly, embodiments support a reduction in upload / download times because a blob can transfer simultaneously or near simultaneously from one or more other devices to any device that needs it. In addition, if the file is needed for a specific device on the grid, and if it exists on other devices on the grid, to place the file at the location where it is currently needed (eg agent) Different blocks can be transferred simultaneously from these multiple devices.

  The platform directs work by assigning work to agents in the grid, generating tasks or work items for individual agents, and sending work items to the agents. A work item is sent with a completion condition for the work item, but embodiments are not so limited. In one embodiment, the agent is a flexible client that performs tasks or work items assigned by the platform.

  As an example, a work item assigned to an agent scans or monitors a directory or folder on a memory device that the agent can access, and continuously reports changes to any file in the directory or folder. To order. This work item is used by the platform to first receive the metadata for the files corresponding to the agent when the agent registers with the platform, and in response to changes to the contents of those files. It is also used to continuously update data. Thus, in response to any change to any file, the scan task causes the agent to report the file's blob or metadata changes to the platform, which responds to one or more other agents depending on the reported change. Assign work to. Furthermore, the platform of an embodiment is used to distribute logic related to new functions to one or more agents. Thus, when a platform needs to provide a new function to an agent, the logic that enables that function can be sent by the platform to the appropriate agent.

  Certain embodiments of tasks or work items generated by the platform and assigned to individual agents for execution include scans, deletes, writes, and updates. The agent recursively monitors the file system or directory and reports scan tasks by reporting changes to the file (eg, file changes, additions to files, deletions to files, file name changes, etc.) to the platform. Run. The change report includes placing the changed file in the local database, and the local database metadata is then provided to the platform.

  The write task of an embodiment includes, but is not limited to, writing a blob as described in detail herein. The agent performs the write task by copying one or more blobs from a first location (eg, source device) to a second location (eg, destination device). Each of the first location and the second location is a location on the storage device (eg, from a first location on a computer hard disk to a second location on the computer hard disk), within the same domain. Location on different storage devices, and location on storage devices in two or more different domains (eg, from agent file system to cloud-based storage device, from first agent file system to second agent via peer-to-peer communication) To the file system, etc.). Thus, the source of the agent may be one of the file system of any agent on the grid and a cloud-based or network-based storage device.

  As described above, the work item is transmitted together with the completion condition of the work item. Completion conditions associated with a write work item include reading the blob or component required to complete the write, and that the blob or component is overwritten with the correct file. If the condition that the blob is overwritten with the correct file is not met, it is reported to the platform.

  The agent performs update tasks by copying blobs from the agent to the cloud-based storage device. Thus, when a new file is placed in the agent's file system, the agent reports the addition of the file to the platform, and the platform assigns the work to the agent to copy the file blob to the cloud-based storage device. In response, the agent determines whether the new file exists in the cloud-based database, and if not, copies the file to the cloud-based database.

  The work item of an embodiment includes three phases including waiting, incomplete, and completion (success or failure). The platform tracks the state of the file corresponding to each agent, and therefore tracks the phase of the work item assigned to each agent.

  To maximize the efficiency of communication between the platform and the agent, some information is stored on both the platform and the agent. For example, the agent of an embodiment polls the platform to identify new work items assigned to the agent through the work database, periodically receives the work database assigned work items, and stores them locally. This reduces the amount of communication between the agent and the server that is necessary for downloading the work item to the agent. Similarly, if a new file is confirmed on the agent, the new file is written to the agent's local library database, and the local library database is replicated in multiple cycles periodically on the platform. This eliminates the need for the agent and / or platform to perform a file transfer for each new file occurrence. In addition, as described in detail herein, certain embodiments of blobs support differential file transfer, and at least some of the information used to track blobs and their location in the system may be Saved in both.

  With regard to obtaining information required by the agent to perform the work task assigned to the platform, the agent is required to complete the task, but to locate information that is not present in the agent-owned file Including the hierarchy. The hierarchy of an embodiment includes the agent first searching local memory and utilizing peer-to-peer communication to obtain the necessary information found in files that are not found locally and belong to the peer. If the required information is not found locally or even at the peer, the agent gets the information from a cloud-based storage (eg Amazon S3, etc.) or another remote storage entity.

  If peer-to-peer communication is used to obtain the required files, the platform provides the agent with a list of peer agents that own the required files, but is not so limited. The list of peer agents supplied by the platform is an ordered list, which allows agents to contact peer agents using the order of the list, although the list in another embodiment is not ordered. Good.

  With reference to FIG. 4B, an agent of an embodiment includes a number of threads or components executing in parallel to provide functionality to the agent. For example, the provider thread retrieves the work item specified by the agent from the platform work item database and supplies or stores it in the local work item database. The runner thread monitors the agent's local work item database, retrieves work items individually from the local work item database, marks the item as pending in the local work item database, and places the item in the appropriate task execution thread. (E.g., write task is supplied to the write thread) When the task execution thread completes the execution of the task received from the runner thread, it reports the execution status to the runner thread (for example, successful completion, unsuccessful completion, reason for unsuccessful completion, etc.), and the runner thread reports this status information. Report to local work item database. The update thread monitors the local work item database for items marked as complete and reports the status information of completed work items to the platform. In response to the completion of work by any agent, the platform updates the platform database as necessary.

  As described in detail herein, an embodiment platform operates with data metadata corresponding to each agent coupled or connected to the platform, as reported to the platform by individual agents. In order to generate metadata for the platform, the agent scans all the files it has in the folder to be synchronized. The agent creates or generates data fragments or components from the data that make up the file by dividing the file into variable-sized fragments (blobs). Blobs are generated using a fingerprint algorithm and are identified by a blob hash (bash). Specifically, the agent generates a blob by running a fast-windowed checksum for a predetermined number of bytes. Some embodiments generate a portion of data having a variable length size between a predetermined minimum length and maximum length.

  The procedure of dividing a file into component parts (referred to herein as fingerprints) includes performing a hash algorithm on the slice of data, which is a specific pattern of data (eg, 12 consecutive “0” bits, “0” "3 consecutive bits", 6 consecutive "1" bits, etc.). The “size” of each part is determined through the setting of a hash character string. This is because a 12-bit continuous character string does not occur very frequently, and therefore a data slice longer than a 3-bit continuous character string is generated. Fingerprints determine the boundaries of data blocks within a file, and the boundaries are used to determine blocks or portions that are common between two or more files.

  A hash is generated for each byte in the file using a hash algorithm over a predetermined size sliding window. For example, in one embodiment, a hash is generated for each byte in the file using a BUZ hash algorithm over a 48 byte window, and the application result of the hash algorithm is a 64-bit number. If the hash has a specific value called a match value (eg, a value in which the lower 12 bits are all “0” bits), the split is performed at that point in the file. The match value of an embodiment occurs, for example, when the lower 12 bits of the window are “0” bits.

  When generating a blob, a hash value where the blob is split or separated from the rest of the file, a hash of the blob's complete content, an offset value based on the blob's split position, and the blob size (eg length) A blob description is generated, including One embodiment hash function used to hash the complete content of a blob is a skein cryptographic hash function (which generates an internal state size of 512 bits and 160 bit output), but embodiments are not limited thereto. . The blob's complete content hash serves as a unique identifier for that blob. Files are reported or described as a list of all blobs that make up the file, and this list of file hashes serves as an inventory of the corresponding file content.

  Some embodiments of blobs are stored in a central storage entity, but are not limited to such forms. In doing so, the platform instructs each agent to transfer blobs that have not been previously reported to the central cloud-based storage entity by any agent. As an example, the central storage entity may include an Amazon simple storage service (S3) available from the Amazon web service. Thus, all blobs are stored locally on one or more agents and in central storage. Later, if the agent reports a blob and the platform determines that the blob does not exist in central storage, the platform forwards the prob to the agent to central storage, unless the blob has already been reported by another agent. To order.

  In addition to the bash described above, the system of an embodiment includes a plurality of other hashes for use in describing data within the system. For example, as described above, the complete list of blobs that form the content of the file is hashed to generate a unique identifier for the complete list. This hash identifier of the entire file, called a content hash (cache), allows a quick comparison to determine whether the two file contents are the same content.

  Another hash of an embodiment includes a hash of a combination of a content hash and a name hash, referred to herein as a file hash (fash). Thus, the Fash identifies the contents of the file at a specific location.

  The hash of an embodiment further includes a path hash and a metadata hash. The path hash or file path hash, referred to herein as pash, is a hash of the file name and file path. A metadata hash called mash is a hash of file metadata (for example, file name, size, date, location, etc.). The hash described here is generated by the agent and sent to the platform, but is not limited to such a configuration.

  An alternative embodiment forms blocks of data as described herein and uses hashing to store individual blocks with version information. This allows versioning such that versioning information is used to reassemble the data to generate an older version of the file.

  Using a hash of an embodiment, a record is generated on the platform for each file, and each record for each file is a full bash of the file, as well as a cache, a cache, a fash, a rush ( pash), and mash. Although an embodiment's library database includes these records for each file, embodiments are not limited to configurations that include these records in the library database. The platform determines which agent has a version of data that has expired due to changes or new data, and reports the agent to the platform and includes it in those records in order to unify the file version among the agents. Use data information or status. The platform uses the cache (hash of file content and file name) to determine the state of a file and to determine if the file reported by the agent is the latest version of the file. The platform then issues the work to an agent that needs to update one or more files to its latest version.

  When the platform is instructed to find a specific part of the data or blob, each agent first looks at the file inventory to determine whether a blob with a specific hash corresponding to that data exists in local storage. Query local copy of. If the agent cannot find the file locally, the agent attempts to find the file on the peer agent using a peer-to-peer protocol. When using the peer-to-peer protocol, the information of the list of agents having the file is used to selectively find the file. The list is received from the platform. Alternative embodiments may include one or more alternative hierarchies for finding data and are not limited to peer-to-peer protocols per se.

  Peer-to-peer communication is supported between agents in certain embodiments, where each agent reports its private IP address and the port it is listening on to the platform. In addition, the platform observes each agent's public IP address and port when the agent reports private information (eg, during agent registration). The platform generates and maintains a list of all agents along with each agent's private IP address and port, public IP address and port. Thereafter, when the first agent wants to establish peer-to-peer communication with the second agent, the first agent queries the platform for the public and private IP address and port of the second agent, receives from the platform, and And try to connect to both using private information.

  Peer-to-peer communication between agents in certain embodiments is encrypted using, but is not limited to, the RSA algorithm for public key encryption. Because some embodiments of encryption have all keys locally held or stored in the grid, non-brid entities cannot access unencrypted data. In addition, complete data security is realized because the platform never stores a complete file.

  In one example involving encrypted peer-to-peer communication between agents, a second agent B wishing to establish communication with the first agent A encrypts data for use in symmetric encryption between agents A and B. A random secret S used in the above is generated. The second agent B encrypts the secret S using the public key of the first agent A acquired from the platform, and supplies the encrypted secret S to the platform. The encrypted secret S is stored in the platform. The second agent B communicates to the first agent A that they are trying to establish a communication session using the session ID, and in response, the first agent A sends the secret S encrypted from the platform. And the secret S is decrypted. The secret is then used by the first agent A and the second agent B to form an encrypted communication channel for peer-to-peer communication.

  With reference to FIG. 4, which is a block diagram of an exemplary system including a platform and an agent, the platform creates or creates a work and assigns the work to an agent, as described herein. Then, the agent executes the work assigned to itself by the platform, and supplies its own data and information of the executed work to the platform. In some embodiments, the agent performs work as directed by the platform, so it is neither an application nor a user. In operation, each agent registers with the platform using a passport before being available for operation. In one embodiment, the passport obtained from the platform includes a key and information identifying the corresponding agent. The platform includes an agent database that is a list of all the agents available to the platform and the associated information for each agent, and agents are placed in the agent database by the platform upon registration with the platform.

  In addition to the agent database, an embodiment platform may have or be connected to a number of other databases in the cloud, and an embodiment agent and platform may be connected to and use the database to store information. Exchange. The database includes, but is not limited to, the agent database, library database, and goal database as described above. Each agent can have one or more libraries, one or more libraries representing their local disk or storage. The library database also includes a list of storage that can be used by the corresponding agent. The goal database includes a list of goals that are collections of libraries in which a particular task is to be performed (eg, to be synchronized).

  For the library database, each agent identifies a storage mount point to the platform, referred to herein as a library. Mounts include, but are not limited to, any device that the agent is authorized to read and / or write access to, such as a disk or disk drive, network mount point, hard drive, flash drive, and storage service, to name a few including. The platform and each agent includes a global library database. The global library database contains a list of each of the system's libraries and corresponding system agents that can access each library.

  In addition, each agent provides information on the contents of each library through the use of periodic “snapshots” or scans of each library, and agents and platforms use the snapshot information to determine the current status of all files in the library. To track. The library database locates each library in the grid and allows queries to identify each agent that can access the library. This architecture allows each agent and platform to know whether they can access a library with a particular ID and whether they are accessing the same library. Thus, each library is represented separately and independently in the library database. Although the platform and agent of an embodiment each include a global library database that includes information for all libraries in the grid, embodiments are not limited to such. Instead, the platform includes a global library database of all libraries in the grid, and each agent includes a local library database that contains information about the libraries accessible to individual agents.

  The platform further includes a goal that is captured or used by the agent to perform the task. Instead of processing agents, mounts, works, etc., the goal is to process only the library. A goal corresponds to one or more libraries and is used to perform operations or work on the contents of the library. Further, the library corresponds to multiple goals in certain embodiments.

  The platform of an embodiment includes a work database corresponding to each agent, and the agent receives the work database from the platform. The platform creates a work database for each agent and adds work tasks to be accomplished by the corresponding agent to the work database. The work database contains a description of each task to be performed by the agent, along with information needed by the agent to perform the work. Any task placed in the work database by the platform may be dynamically changed and / or deleted in response to other work performed in the grid. For example, if a task “Rename file A to file B” is added to the work database and then file B is renamed to file C, the platform will rename the work “rename file A to file C. To "Yes". In addition, any tasks that are no longer used are removed from the work database.

  In one embodiment, the agent provides information about its capabilities to the platform. Alternatively, the platform includes or generates a capability database that has general information about the capabilities of all agents in the grid.

  As described above, the agent receives work assignments or tasks generated on the platform through its work database. Each separate work item in the work database includes a verb that describes the action to be performed by the agent. Tasks or work items that an agent of an embodiment can perform are, for example, scan, delete, write, and upload. The work item is sent to the agent along with its completion condition.

  For example, the scan task reports the current status of files in the agent's library to the platform, and changes made to the library contents (eg, adding one or more files, deleting one or more files) and Includes reporting changes made to any file in the library to the platform.

  Another task is that the agent locates the appropriate file, splits the file into blobs, encrypts the blob, and writes or transfers the list of blobs that form the file to an object store such as a cloud-based storage service This is an upload task. When data is written to the object store, the data is content addressed, so the blob hash is used to reference the blob / data after it is transferred to the object store (ie, the file name is a hash).

  The task of an embodiment further includes locating one or more blobs in the file and copying the blob from a first location (eg, source device) to a second location (eg, destination device). Includes writing tasks. In response to the received write task, the agent attempts to find the blob locally in the corresponding agent library. If the blob is not found locally, the agent tries to find the blob with the peer agent reporting that it has the blob. If the blob cannot be found at the peer agent, the agent retrieves the file from an object store (eg, a cloud-based storage device). If the entire file is written, the list of blobs is reported or written as a file inventory.

  With reference to FIG. 4, an exemplary operation includes synchronizing a document on a device / agent in response to determining that the version of the document is not the latest version as found on other devices / agents. Following registration of the agent AGENT_X using the passport, the agent is instructed to generate a library file containing the contents of the library LIBRARY_X and perform a scan to send to the platform. To initiate a scan, the platform adds a scan task to the work database (eg WORK_AGENTX) for a specific agent (eg WORK_AGENTX), and the platform does everything necessary to synchronize files to the work database for that agent. Write the workpiece.

  Library LIBRARY_X stores metadata about files residing on mounts accessible to AGENT X, and includes metadata that defines a subset of files by library identifier or file filter. Thus, the library contains the metadata for the fileset that exists on the mount. Scanning is a job that runs continuously. As a result of the job, for each library, metadata indicating that the contents of the library (for example, LIBRARY_X) are “scanned” is supplied to the platform. As a result, a mirror of each library on each agent is typically generated so that the library database contains metadata that is filtered based on the application of a specific filter to the mounts that each agent can access. Placed in the platform library database.

  As a result of providing the operations described above for all agents accessible by the platform, the platform includes metadata of the contents of all files of the agent that it can access. The platform also includes “synchronized” goals for LIBRARY_X (in this example) and LIBRARY_Y (not shown) of other agents (not shown) in the goal database. This action generates a synchronized document that tracks the current signature or goal state of each file for each of these libraries, using an entry corresponding to each file, and the library file or item within an individual library. The goal state is updated with the change of. Using this example, the synchronization document tracks the current goal state of LIBRARY_X and LIBRARY_Y, and if either file is not in the correct goal state, the work to make LIBRARY_X and LIBRARY_Y the correct (same) goal state is the platform Generated and assigned by In this example, the goal state is determined by the last or latest state, and the LIBRARY_Y document is the latest file version, so work is assigned to LIBRARY_X to update the LIBRARY_X file to the same state as the LIBRARY_Y file. . The system then initiates a session with the appropriate entities (eg, one or more peer agents, one or more remote storage devices, etc.) so that the information necessary to exchange data and synchronize the library can be obtained.

  Another example with file synchronization between agents is shown in FIG. FIG. 5 illustrates an example block diagram with file synchronization between a platform and multiple agents, according to an embodiment. This example includes three agents A 502, B 504, and C 506, each agent having a corresponding library LIB_A 508, LIB_B 510, and LIB_C 512, but embodiments are not limited to such a form. The platform includes a goal 514 in the goal database to synchronize these three libraries. The platform includes a library database having a number of records 516. Each record 516 corresponds to one file managed by the platform. Each record contains information in the form of one entry for each instance of the file, and the entry contains library information, including a file and a hash (P) of the file path.

  For this example, assume that agent A reports a hash (eg, H2) that differs from the hash (eg, H) reported by agents B and C for a particular file. The platform determines from the information in the library database that agent A is reporting hash information different from agents B and C for the file and that the file on agent A is the latest version. In response, the platform generates work for agents B 518 and C 520 (eg, WORK_B and WORK_C, respectively) to update agent B 518 and C 520 files to the latest version owned by agent A. The generated work is for specifying and acquiring the position of the corresponding file, and the acquired information is used for updating the file. Also, work tasks are placed in the platform work database by the platform. The work task generated by the platform of an embodiment includes, but is not limited to, information about the location in the system (eg, Agent A) where the latest version of the file is stored. The work task obtains the corresponding file from one or more appropriate locations (eg, one or more agents controlling the blobs of the file) and uses the obtained information to update the file. Including. Work tasks are placed in the platform work database by the platform.

  In this example, agents B and C obtain their respective work (eg, WORK_B and WORK_C, respectively) from the platform work database. As indicated by the hash H2 (not shown) in the library of Agents A, B, and C, each of Agents A, B, and C includes the latest version of the file when the work is complete. In addition, each of agents B and C reports the completion of the work task to the platform.

  More specifically, FIG. 6 is an exemplary flow diagram including agents, platforms, and databases according to an embodiment. In this example, a new agent A registers with the platform at 602 and an entry is created in the agent database corresponding to the new agent A. When agent A is added to the agent database, the platform creates a work database WORK_A for new agent A at 604. The work database WORK_A is generated to include a work or task for the corresponding agent A.

  In addition, a new library entry is added at 606 to the library database corresponding to the new agent A, and in response, a new database LIB_LA is created at 608 for information about the agent A library. In addition, a new synchronization goal SYNC (LA) is generated at 610 and added to the goal database, the new synchronization goal has information, and the platform uses this information to create a new agent A file and other platform accessible platforms. Command synchronization with agent files.

  In response to the registration of the new agent A, the platform first needs information about the contents of library A, and then needs information about changes to the contents of library A. FIG. 7 is an example flow diagram according to an embodiment involving the use of synchronization goals to scan and synchronize a library. Following the registration process 702 of agent A, the synchronization goal SYNC (LA) generates a work task at 704 and adds it to the work database WORK_A of agent A. As described in detail herein, Agent A responds to the work task by scanning the contents of Library A to examine or learn the initial contents of the library. In addition to passing the metadata of all the contents of library A to the platform, information on the contents of library A is then used to determine when a change has occurred in the contents of library A.

  Using the content information in library A, agent A then monitors library A for changes to the content. Changes can include, but are not limited to, adding new files, deleting files, changes to file contents, and renaming files. In response to detecting a change to the contents of library A, agent A posts the change to library database LIB_LA at 706. In addition, Agent A passes metadata corresponding to all subsequent changes in the contents of Library A to the platform.

  In some embodiments, the platform includes a synchronization database. In order for the platform to include and maintain agent records and file states, it more specifically includes an entry for each file and / or each agent that includes a content hash corresponding to that file and / or agent. Generate a synchronization database for inclusion. When the platform detects posting changes in the database library LIB_LA, the platform updates the synchronization database to reflect the changes in Library A at 708. This change to library A means that the other libraries (eg B and C) are no longer synchronized with library A. As a result, the platform 710 generates work for the agents of these other libraries so that these other libraries are in sync with the contents of library A. Thus, in this example, work tasks are generated for agents B and C to synchronize their libraries.

  As another example, the platform and agents of an embodiment form a virtual NAS. Therefore, instead of storing all data on the network storage device (in this case, the data needs to be transferred to the network storage device) to create a virtual NAS that uses the data index and the corresponding metadata of the platform Platforms and agents can be used. In this way, a NAS is formed using a collection of computing devices that form a grid, and thus the NAS is controlled on-site and by an owned agency.

  As an example, a small office environment includes a network attached storage (NAS) device for use in backing up all computers in the office. However, after some time, the office computer data consumes the entire memory of the NAS device, and additional storage is required to back up the networked computer. Using conventional technology, the first NAS needs to be replaced with a NAS with larger memory, or each computer in the office needs to be assigned to the NAS so that not all computers are backed up to the same NAS I will. However, the use of a platform according to embodiments removes this problem because the platform can be used to communicate with agents hosted on each of the memory devices or hosted on each computer in the office. Each agent has the ability to write to each NAS. In that case, the platform communicates with all members of the grid formed by the agent, and by the communication, the platform has information on the location of all data of the office. Using office data metadata, the platform tells each agent as to where it should store backup data, or where it should go to retrieve or retrieve the data it needs to achieve processing operations. Can be directed.

  As another application example using the platform according to the embodiment, a user adds a digital photo to a folder, and then tweetes all the photos stored in the folder using a Twitter (registered trademark) application. Suppose you want to. In one embodiment, the agent is installed in the Twitter API and the platform includes a goal that all photos placed in the folder are served to Twitter. With this goal, the platform can instruct the agent to supply the photo to Twitter, and then the Twitter agent can be instructed to generate a tweet using the photo. As yet another example, a Twitter agent can be instructed by the platform to store all tweets received on an account at a specific location on the NAS.

  As yet another example, an embodiment platform provides content transcoding and streaming instead of file transfer. In this case, a relatively large file to download to the device can be transcoded and streamed from one or more peer agents to the device instead of downloading the complete file to the device.

  To help companies protect sensitive data, as described in detail here, an embodiment platform provides visibility, analytics, management and security for mobile content, Includes software-as-a servece (SaaS) -based data synchronization and content management platforms that span desktops and public, private or local clouds. The platform includes a cloud-based distributed data system that can discover, track, and manage content across devices, users, and geographies. The platform balances the strength of peer-to-peer (P2P) systems with the centralization and control of more traditional cloud data storage platforms. This new hybrid approach provides uncompromised control and security.

  An embodiment platform includes a cloud-based policy engine, an agnostic file storage environment, highly efficient data transfer, automated data discovery and segmentation, and multi-layered entertainment. Differentiate from other cloud-based synchronization and content management platforms with prize-grade security. In traditional cloud service environments, most of the platform intelligence is hosted on mobile and desktop clients, and the cloud functions as “dumb storage”. Certain embodiments of the platform take a more flexible, powerful and secure approach. Instead of “smart clients” accessing “non-intelligent storage”, the platform has a “smart cloud” that manages “non-intelligent clients”. With this approach, the platform issues commands to clients (mobile and desktop devices) to improve functionality and security.

  FIG. 8 is a block diagram of a platform including a policy database and a rule engine according to the embodiment. The platform has all service policies in the policy database at the cloud level, and the policy database is connected to the platform rule engine. For example, the policy may be “Copy any new files in folder X to user A, B, C devices”. In addition to policy database policies, the platform maintains a complete database of file metadata, users, devices, groups and organizations, giving the platform a complete understanding of the environment. The platform configuration includes a policy database connected to receive device characteristics, file metadata, and permissions (eg, user permissions, group permissions, organization permissions, etc.).

  At the client level, the local agent (i) monitors for events (eg, new file additions to folder X), (ii) performs indexing on files (files contain the following key terms), ( iii) Upload file metadata, device status, and new events to the platform, and (iv) process work published from the platform (eg, data movement, folder scan, file generation, connect to peer, etc.). The agent reports each piece of data moving through the network to a policy engine in the cloud, allowing the platform to know where the data is located and where it needs to be placed.

  When new policies are added at the agent level or new events occur, the platform acts as an administrator or “traffic cup” to determine what action to take. Based on the policy engine, the platform issues commands to cause one or more local agents to perform tasks. One or more agents execute only work issued to them from the policy engine. This approach facilitates abstraction of other critical components of the service, new iterations of new use cases (eg data backup, information management) due to the presence of intelligence in the cloud, and metadata files An improved security model, separated from the data, becomes possible.

  Previous cloud service providers forced customers to upload and store data on a cloud infrastructure as a file storage access point. These providers benefit from this architecture by charging customers for storage and consolidating all of the user data into a single storage system. This allows service providers to de-duplicate content or charge for storage that is not actually used. There are several security issues with this approach, including the following: The company cannot select its own storage location and therefore has no control over its data. The company must also trust that the service provider correctly deduplicates the data and does not mix its sensitive business content with other users on the system. Furthermore, a company must share its encryption key with a service provider in order to use functions such as sharing. In addition, the data is exposed to potential violations.

  In contrast to previous cloud services, the platform of an embodiment uses various public, private, and local clouds as file storage access points. To provide this agnostic file storage access point, the platform separates the file content from the metadata and treats the two in very different ways. FIG. 9 is a block diagram illustrating metadata separated from another platform component, according to an embodiment. More specifically, the metadata is aggregated by the corresponding server and stored in the cloud, while the actual file or file data is stored in one or more storage access points as determined by the customer. In addition, advanced data fingerprinting and hashing controlled by the platform allows file data to be securely stored in multiple data stores. This approach allows customers to choose where to store files, improving security, leveraging existing infrastructure, enforcing policies to store data locally, and safe Or meeting harbor requirements.

  Traditional cloud service providers typically perform an inefficient two-step procedure for uploading all data from the client to the provider's central cloud and then downloading it to the connected client for data transfer . Depending on the level of data deduplication, hashing, and P2P capability, this data transfer procedure can take a lot of time or can require significant data transfer costs.

  In contrast, an embodiment platform uses a series of techniques to transfer data. For example, the platform of an embodiment performs block-level file transfers by moving blocks of data rather than entire files. As an example, if a certain 1 GB file is changed, a hashed block (for example, 10 MB block) of the changed file is transferred instead of the entire 1 GB file.

  The platform includes a data store index that indexes the data blocks throughout the system, thereby allowing the data blocks to be transferred from the nearest point to the agent requesting the data. For example, if a file has been uploaded to the cloud from one agent, the requesting agent downloads the file directly from the cloud instead of requiring the uploading agent to re-upload the file for download .

  FIG. 10 is a diagram illustrating platform connection types according to the embodiment. An embodiment platform uses two connection types. The first connection type has HTTP / TLS between various local components and online servers and services. This is equivalent to a web browser and has the same firewall as the browser on the same machine. The second connection type of an embodiment is a peer to peer (P2P) protocol. FIG. 11 is a diagram illustrating the P2P connection type of the platform according to the embodiment. Platforms take advantage of P2P communication to efficiently move data between agents. For example, two agents on the same LAN can move data directly more efficiently than when going through a cloud storage provider.

  A platform of an embodiment uses UDP (User Datagram Protocol) to communicate P2P, but is not limited thereto. Using information from the service, agents will attempt to establish a UDP session between each other. This works if both agents are behind the same firewall. For peers outside the firewall, communication is more complex. Some enterprise firewalls will not allow this traffic. In such cases, the platform will “relay” data between peers using HTTP / TLS.

  The platform performs a collaborative file transfer. In collaborative file transfer, multiple agents pool their upstream bandwidth in order to move large files apart at a rate faster than any single agent might be able to achieve on their own. Is possible. This technique can be used for both cloud storage relay and P2P data transfer, but is not so limited.

  The platform of an embodiment performs out-of-order / simultaneous transfers where out-of-order data transfers are supported with simultaneous file uploads and downloads. Using this method, the data is reassembled at the agent when all blocks of data arrive, regardless of the transfer order of the data blocks.

  Previous cloud service providers offer minimal data visibility within the platform. Users can access their files, but information technology (IT) administrators have little or no visibility into the data in the system. Some service providers allow IT administrators to see a list of file names across the system, while other service providers do not provide any details about the files in the system.

  The platform of an embodiment not only provides the IT administrator with a complete list of organizational files that exist in the system, but also how sensitive and confidential each document is. Analyze the contents of each file to determine. In order to achieve this, the platform first clarifies personal content and business content, and then determines the confidentiality of the specified business data.

  To clarify personal and business content, the agent scans documents in the local data store and extracts key terms from those documents. These key terms are then compared across the organization to generate a list of company specific key terms. Using this organizational key term set and other user-specific methodologies, the platform segregates business and personal data across local stores, allowing IT administrators to focus on managing enterprise data.

  To determine the confidentiality of business data, the platform searches for the presence of specific key terms (eg, “trade secret”) and analyzes the specific syntax of the data (eg, credit card number, email address, phone number, etc.) And perform other user-specific analysis. This decomposition enables the IT administrator to specify the most confidential content in the company. When document confidentiality becomes part of the metadata, the platform allows IT administrators to establish specific controls that limit where and how data can be accessed and shared. .

  Automated data discovery and content decomposition eliminates the need for IT managers to identify sensitive business content without relying on end-user tagging or classification to scrutinize personal or non-critical business files It is possible to find critical content and generate specific controls to protect sensitive content.

  In previous cloud service platforms, each individual user data is stored together and deduplicated across the entire data set (eg, all user data is stored by the cloud service provider). This mixed approach can lead to potential data leaks if the file is not indexed correctly.

  The platform of an embodiment is configured to manage data in a controlled and secure manner throughout the framework that provides security at the access, application, infrastructure, network, transmission, and data levels. As described in detail herein, this approach allows data from each user and organization to be managed within a “security silo” with its own set of security information.

  The platform provides each organization with a unique organizational encryption key that is used to encrypt data in shared storage or the cloud. Furthermore, an eigenvalue of the organization is given as a salt to all hashes calculated for the data. Even if the basic data is the same, no two organizations share the same hash. This allows the platform to deduplicate data between customers, i.e., security weaknesses, while enabling deduplication of data at the organization level, saving storage space and minimizing unnecessary data transfer. Realize. If an unauthorized user accesses a backend service or data store, the user will be limited to seeing only encrypted blocks that are not available.

  The platform is configured to facilitate file sharing between authorized parties. Authorized agents and clients can access and post organization data seamlessly. Since the data is not encrypted at the user level, the data belongs to the organization. When a new user is added to an organization, the new user receives a key necessary for generating and accessing data related to the organization. The data generated by the user can be used by other users in the organization even after the user leaves.

  The organization's encryption key is generated by the platform when the account is provided and then encrypted using passwords from the organization's IT administrator and user. The key cannot be accessed without a user or administrator password, and these passwords are never stored on the platform. The platform cannot perform decryption of customer data and there is no master key unless the user is in an active session and provides his credentials. However, the platform stores a recovery key that can be used to apply a new password to an account whose password has been forgotten.

  Device level data is stored in its local file system and the platform does not encrypt or encode local device file system data, but is not so limited. However, any data or metadata coming out of the device is SSL (device to web), end-to-end encryption (device to device), or at-rest encryption (device to cloud or shared) To storage).

  In one embodiment, each agent on the device has two 2048 bit RSA public / private key pairs. One is used to ensure end-to-end encryption and the other is used to sign messages and authenticate message sources. These keys are pre-generated and assigned by the platform, but once assigned, they are not stored in the cloud. Keys and certificates are always stored in a secure key store provided by the operating system. Unlike previous P2P, there is no “anonymous peer” in the platform. Any connection between agents occurs as a result of the central cloud determining that it is the most efficient path between agents. The ID of the source and destination of the data transfer is known before the connection and is verified separately by each agent. Agents cannot accept connections they do not expect.

  When agents join or connect to each other for data exchange, they use public key cryptography to provide end-to-end encryption of all data transmitted between agents. The platform holds public keys for all agents and gives public keys to both ends of all P2P requests. The platform does not hold a secret key and the secret key is only held at the agent level, but is not limited to such an embodiment. The agent uses these secret keys to establish symmetric encryption keys that are used to encrypt data between peers using the AES-256 algorithm used in government agencies. This ensures that unauthorized users cannot decrypt the content of messages between peers. Sessions between agents are temporary, and if they expire, a new connection must be established and a new key exchange is required.

  SSL is widely accepted as a standard for secure communication with web servers. As long as the client or browser is connected to the platform, it uses SSL to secure the traffic. Metadata and agent instructions are also secured in the same way.

  The platform ensures that all data stored in the cloud or shared storage device is encrypted. When an agent is asked to save data, it uses an organizational encryption key to encrypt the data. Data stored in this way can only be decrypted by authorized agents. At-rest data is also encrypted using the AES-256 algorithm.

  The separation of data and metadata in certain embodiments provides additional security and complete control while minimizing costs. All data is hashed and encrypted using the customer's unique key, and metadata is stored separately from the data.

  The platform provides an easy way to access, update and share critical business data across mobile devices, desktops and the web. Users can access files across iOS and Android smartphones and tablets, Windows and Apple computers, and web browsers. Files modified on one device are automatically synchronized to all other authenticated and connected devices. Users can also share files with colleagues and individuals outside the organization through interactive shared folders or one-way web links.

  More specifically, the platform of an embodiment provides backup and recovery that protects data, applications and operating systems across various environments (eg, desktops, mobile devices, servers, cloud services, etc.). This allows the user to monitor installed backup applications, integrate key backup metrics into management code, and alert the user that sensitive files are at risk.

  The platform ensures that all sensitive documents are included in the backup plan and verifies that those documents are currently backed up. Administrators are alerted to any confidential documents that are not protected within a period of time, and the location of the confidential documents in the organization is identified. The platform further restores from alternate locations if backup fails, and ensures that confidential documents are backed up to multiple locations by duplication or duplication of backup locations across multiple users. File backups are deleted according to a specific retention policy, ensuring that confidential documents are backed up to a more secure off-site storage location. In addition, the platform provides a backup service that allows remote configuration and deployment of backup operations.

  Backup and recovery provided by the platform includes file source backups of desktops, servers, shared storage, mobile devices and online storage (these are examples). File types involved in backup and recovery operations include data files (eg, documents, videos, music, images, etc.) and application specific files (eg, Outlook, PST, MS exchange, database, etc.). File selection selects the files to back up based on file type and / or folder location, or performs a backup of the entire drive or computer system (eg, system image). File storage stores selected files on an external hard drive, NAS or online, or chooses multiple locations for redundancy. Automated or scheduled backup automatically adds new or modified files to the backup everywhere the storage device is detected and schedules the backup based on the schedule.

  The file recovery of an embodiment is configured to easily restore a selected file or restore all files completely if a computer crashes or a file is inadvertently deleted. Data loss prevention prevents data loss by monitoring data in use (eg, endpoint actions), moving (eg, network traffic), and stored (eg, data storage). The file level DLP software identifies a confidential file and then embeds an information security policy in the file. Platform anytime, anywhere access provides access to all files and folders at any time online through the user terminal. Version history saves past versions of a file and allows the user to revert to those versions. The deleted file is stored for a period of time so that if the deletion is accidental, the user can restore the deleted file. File systems and unstructured content are achieved based on corporate policies to reduce costs and mitigate risk.

  Certain embodiments of the platform provide file sharing and access that allows users to access and share files and other content across various environments (eg, computers, mobile devices, servers, cloud services, etc.) and the web. I will provide a. Files can be shared via interactive folder sharing and web link sharing, and key metrics and controls can be integrated into the management terminal.

  The platform limits or controls sensitive documents by preventing sharing with a given user, group, or outside the company. Sensitive documents cannot be accessed through the platform and / or downloaded to the mobile device, and only online viewing is allowed to maintain control. And document download is limited to temporary copies that are erased when they are no longer connected. Files cannot be copied to platform-related folders or to external storage devices or locations.

  Furthermore, the file cannot be emailed as an attachment, sent through text electronic messaging, or uploaded to social media. Some embodiments include exception handling to allow a one-time request. Detailed reports on the location of the document, who has access to the document, revisions and access history are also available.

  The file sharing and access of the platform of the embodiment provides content management that creates, manages, edits and stores files across multiple devices and systems. The synchronization of an embodiment selects files to be synchronized across multiple devices in order to make the files available from any device. Changes are also automatically synchronized. Anytime, anywhere access allows access to any file from any device (eg, desktop, laptop, phone, tablet, online, etc.). A shared folder provides access to a folder that allows an internal or possibly external user to view and edit files contained within the folder and to synchronize changes with all users. Finer permissions are included (eg, manager, editor, viewer, etc.), but embodiments are not so limited.

  The platform includes public links that allow files to be shared by links accessible to anyone, including non-users. The file is read-only to prevent editing, but the user can view the file and download is limited in some cases. External document management and security controls what happens to a file after it has been shared externally by locking the file so that it cannot be printed or edited. Collaborative inspection provides an exhaustive inspection and tracking tool to give a better view of content sharing and file download activity. The user can see who has not downloaded the associated file. Large file transmission provides an alternative to e-mail for transmitting large files online, such as sharing public links. Email integration allows a user to email any file directly from any device by right clicking on the file or selecting an email option. Mobile preview allows the user to view any file on any mobile device. Mobile document editing provides the ability to edit files on mobile devices. Mobile printing allows a user to print any file on any mobile device by specifying a default remote printer.

  Mobile scanning allows a user to scan an image or paper document from a mobile device for access across multiple devices. The quarantine system uses antivirus capabilities to isolate infected files from the hosting system. Regulatory compliance will take measures in compliance with SOX, HIPPA, PII, PCI, FINRA, FDIC, SAS70, Basel II, Dodd-Frank, and FCPA, to name a few examples.

  The platform of an embodiment includes collaborative production, including simultaneous editing of documents (eg, word processing, spreadsheets, presentations, project plans, etc.) across various environments (eg, desktops, mobile devices, servers, cloud services, etc.) Provide an environment. Alerts that shared documents are in use, notifications of changed documents, and modification and access history are available. Therefore, the platform notifies the user to prevent a conflict if a document is in use, locks out the user if the document is in use, and someone edits or adds to the contents of a shared folder or file Or send an alert and notify the user of the released document. The platform allows online collaboration only through simultaneous editing and provides a correction mediation and merging tool, correction and access history, check-in / check-out system, and offline integration.

  An embodiment collaborative environment provides many important functions. Multiple people can view and edit the same file. The permission gives a plurality of sharing authorities with different granularities for editing the same file (for example, an administrator, an editor, a viewer). The collaboration auditing feature provides an exhaustive inspection and tracking tool to give a better view of content sharing and file download activity. The user can see who downloaded the relevant file and who did not download it.

  The activity history function tracks all file activity, such as when and when a file change was made. The email notification feature sends an alert when someone edits or adds content to a shared folder or file. The comment feature allows comments on files and / or discussion areas to add feedback and discuss changes. The annotation function adds annotations for collaborative production and discussion to existing files. The business purpose management function provides automation of business processing using information technology so that it can be changed more efficiently, more effectively, and better. This includes items such as workflow or task management.

  Workflow / task management assigns tasks and deadlines as part of the preview process and assigns file specific tasks for IRA update, review and approval. Project management breaks down goals into milestones with individual deadlines to progress towards the final goal and assign tasks based on the various milestones. The online document editing function edits various file types online without having to first download them to the user's local computer. The mobile document editing function edits a file on a mobile device.

  The mobile preview feature provides the ability to view any file on any mobile device. The mobile print function prints an arbitrary file on an arbitrary mobile device by designating a default remote printer. The mobile scan function scans an image or paper document from a mobile device for access across multiple devices. This feature digitizes paper documents on the go.

  The platform of an embodiment provides document management that controls access to documents based on user rights, document availability, and document status. Certain embodiments of document management integrate with third-party applications (eg, offices) to maintain workflow, enable versioning and audit trails for documents, and to enforce security requirements for documents. The platform therefore allows users to index, track, catalog, and classify file confidentiality, with limited rights management for individuals or groups. Give file access.

  The platform assigns users rights to files (eg, administrators, editors, viewers) and restricts confidential documents in various ways. The file cannot be changed for a given user and is read-only. Unless there is a request not to be destroyed, the file is destroyed after a certain period of time and cannot be captured by image capture software. The file format is converted (eg, to PDF) before distribution so that the document content cannot be copied or pasted into another document (eg, read-only). Furthermore, the file can only be stored on the corporate server and therefore cannot be stored on the public cloud. Files and file derivatives are then tracked across one or more storage devices. The platform also includes files and non-file based content (eg, Wiki, Evernote, social media, etc.), and there is exception handling that allows one-off requests. The platform provides a detailed report on the location of the document, who has access to the document, and who actually viewed the document, and a detailed report on which device is in which location. Modifications and access history are also available through the platform.

  The platform uses business information identification to find confidential files. The platform extracts keywords and compares the extracted terms across the organization. The platform includes document name, document size, proximity (eg, 90% of a folder is a business document, 100% is considered a business document), keyword comparison, and hash comparison (these are part of the example) One or more of can be used to identify business documents across multiple sources.

  The platform also classifies sensitive data. The marked document is manually tagged as confidential. Examples of selected key terms include “trade secret”, “income statement”, “partner pipeline”, “password”, and “credit card”. The key term that is most relevant to documents that are marked "secret" is the learned key terms. The platform performs inference based on the detected content (for example, five examples of company names). The platform also has syntax (eg “###-##-####”, “3767-######-#####”, aaaaaaaa@aaa.com, “###-# ##-#### ”) to classify the data. The platform records document attributes such as who owns it, where it is located, what permissions it is, and who can access it.

  The platform further classifies sensitive data including intellectual property, customer information, employee information, sales information, marketing information, and accounting information. For example, intellectual property includes proprietary engineering designs, scientific formulas, source code, trade secrets, and new (unknown) product information. Customer information includes name, email address, customer password, phone number, social security number, address, credit card number (last 4 digits), and birthday. Employee information includes social security number, date of birth and address. Sales and marketing information includes customer lists, quotes, marketing and sales plans, material costs and profit margins, project plans and spending plans. Accounting information includes accounting records and strategic business plans.

  The document management included in the platform is the search / indexing of tracks and the search and retrieval of electronic documents by indexing unique document identifiers, metadata or throughword indexes extracted from the content of the document including. Authority management allows an administrator to give access to a document only to a given person or group of persons based on type.

  The collaboration provided by the platform allows documents to be acquired and modified or edited by authorized users. In a collaborative session, a single document can be viewed and modified by multiple users simultaneously, and the platform stores markup made by individual users during the collaborative session. However, other users should be blocked while the document is being modified or changed by one user. The platform also sends alerts and notifications about document availability, but is not limited to such forms.

  The platform versioning of an embodiment obtains past versions of a document and allows the user to continue working from a selected point in time. The third-party integration feature integrates the third-party application so that the user can retrieve existing documents directly from the document management system repository, make changes, and return the changed documents to the repository as a new version. Publication and distribution of documents through the platform includes proofreading, peer or public review, authorization, printing and approval. Security includes compliance requirements for a given document and can be relatively complex depending on the type of document (SOX, HIPPA, PII, PCI, FINRA, FDIC, SAS70, Basel II, Dodd-Frank, FCPA, etc.). The platform includes fine-grained functions such as printing, copying and transfer control, and the ability to add watermarks to documents and wipe documents. The workflow provides automatic routing of individual documents or binders using schedule management functions such as scheduling, checklists, review note management, and the like. External document management and security functions control how files are shared and integrate with other modules and third-party systems.

  The platform of an embodiment provides data intelligence and information governance and identifies data (eg, documents, evernotes, wikis, etc.) across desktop, mobile devices, local storage, network storage and cloud services. This data intelligence and information governance generates more information-based decisions regarding the management, retention and disposal of business data, identifies compliance violations, and determines infrastructure plans. The platform examines business data location, ownership and usage analysis and includes limited information compliance (such as record retention) and limited information governance (such as SS numbers, credit cards, etc.).

  The platform allows the user to go through all of the sensitive files and understand certain information about the data. Users can also delete files from any location that is not secure (eg, mobile devices, public clouds, home computers). Policy selection may be applied to file metadata (eg, retention, expiration, access, etc.) for record retention, defensive deletion, and compliance enforcement. A full document search can be performed across all data sources using file names, document key terms, or full text search.

  The platform's data intelligence and information governance determines how much data belongs to the user, where the data is located, who owns the data, where the data was, the age of the data, and the user's work Dimensional maps and business analysis or rules to find out how much data is associated with. The platform improves file visibility by providing file classifications wherever it is hosted, searches the content, discovers compliance and regulatory violations, and discovers stale non-business related data . The information tree provides a time series set of security-related records and a destination and source of records that provide a series of activities. The platform tracks the relationship and location of all files, information governance provides policy management including record retention, defensive deletion and compliance enforcement, as well as account numbers, social security numbers, credit card numbers, trade secrets, accounting records Protect business critical information assets such as strategic business plans and IP / source code. Once sensitive data at risk is found, the automated policy can change the security attributes of the file and move the data to a more secure system in the enterprise or delete the risk item.

  The record management implemented by the platform implements and automates proactive and ongoing record retention and disposal strategies. File share cleanup allows users to achieve control of business data sprawl by understanding the user's environment. The platform classifies data, updates data attributes, and assigns permissions. Intelligent e-Discovery provides legal hold notification, early assessment of true litigation cases, and information gathering. Platform data removal and cleanup leverages data topology maps and classifications to identify non-business data and information management such as storage / data migration, e-discovery, records management, or compliance mandates Perform defensive data cleanup prior to initiative. Storage optimization optimizes the use of existing storage resources to reduce storage costs and leverages data topology maps and classifications to monitor and plan for future storage needs.

  The platform of certain embodiments provides mobile device management that monitors and supports mobile devices and controls data to help organizations enforce policies as well as maintain the desired level of IT control across multiple platforms and applications. As well as allow protection. The platform provides end-to-end security through centralized management of mobile devices using one mobile device software product. The platform also includes device usage tracking, such as GPS asset tracking, password compliance / SSO, remote lock / wipe, uploading, viewing, and tracking of edited applications and data, and device application management.

  The platform allows users to perform remote wipes across all sources, delete original files and derivatives, perform remote locks, and detect jailbroken devices and protect against jailbroken devices Can be executed. There is password compliance that uses single sign-on and device settings to monitor network, email settings, and battery life.

  Application management limits applications and pushes essential applications. Application management also includes application monitoring (eg, data uploaded, viewed, etc.) to ensure that all applications are up to date and function properly. The platform further includes GPS tracking and reporting / real-time notification of non-compliant devices.

  Certain embodiments of mobile device management include mobile risk management that provides real-time visibility to all mobile devices connected to relevant resources, including acceptance of employee choice (eg Private Device Utilization (BYOD)) and Achieve a balance with corporate data protection. Mobile security provides anti-virus technology, evolved firewall, and anti-spam features. GPS asset tracking measures and tracks the coordinates of mobile devices at a central console. Remote lock / wiping protects the user's device and its data in case of unauthorized access. Platform device inventory tracking to track files and data uploaded to the device or viewed and edited on the device, for sorting files (eg by classification, recently viewed / edited, etc.) Provides superior visibility to the administrator through the search function.

  Platform device application management distributes applications based on group settings, restricts or allows specified public market apps, and pushes required applications. The platform further alerts and disables access to non-compliant devices, manages private applications, and delivers them to users without submitting them to a public application store. The platform further performs updates to the device. Security policy management in certain embodiments includes push sending of certificates, email server settings, restrictions, and camera settings for devices (Android devices, iOS devices, etc.). The platform lists running processes, installed applications, battery life, and network information. Users can get real-time notifications and alerts to the administration regarding exceptions.

  The platform generates a user report by completing an inspection report of user activity (eg, file upload, file download, application usage, sharing activity, etc.). The platform manages mobile content sharing by pushing corporate documents and sensitive files to mobile devices in a secure manner using data loss prevention, and sends remote updates to files without end user intervention, Configured to delete. Instant recovery and backup provide the ability to restore all files to the device at once if the files are lost or corrupted. The platform wipes the device remotely, while providing policies and applications to the device, putting the application on a black or white list, detecting jailbroken devices, and protecting against jailbroken devices. You can also.

  The platform of an embodiment is a resource that integrates contextual information and provides visibility to various environments (eg, mobile devices, desktops, servers, etc.) to monitor, optimize, and protect data, applications, and devices. Provide utilization and security. The platform has application information tracking capabilities for hardware, software and application utilization, storage resource consumption, RAM and network monitoring, and optimizes storage, data cleanup and archiving.

  Certain embodiments of data deletion and cleanup leverage data topology maps and classifications to identify non-business data. The platform performs defensive data cleanup prior to information management initiatives such as storage / data migration, e-discovery, records management, or compliance requirements, and archives old data that needs to be maintained. Storage optimization optimizes the use of existing storage resources and reduces storage costs (eg, data deduplication). In addition, storage optimization utilizes data topology maps and classifications to monitor and plan future storage demand and track resource consumption (eg, storage, RAM, network monitoring).

  Platform resource utilization and security includes application and information tracking that collects information about how hardware, software, and applications are being used. Users can track the most utilized programs, websites, and file types. The resource consumption report indicates how much the user is using RAM, bandwidth, and storage space. Context-aware optimization software tracks and establishes various usage patterns over time, thus increasing visibility into application usage and data movement, reducing unpredictability . The platform redirects valuable resources to important users, applications and / or programs. Increased visibility enables planned and real-time assignments.

  Platform client security provides anti-virus technology, evolutionary firewall, and anti-malware. Platform activity security monitors abnormal activity on devices and / or clients and alerts administrators. Platform cloud security includes policies, technologies, and controls that protect data and applications in the cloud.

  Deployment and imaging deploys and manages computers from a central location. Android code signing manages proof keys and applications for simple application version updates in Google Play. Device setting management distributes applications / software based on group settings, restricts or permits specified applications / software, and pushes essential programs. The platform warns and disables access to non-compliant devices. The platform manages private applications / software, distributes them to users without exposing them to the outside, and implements updates to devices. Endpoint security ensures that the operating system, web browser, and other applications are up to date before allowing access.

  The platform of an embodiment includes a management console that provides IT administrators with unmatched visibility and control over how business data is accessed, viewed, modified, and shared. FIG. 12 is a diagram illustrating a screen example of the management console according to the embodiment. The online management console includes how active the user is, what device the user is connecting to the service on, what files the user is synchronizing, and who the user is sharing data with, Provide real-time visibility for all aspects of the service. The management console also allows IT administrators to actively manage critical functions of the service, including user permissions, file access restrictions, sharing controls and remote data wipe.

  The management console includes the ability to control various aspects of the file synchronization procedure. Because the platform automatically identifies sensitive documents, IT managers can restrict critical business documents from being synchronized across services and shared internally or externally. In addition, IT administrators can set the maximum size of files that can be synchronized to control storage and bandwidth consumption, or specify file categories that are prohibited from existing on the network (eg music and video). be able to. The IT administrator can also define whether the file can be accessed and downloaded from the user web portal. FIG. 13 is a diagram illustrating a screen example including file synchronization control according to the embodiment.

  The platform provides IT administrators with real-time automated activity monitoring, inspection and reporting that enumerates important system activities by date, time, user, device, file and other factors. To enforce security policies, IT administrators can see the number of invalid logins, password reset attempts, sensitive file sharing events, and other activities. FIG. 14 is a diagram illustrating an example of a screen including security activity monitoring and notification according to the embodiment. In this example, the IT administrator has been alerted to a potential security issue because there have been five invalid login attempts to the IT administrator's account from an unknown location in Cluj, Romania. If a security event occurs in the system, the IT administrator can receive an alert notification through the management console, iOS / Android application, text or email (these are part of the example).

  FIG. 15 is a diagram illustrating an example of a screen including a document audit trail according to the embodiment. The platform records the complete file history of the documents in the service. For each document, the IT administrator can see when the file was added, changed, or deleted. In addition, the IT administrator can determine which device accessed the document and the geographical location of the device. This is an essential feature for shared documents being modified by multiple users across several devices. In addition to creating a document audit trail, if the IT administrator or user wants to revert to a previous version of the file, a complete revision history of each file is also saved.

  FIG. 16 is a diagram showing an example of a screen including file search and classification according to the embodiment. Using automatic data discovery and decomposition functions, IT managers can identify sensitive business files based on deep content search and key term extraction. IT managers can then search and view specific documents containing proprietary information (eg, customer, employee, and accounting information) and set specific restrictions to control this information. . In addition to strictly managing confidential documents, IT administrators can search for files in the organization by file name. The IT administrator can also see the total number of files synchronized in the organization to understand file type details (document, music, video, etc.) or file distribution based on size. This information helps IT managers identify users who may be adding personal content to the service or misusing the solution.

  The platform allows the IT administrator to determine which users can share documents within an organization by setting up shared folders with specific collaborators. Unauthorized users do not have the ability to access shared folders without the permission of the IT administrator or a trusted user. The IT administrator can also control which users are allowed to share one-way web links. In addition, for both web links and shared folders, the IT administrator has a complete history of all sharing activities (adding, changing, or deleting files by user, device, location, etc.). FIG. 17 is a diagram illustrating an example of a screen including controlled user sharing according to the embodiment.

  FIG. 18 is a diagram showing an example of a screen including remote device management and geolocation service according to the embodiment. IT administrators can remotely manage mobile devices and desktops through a management console. For example, if an employee is fired from an organization, the IT administrator remotely removes the user from any sharing activity, disconnects the device from the user's account, and temporarily suspends service for the device be able to. The IT administrator can also perform a remote wipe of a specific device, thereby removing sensitive business content immediately. The platform integrates a geolocation service into the management console to identify the specific location of system events and to monitor the location of mobile devices and computers. This feature is particularly useful if the IT administrator wants to restrict business content from home computers and other private devices.

  The platform allows IT managers to select a storage location for core business files. The IT administrator can choose to store the file in the Memeo data center using another cloud storage provider (eg HP or Amazon) or on-premises (eg NETGEAR) in a server or NAS environment. FIG. 19 is a diagram illustrating an example of a screen including selection of a storage location according to the embodiment. A flexible storage model enhances security for customers by allowing customers to store files locally or using a cloud provider that the customer already uses. In addition, this approach allows IT administrators to utilize existing storage infrastructure, reducing operational costs.

  The embodiments described herein include a system having a platform that includes a processor connected to a plurality of databases. The system includes a grid having a plurality of agents connected to the platform. Each agent of the plurality of agents is a platform agent that runs on the client device. The system has metadata of memory contents accessible from a plurality of client devices corresponding to a plurality of agents. Each agent of each client generates metadata and supplies it to the platform. The platform has metadata instead of content, and the metadata examines the location of the content, generates a goal representing an action for maintaining the state of the content, generates a plurality of tasks corresponding to the goal, Used to assign each task to an agent that can access the content corresponding to that task. Each task is a processing operation instructed by the platform and is performed by the agent on content accessible to the agent.

  The embodiments described herein include the following systems. A grid with processors connected to multiple databases, multiple agents connected to a platform, where each agent of multiple agents is an agent of a platform running on a client device and corresponds to multiple agents Content metadata in memory accessible to a plurality of client devices, each agent of each client device generates and provides metadata to the platform, the platform having metadata instead of content, A goal representing the action for examining the location of the content and maintaining the state of the content is generated, a plurality of tasks corresponding to the goal are generated, and each task can access the content corresponding to the task. Using metadata for assignment to Ento, each task is a process operation instructed by the platform, by the agent, the agent runs accessible content.

  In some embodiments, the platform is a master controller for multiple agents, and the multiple agents work with the platform to be controlled by multiple tasks.

  Each agent of the plurality of agents of an embodiment functions independently of all other agents of the platform and the plurality of agents.

  In some embodiments, the platform receives content metadata instead of receiving content, and the plurality of client devices hosting the plurality of agents have distributed storage devices containing the content.

  The platform of an embodiment uses metadata to maintain a master index of content for multiple agents in multiple databases.

  The metadata of an embodiment includes data regarding the location of the content.

  The metadata of an embodiment includes data regarding the state of the content.

  The metadata of an embodiment includes data regarding the identity of multiple agents.

  The metadata of an embodiment includes data regarding content accessible to each agent.

  The metadata of an embodiment includes information on a plurality of tasks executed by a plurality of agents.

  An embodiment task relates to tracking the location of content across a grid.

  The task of an embodiment relates to managing the storage of content across the grid.

  The task of an embodiment relates to managing the movement of content across the grid.

  The task of an embodiment relates to processing content across the grid.

  Each task of an embodiment has a completion condition for the task.

  In some embodiments, multiple agents collectively monitor content as directed by the platform.

  In some embodiments, multiple agents collectively store content as directed by the platform.

  Agents in certain embodiments collectively transfer content as directed by the platform.

  Agents in certain embodiments collectively perform processing operations on content as directed by the platform.

  Each agent of each client device indexes the contents of memory accessible to the client device.

  The plurality of databases of an embodiment includes an agent database having information about each agent available to the platform and each agent.

  Each agent of an embodiment includes at least one library, which represents an accessible device that the agent includes at least one of read access and write access.

  The plurality of databases of an embodiment includes a library database having a list of libraries corresponding to a plurality of agents, and each library of the grid is represented separately in the library database.

  Each agent of an embodiment can access each library corresponding to the agent.

  The databases of an embodiment include a goal database having a list of goals, each goal in the list of goals is generated by the platform and corresponds to at least one library, and the goal performs an action on the corresponding library content Used to do.

  In one embodiment, the list of goals includes a collection of libraries on which the task is performed.

  Each goal of an embodiment corresponds to at least one library and is used by an agent to perform at least one task on the contents of the corresponding library.

  An embodiment platform uses goals to read multiple libraries of multiple agents.

  The platform of an embodiment uses goals to identify at least one of inconsistencies and differences in content accessible to multiple agents.

  The platform of an embodiment uses the goal to generate at least one task for removing at least one of inconsistencies and differences identified in the content.

  In some embodiments, the platform includes a work database.

  The platform of an embodiment includes a work database corresponding to each agent, which includes tasks performed by the corresponding agent.

  The work database of an embodiment includes a description of each task performed by the corresponding agent and information required by the agent to execute that task.

  The plurality of databases of an embodiment includes a capability database having information on capabilities of a plurality of agents.

  The plurality of databases of an embodiment includes a synchronization database that includes and maintains records of a plurality of agents.

  The synchronization database of an embodiment includes a file status record of content for multiple agents.

  The synchronization database of an embodiment includes an entry for each file, the entry includes a content hash corresponding to the file, the content hash has a hash of a list of blobs representing the contents of the file, and each blob in the list of blobs is a content The fragment is a component of the file.

  The synchronization database of an embodiment includes an entry for each agent, the entry includes a content hash corresponding to the agent's file, the content hash having a hash of a list of blobs representing the contents of the file, and each blob in the list of blobs Has a representation of a fragment of the file in the content, where the fragment is a component of the file.

  An embodiment platform controls the transfer of content between client devices using multiple agents.

  The platform of an embodiment controls content synchronization between client devices using multiple agents, and content synchronization includes synchronizing content in response to content changes.

  The transfer of content in one embodiment includes block level and unordered transfer of content.

  The transfer of content in an embodiment includes transferring a first block of content from a second client device to a first client device, and transferring a second block of content from a third client device to the first client. Transferring to the device.

  In some embodiments, the plurality of tasks includes a scan task.

  The scan task of an embodiment includes an agent recursively monitoring a library corresponding to the agent and reporting changes to the library to the platform.

  The report of an embodiment includes placing the library file containing the changes in a local database of the client device hosting the agent and providing the metadata of the local database to the platform.

  The plurality of tasks of an embodiment includes a write task.

  The write task of an embodiment includes an agent copying at least one blob of a file from a first location to a second location, each of the first location and the second location being connected to a grid. Corresponding to the client device.

  At least one blob of an embodiment has a representation of a fragment of a file in the content, where the fragment is a component of the file.

  The write task of some embodiments has a completion condition.

  The completion condition of an embodiment has at least one of obtaining a blob corresponding to the write task and identifying that the blob that is overwritten during the write task corresponds to the correct file. .

  The tasks of an embodiment include an upload task that includes copying a representation of content from an agent accessible device to a remote storage device.

  In one embodiment, the agent reports to the platform the addition of a file to content that the agent can access, and in response the platform assigns the agent a task to upload the file.

  The agent of an embodiment is responsive to the task to determine if the file exists on the remote storage device and uploads a representation of the file to the remote storage device if it is determined that the file does not exist.

  In one embodiment, the upload task includes an agent copying at least one blob of a file from a device accessible to the agent to a remote storage device.

  At least one blob of an embodiment has a representation of a fragment of a file, where the fragment is a component of the file.

  The tasks of some embodiments include delete tasks.

  A task of an embodiment has multiple phases including at least one of waiting, incomplete, and completion, and the platform tracks the phase of each task of the multiple tasks.

  Each agent of an embodiment maintains the tasks assigned to the agent locally on the client device.

  The agent of an embodiment periodically polls the platform to identify assigned tasks.

  The agent of an embodiment has a hierarchy for finding task information that is required to complete a task and is located on a remote device.

  In one embodiment, the hierarchy includes an agent searching a local database of a client device hosting the agent.

  The hierarchy of an embodiment includes the agent communicating with at least one peer agent of the plurality of agents to find task information.

  An agent of an embodiment has an ID of a peer agent that possesses task information.

  An embodiment hierarchy includes an agent obtaining task information from a remote storage device.

  The agent of an embodiment receives peer agent IDs from the platform in an ordered list and searches for task information according to the ordered list.

  Each agent of an embodiment includes multiple components that execute in parallel.

  The components of an embodiment include a provider component that obtains a task specified for the agent from the platform and stores the obtained task in a local task database of a client device that hosts the agent.

  The components of an embodiment have task execution components.

  A plurality of components of an embodiment is a runner component that monitors a task database, obtains each task from the task database, supplies the obtained task to a task execution component, and instructs the obtained task to have an incomplete status including.

  In one embodiment, the task execution component executes the task and reports the status of task execution to the runner component.

  In one embodiment, the runner component reports status to a task database. The components of an embodiment include an update component that monitors the task database for tasks with completion status and reports status information of completed tasks to the platform.

  The platform of an embodiment updates multiple databases in response to status information.

  The metadata of an embodiment is generated by a plurality of agents, and the metadata generated for an agent corresponds to content accessible to the agent.

  In one embodiment, the agent generates metadata by scanning the contents of each file accessible to the agent.

  An agent of an embodiment generates metadata by dividing the contents of a file into a plurality of fragments, each fragment having a variable size component of the file.

  In some embodiments, the variable size fragment is between a pre-specified minimum and maximum length.

  An agent of an embodiment generates metadata by generating a plurality of blobs that represent a plurality of fragments, each blob representing a fragment.

  The agent of an embodiment generates, for each byte of content, a plurality of blobs using a data fingerprinting algorithm that includes performing a hash algorithm on the content component, where the hash algorithm identifies a specified pattern of data Set to do.

  The generation of a blob in one embodiment includes generating a blob description that includes a hash value at a location where the fragment represented by the blob is separated from the rest of the file.

  In some embodiments, the generation of blobs comprises generating an offset value based on the fragmentation position of the fragment represented by the blob.

  The blob generation of an embodiment includes generating a blob complete content hash, where the blob complete content hash is a blob identifier.

  The generation of blobs in one embodiment includes generating a list of blobs that represent the contents of the file.

  In some embodiments, the generation of blobs includes generating a content hash having a hash of a list of blobs representing the content of the file, where the content hash is an identifier for the file.

  In some embodiments, the generation of the blob includes generating a name hash having a hash of the file name corresponding to the file, where the content hash is an identifier of the file.

  Generating a blob in an embodiment includes generating a file hash having a hash of a combination of a content hash and a name hash.

  The generation of a blob in an embodiment includes generating a path hash having a file name corresponding to the contents of the file and a hash of the file path.

  Generating a blob of an embodiment includes generating a metadata hash having a file metadata hash of the file.

  The platform of an embodiment generates a record for the file, the record having a file blob hash, content hash, file hash, path hash, and metadata hash.

  In some embodiments, the plurality of databases includes a library database, the library database having records.

  Generating the blob of an embodiment includes generating the size of the blob.

  The file of an embodiment is described as a list of blobs that make up the file.

  Each agent of an embodiment stores blobs locally on the client device hosting the agent and forwards blobs not previously reported to the platform's central storage.

  The embodiments described herein include a system having a platform that includes a processor connected to a plurality of databases. The system includes a grid having a plurality of agents connected to the platform. Each agent of the plurality of agents is a platform agent that runs on the client device. The system has metadata of memory contents accessible to a plurality of client devices corresponding to a plurality of agents. The agent generates metadata corresponding to the content by hashing a plurality of fragments of the content, and supplies the metadata to the platform. The platform uses metadata instead of content to generate multiple tasks and assign them to multiple agents, including tasks that control at least one of content storage, transfer, and processing. A task is a processing operation that is performed on content accessible to an agent responsible for the task.

  The embodiments described herein include the following systems. A grid with processors connected to multiple databases, multiple agents connected to a platform, where each agent of multiple agents is an agent of a platform running on a client device and corresponds to multiple agents Content metadata in memory accessible by a plurality of client devices, and the agent generates metadata corresponding to the content by hashing the plurality of fragments of the content, and supplies the platform to the platform, Use metadata instead of content to generate multiple tasks and assign them to multiple agents, including tasks that control at least one of content storage, transfer, and processing. Is a processing operation which the agent responsible for the task to run at a content accessible.

  The embodiments described herein include a system having a platform that includes a processor connected to a plurality of databases. The system includes a grid having a plurality of agents connected to the platform. Each agent of the plurality of agents is a platform agent that runs on a client device. Each agent of each client device provides the platform with metadata of the contents of memory accessible to the client device. The platform examines the location of the content, generates a goal that represents the action to maintain the state of the content, generates multiple tasks corresponding to the goal, and each task corresponds to the task Used to assign to agents that can access A task is a processing operation instructed by the platform and is executed by an agent on content accessible to that agent. The metadata supplied by the plurality of agents includes information on a plurality of tasks executed by the plurality of agents.

  The embodiments described herein include the following systems. A platform having a processor connected to a plurality of databases, a grid having a plurality of agents connected to the platform, wherein each agent of the plurality of agents is an agent of a platform running on a client device, and each client device Each of the corresponding agents provides memory content metadata accessible to the client device to the platform, and the platform is responsible for examining the location of the content and representing actions to maintain the state of the content. Is used to generate multiple tasks corresponding to the goal and assign each task to an agent that can access the content corresponding to that task. A process operation instructed by beam, by the agent, the agent is executed content accessible, metadata provided by a plurality of agents, including information of a plurality of tasks performed by multiple agents.

  The embodiments described herein include a system having a platform that includes a processor connected to a plurality of databases. The system includes a grid having a plurality of agents connected to the platform. Each agent of the plurality of agents is a platform agent that runs on the client device. The system has metadata of memory contents accessible to a plurality of client devices corresponding to a plurality of agents. Each agent generates metadata corresponding to content accessible by the agent and supplies it to the platform. Each agent generates metadata by hashing multiple fragments of content to generate multiple blobs that represent multiple fragments. The platform uses metadata instead of content to generate multiple tasks and assign them to multiple agents. A task is a processing operation that is performed on content accessible to an agent responsible for the task. The plurality of tasks include a task that is at least one of content monitoring, storage, transfer, and processing.

  The described embodiment includes a processor having a plurality of databases connected to a platform, a grid having a plurality of agents connected to a platform, wherein each agent of the plurality of agents is an agent of a platform running on a client device. And metadata of memory contents accessible by a plurality of client devices corresponding to a plurality of agents, and each agent generates metadata corresponding to the contents accessible by the agent and supplies it to the platform Each agent generates metadata by hashing multiple fragments of content to generate multiple blobs representing multiple fragments, and the platform generates multiple tasks Metadata is used instead of content to assign to agents, tasks are processing operations performed on content accessible to the agent responsible for the task, and multiple tasks can be monitored, stored, transferred, And a system including a task that is at least one of the processes.

  Embodiments described herein include a method that includes connecting a platform having a processor to a plurality of databases. The method includes connecting a plurality of agents to the platform to form a grid. Each agent of the plurality of agents is a platform agent that runs on the client device. The method includes generating metadata at each agent and providing the metadata to the platform instead of the content. The metadata corresponds to the contents of a memory that can be accessed by a plurality of client devices corresponding to a plurality of agents. The method includes using the metadata to locate the content. The method includes using the metadata to generate a goal that represents an action for maintaining the state of the content. The method includes generating a plurality of tasks corresponding to the goal using the metadata. The method includes assigning each task to an agent that can access content corresponding to the task. Each task is a processing operation instructed by the platform and is performed by the agent on content accessible to the agent.

  Embodiments described herein include connecting a platform having a processor to a plurality of databases, connecting a plurality of agents to the platform to form a grid, wherein each agent of the plurality of agents is a client A platform agent running on a device that generates metadata on each agent and supplies it to the platform instead of content, where the metadata is multiple clients corresponding to multiple agents Corresponding to the contents of the memory accessible to the device, using the metadata to locate the content, using the metadata to generate a goal representing the action to maintain the content state, Go with the data Generating a plurality of tasks corresponding to the task and assigning each task to an agent accessible to the content corresponding to the task, wherein each task is a processing operation commanded by the platform; Including a method performed by an agent on content accessible to the agent.

  The method includes the platform acting as a master controller for multiple agents, wherein the multiple agents cooperate with the platform such that they are controlled by multiple tasks.

  The method includes that each agent of the plurality of agents functions independently of the platform and all other agents of the plurality of agents.

  The method has the platform receiving content metadata instead of receiving content, and the plurality of client devices hosting the plurality of agents have distributed storage devices containing the content.

  The method comprises the platform using the metadata to maintain a master index of the content of multiple agents in multiple databases.

  The metadata of an embodiment includes data regarding the location of the content.

  The metadata of an embodiment includes data regarding the state of the content.

  The metadata of an embodiment includes data regarding the identity of multiple agents.

  The metadata of an embodiment includes data regarding content accessible to each agent.

  The metadata of an embodiment includes information on a plurality of tasks executed by a plurality of agents.

  An embodiment task relates to tracking the location of content across a grid.

  The task of an embodiment relates to managing the storage of content across the grid.

  The task of an embodiment relates to managing the movement of content across the grid.

  The task of an embodiment relates to processing content across the grid.

  Each task of an embodiment has a condition for completion of the task. The method includes collectively monitoring content such that multiple agents are commanded to the platform.

  The method includes storing content collectively so that multiple agents are instructed to the platform.

  The method includes transferring content collectively such that multiple agents are instructed to the platform.

  The method includes collectively performing processing operations on the content such that multiple agents are instructed to the platform.

  The method includes that each agent of each client device indexes the contents of memory accessible to the client device.

  The plurality of databases of an embodiment includes an agent database having information about each agent available to the platform and each agent.

  Each agent of an embodiment includes at least one library, which represents an accessible device that the agent includes at least one of read access and write access.

  The plurality of databases of an embodiment includes a library database having a list of libraries corresponding to a plurality of agents, and each library of the grid is represented separately in the library database.

  The method includes that each agent has access to each library corresponding to the agent.

  The databases of an embodiment include a goal database having a list of goals, each goal in the list of goals is generated by the platform and corresponds to at least one library, and the goal performs an action on the corresponding library content Used to do.

  The method includes generating a list of goals to include a collection of libraries on which the task is performed.

  Each goal of an embodiment corresponds to at least one library and is used to perform at least one task on the contents of the corresponding library.

  The method includes using a goal for the platform to read multiple libraries of multiple agents.

  The method includes the platform using a goal to identify at least one of inconsistencies and differences in content accessible to multiple agents.

  The method includes the platform using a goal to generate at least one task to remove at least one of inconsistencies and differences identified in the content.

  The plurality of databases of an embodiment includes a work database.

  The method includes the platform generating a work database corresponding to each agent, the work database including tasks performed by the corresponding agent.

  The work database of an embodiment includes a description of each task performed by the corresponding agent and information required by the agent to execute that task.

  The plurality of databases of an embodiment includes a capability database having information on capabilities of a plurality of agents.

  The plurality of databases of an embodiment includes a synchronization database that includes and maintains records of a plurality of agents.

  The synchronization database of an embodiment includes a file status record of content for multiple agents.

  The synchronization database of an embodiment includes an entry for each file, the entry includes a content hash corresponding to the file, the content hash has a hash of a list of blobs representing the contents of the file, and each blob in the list of blobs is a content The fragment is a component of the file.

  The synchronization database of an embodiment includes an entry for each agent, the entry includes a content hash corresponding to the agent's file, the content hash having a hash of a list of blobs representing the contents of the file, and each blob in the list of blobs Has a representation of a fragment of the file in the content, where the fragment is a component of the file.

  The method includes a platform controlling the transfer of content between client devices using a plurality of agents.

  The method includes controlling, using a plurality of agents, the platform to synchronize content between client devices, and synchronizing the content includes synchronizing the content in response to content changes.

  Content transfer includes block-level and unordered transfer of content.

  The transfer of content includes transferring a first block of content from the second client device to the first client device and transferring a second block of content from the third client device to the first client device. Have.

  In some embodiments, the plurality of tasks includes a scan task.

  The scan task of an embodiment includes an agent recursively monitoring a library corresponding to the agent and reporting changes to the library to the platform.

  Reporting comprises placing the library file containing the changes in the local database of the client device hosting the agent and providing the local database metadata to the platform.

  The plurality of tasks of an embodiment includes a write task.

  The write task of an embodiment includes an agent copying at least one blob of a file from a first location to a second location, each of the first location and the second location being connected to a grid. Corresponding to the client device.

  At least one blob of an embodiment has a representation of a fragment of a file in the content, where the fragment is a component of the file.

  The write task of some embodiments has a completion condition.

  The completion condition of an embodiment has at least one of obtaining a blob corresponding to the write task and identifying that the blob that is overwritten during the write task corresponds to the correct file.

  The tasks of an embodiment have an upload task that includes copying a representation of a file of content from a device accessible to an agent to a remote storage device.

  The method includes the agent reporting to the platform the addition of a file to content accessible to the agent and responsively assigning a task to the agent for the platform to upload the file.

  In response to the task, the method includes determining whether the file exists on the remote storage device and uploading a representation of the file to the remote storage device if it is determined that the file does not exist.

  In one embodiment, the upload task includes an agent copying at least one blob of a file from a device accessible to the agent to a remote storage device.

  At least one blob of an embodiment has a representation of a fragment of a file, where the fragment is a component of the file.

  The tasks of some embodiments include delete tasks.

  A task of an embodiment has multiple phases including at least one of waiting, incomplete, and completion, and the platform tracks the phase of each task of the multiple tasks.

  The method includes each agent maintaining a task assigned to the agent locally at the client device.

  The method includes the agent periodically polling the platform to identify assigned tasks.

  The agent of an embodiment has a hierarchy for finding task information that is required to complete a task and is located on a remote device.

  In one embodiment, the hierarchy includes an agent searching a local database of a client device hosting the agent.

  The hierarchy of an embodiment includes the agent communicating with at least one peer agent of the plurality of agents to find task information.

  An agent of an embodiment has an ID of a peer agent that possesses task information.

  An embodiment hierarchy includes an agent obtaining task information from a remote storage device.

  The method includes an agent receiving peer agent IDs from a platform in an ordered list and searching task information according to the ordered list.

  Each agent of an embodiment includes multiple components that execute in parallel.

  The components of an embodiment include a provider component that obtains a task specified for the agent from the platform and stores the obtained task in a local task database of a client device that hosts the agent.

  The components of an embodiment have task execution components.

  A plurality of components of an embodiment is a runner component that monitors a task database, obtains each task from the task database, supplies the obtained task to a task execution component, and instructs the obtained task to have an incomplete status including.

  The method includes the task execution component executing the task and reporting the task execution status to the runner component.

  The method includes a runner component reporting status to a task database.

  The components of an embodiment include an update component that monitors the task database for tasks with completion status and reports status information of completed tasks to the platform.

  The method includes the platform updating a plurality of databases in response to the status information.

  The method includes a plurality of agents generating metadata, wherein the metadata generated by an agent corresponds to content accessible to that agent.

  The method includes the agent generating metadata by scanning the contents of each file accessible to the agent.

  The method includes generating metadata by splitting the contents of a file into a plurality of fragments, each fragment having a variable size component of the file.

  In some embodiments, the variable size fragment is between a pre-specified minimum and maximum length.

  The method includes the agent generating metadata by generating a plurality of blobs that represent a plurality of fragments, each blob representing a fragment.

  The method includes generating a plurality of blobs using a data fingerprinting algorithm, wherein the agent has to perform a hash algorithm on a component of the content for each byte of content, the hash algorithm being specified for the data It is set to specify the pattern.

  The generation of a blob in one embodiment includes generating a blob description that includes a hash value at a location where the fragment represented by the blob is separated from the rest of the file.

  In some embodiments, the generation of blobs comprises generating an offset value based on the fragmentation position of the fragment represented by the blob.

  The blob generation of an embodiment includes generating a blob complete content hash, where the blob complete content hash is a blob identifier.

  The generation of blobs in one embodiment includes generating a list of blobs that represent the contents of the file.

  In some embodiments, the generation of blobs includes generating a content hash having a hash of a list of blobs representing the content of the file, where the content hash is an identifier for the file.

  In some embodiments, the generation of the blob includes generating a name hash having a hash of the file name corresponding to the file, where the content hash is an identifier of the file.

  Generating a blob in an embodiment includes generating a file hash having a hash of a combination of a content hash and a name hash.

  The generation of a blob in an embodiment includes generating a path hash having a file name corresponding to the contents of the file and a hash of the file path.

  Generating a blob of an embodiment includes generating a metadata hash having a file metadata hash of the file.

  The method includes the platform generating a record for the file, the record having a file blob hash, content hash, file hash, path hash, and metadata hash.

  In some embodiments, the plurality of databases includes a library database, the library database having records.

  Generating the blob of an embodiment includes generating the size of the blob.

  The file of an embodiment is described as a list of blobs that make up the file.

  The method includes each agent storing a blob locally on the client device hosting the agent and transferring a blob that has not been reported to the central storage of the platform.

  The platform has a policy database and a rules engine.

  The method includes analyzing each file of content.

  The method comprises defining each file of content as at least one of business content and personal content.

  The method includes determining a security level for each file of content for operational content.

  The method includes scanning each file of content to extract key terms.

  The method includes comparing key terms across companies and generating a list of company specific key terms.

  Determining the confidentiality level includes identifying a specific key term among the key terms.

  Generating the metadata includes generating the metadata to include the confidentiality level of the corresponding content.

  The method includes controlling access to each file of content based on a security level.

  Controlling access includes controlling browsing of each file of content.

  Controlling access includes controlling downloading of each file of content.

  Controlling access includes controlling access to each file of content.

  Controlling access includes controlling copying of each file of content.

  Controlling access includes controlling the distribution of each file of content.

  The method includes clarifying each file of content based on inferences from the content.

  The method includes clarifying each file of content based on the syntax of the content.

  The method includes identifying operational content of content across at least one of multiple locations and multiple agents, wherein identifying the operational content includes document name, document size, proximity, keyword comparison, and Using at least one of the hash comparisons.

  The method comprises controlling collaboration between multiple devices accessing a file of content.

  The method includes controlling editing of content between multiple devices that simultaneously edit the file of content.

  The method includes tracking activity related to files of content.

  The method includes tracking a modification history of the content file.

  The method has mobile device management that includes monitoring and controlling access to content through the mobile device.

  Mobile device management includes tracking the location of mobile devices.

  Mobile device management includes controlling access to content based on location.

  Mobile device management includes tracking mobile device data.

  The method includes connecting a console to the platform. The console includes automated activity monitoring, inspection and reporting related to the content.

  The method includes controlling activity related to the content through the console. The method includes controlling the location through the console.

  Controlling activity controls browsing each file of content, controlling downloading each file of content, controlling access to each file of content, each file of content And at least one of controlling the distribution of each file of the content.

  Embodiments described herein include a method that includes connecting a platform having a processor to a plurality of databases. The method includes forming a grid having a plurality of agents connected to the platform. Each agent is a platform agent that runs on the client device. The method includes generating metadata at each agent by hashing multiple fragments of content accessible to the agent. The metadata corresponds to the contents of a memory that can be accessed by a plurality of client devices corresponding to a plurality of agents. The method includes providing metadata to the platform instead of content. The method includes generating a plurality of tasks using the metadata, including a task that controls at least one of content storage, transfer, and processing. A task is a processing operation that is performed on content accessible to an agent responsible for the task. The method includes assigning multiple tasks to multiple agents.

  Embodiments described herein connect a platform having a processor to a plurality of databases and form a grid having a plurality of agents connected to the platform, where each agent is on a client device. An agent for a running platform, where each agent generates metadata by hashing multiple fragments of content accessible to the agent, where the metadata is a plurality of corresponding multiple agents A plurality of tasks that correspond to the contents of the memory accessible to the client device, including providing the metadata to the platform instead of the contents and controlling at least one of the storage, transfer, and processing of the contents A task is generated using metadata, and a task is a processing operation performed on content accessible to an agent responsible for the task, and a plurality of tasks are assigned to a plurality of agents. Including a method.

  Embodiments described herein include a method that includes connecting a platform having a processor to a plurality of databases. The method includes forming a grid having a plurality of agents connected to the platform. Each agent is a platform agent that runs on the client device. The method includes that each agent of each client device provides the platform with metadata of the contents of memory accessible to the client device. The method includes using the metadata to locate the content.

  The method includes using the metadata to generate a goal that represents an action for maintaining the state of the content. The method includes generating a plurality of tasks corresponding to the goal using the metadata. The method includes assigning each task to an agent that can access content corresponding to the task. A task is a processing operation instructed by the platform and is executed by an agent on content accessible to that agent. The metadata supplied by the plurality of agents includes information on a plurality of tasks executed by the plurality of agents.

  Embodiments described herein connect a platform having a processor to a plurality of databases and form a grid having a plurality of agents connected to the platform, where each agent is on a client device. A running platform agent, where each agent on each client device supplies the platform with content metadata in memory accessible to the client device, uses the metadata to locate the content, Use data to generate goals that represent actions to maintain the state of content, use metadata to generate multiple tasks that correspond to goals, and each task to a task Access content A task is a processing operation instructed by the platform, performed by the agent on content accessible to the agent, and the metadata provided by multiple agents is multiple Including a method comprising information of a plurality of tasks performed by a plurality of agents.

  Embodiments described herein include a method that includes establishing a coupling between a platform having a processor and a plurality of databases. The method includes connecting a plurality of agents to the platform to form a grid. Each agent of the plurality of agents is a platform agent that runs on the client device. The method includes generating, at each agent, metadata representing the contents of memory accessible to a plurality of client devices corresponding to the plurality of agents. Generating includes hashing the plurality of fragments of content to generate a plurality of blobs that represent the plurality of fragments. The method includes providing metadata to the platform instead of content. The method includes generating a plurality of tasks using the metadata. A task is a processing operation that is performed on content accessible to an agent responsible for the task. The plurality of tasks include a task that is at least one of content monitoring, storage, transfer, and processing. The method includes assigning multiple tasks to multiple agents.

  Embodiments described herein include establishing a coupling between a platform having a processor and a plurality of databases, connecting a plurality of agents to the platform to form a grid, and Each agent is a platform agent that runs on a client device, where each agent generates metadata representing the contents of memory accessible to multiple client devices corresponding to multiple agents, where Generating includes hashing a plurality of fragments of content to generate a plurality of blobs representing the plurality of fragments, supplying metadata to the platform instead of the content, and using the metadata to of Task, where a task is a processing action performed on content accessible to the agent responsible for that task, and multiple tasks are responsible for content monitoring, storage, transfer and processing Including a task that is at least one and assigning a plurality of tasks to a plurality of agents.

  As noted above, suitable computer networks for use with the embodiments described herein are a local area network (LAN), a wide area network (WAN), the Internet, or the World Wide Web, the public Internet, the private Internet, a private computer. Includes other connection services and network derivatives such as networks, public networks, mobile networks, cellular networks, value added networks, and the like. A computing device coupled or connected to the network can be a personal computer, workstation, server, minicomputer, mainframe computer, laptop computer, mobile computer, palmtop computer, handheld computer, mobile phone, television set top box, or It can be any device controlled by the microprocessor that allows access to the network, including terminal devices such as combinations thereof. The computer network may include one or more of multiple LANs, multiple WANs, multiple Internets, and multiple computers. The computer can function as a server, a client, or a combination thereof.

  The components described herein can be components of a single system, multiple systems, and / or geographically separated systems. The components described herein can also be subsystems or subcomponents of a single system, multiple systems, and / or geographically separated systems. The components described herein may be connected to one or more other components (not shown) of the host system or systems connected to the host system.

  The components described herein may include a processing system or may operate under and / or in cooperation with a processing system. A processing system includes any group of devices or computing devices that use a processor or components of a processing system or device that operate together, as is known in the art. For example, the processing system can include one or more of a portable computer, a portable communication device operating within a communication network, and / or a network server. The portable computer may be any of a plurality of devices and / or combinations of devices selected from, but not limited to, personal computers, personal digital assistants (PDAs), portable computing devices, and portable communication devices. The processing system may include components within a larger computer system.

  The processing system of an embodiment includes at least one processor and at least one memory device or subsystem. The processing system may also include or be connected to at least one database. The term “processor” as generally used herein refers to any logical processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), and the like.

  Unless the context clearly dictates, throughout the specification and claims, the words “having”, “having” and similar words have a comprehensive meaning, as opposed to an exclusive or exhaustive meaning. That is, it is understood to mean "including but not limited to". Words using the singular or plural form also include the plural or singular number, respectively. Furthermore, “here”, “below”, “above”, “below” and similar meaning words when used in this application refer to the entire application and do not refer to any particular part of the application. The word "or" when used in connection with a list of two or more items covers all of the following interpretations of that word: any of the items in the list, all of the items in the list, and the list Any combination of items in

  The descriptions of embodiments and corresponding systems and methods are not intended to be exhaustive or to limit the systems and methods to the precise forms disclosed. Specific embodiments of the systems and methods, and examples of systems and methods are described for purposes of illustration, and various equivalents within the scope of the systems and methods will be appreciated by those skilled in the relevant art. Variations are possible. The teachings of the systems and methods provided herein are applicable not only to the systems and methods described above, but also to other systems and methods.

  The elements and behaviors of the various embodiments described above can be combined to provide another embodiment. These and other changes can be applied to the systems and methods in light of the above description.

Claims (125)

Connecting a platform having a processor to multiple databases;
Connecting a plurality of agents to the platform to form a grid, wherein each agent of the plurality of agents is an agent of the platform running on a client device;
Generating metadata at each agent and supplying the metadata to the platform instead of content, wherein the metadata is stored in a memory accessible to a plurality of client devices corresponding to the plurality of agents; Corresponding to the content,
Using the metadata to locate the content;
Using the metadata to generate a goal representing an action for maintaining the state of the content;
Generating a plurality of tasks corresponding to the goal using the metadata;
Assigning each task to an agent accessible to the content corresponding to the task, wherein each task is a processing operation instructed by the platform, to the content accessible to the agent by the agent. The method that is performed.   The method of claim 1, wherein the platform comprises acting as a master controller for the plurality of agents, wherein the plurality of agents cooperate with the platform to be controlled by the plurality of tasks.   The method of claim 1, comprising each agent of the plurality of agents functioning independently of the platform and all other agents of the plurality of agents.   The platform of claim 1, comprising receiving the metadata of the content instead of receiving the content, and wherein a plurality of client devices hosting the plurality of agents have a distributed storage device containing the content. The method described.   The method of claim 1, wherein the platform comprises using the metadata to maintain a master index of the content of the plurality of agents in the plurality of databases.   The method of claim 1, wherein the metadata comprises data regarding the location of the content.   The method of claim 1, wherein the metadata comprises data relating to the state of the content.   The method of claim 1, wherein the metadata comprises data regarding the identity of the plurality of agents.   The method of claim 1, wherein the metadata comprises data regarding the content accessible to each agent.   The method of claim 1, wherein the metadata comprises information of the plurality of tasks executed by the plurality of agents.   The method of claim 1, wherein the task relates to tracking the location of the content across the grid.   The method of claim 1, wherein the task relates to managing storage of the content across the grid.   The method of claim 1, wherein the task relates to managing movement of the content across the grid.   The method of claim 1, wherein the task relates to processing the content across the grid.   The method of claim 1, wherein each task has a completion condition for the task.   The method of claim 1, comprising collectively monitoring the content such that the plurality of agents are instructed to the platform.   The method of claim 1, comprising storing the content collectively such that the plurality of agents are instructed to the platform.   The method of claim 1, comprising transferring the content collectively such that the plurality of agents are instructed to the platform.   The method of claim 1, comprising collectively performing processing operations on the content such that the plurality of agents are instructed to the platform.   The method of claim 1, wherein each agent of each client device includes indexing content of memory accessible to the client device.   The method of claim 1, wherein the plurality of databases includes an agent database having agents available to the platform and information for each agent.   The method of claim 1, wherein each agent includes at least one library, the library representing an accessible device that the agent includes at least one of read access and write access.   23. The method of claim 22, wherein the plurality of databases includes a library database having a list of libraries corresponding to the plurality of agents, and each library of the grid is represented separately in the library database.   24. The method of claim 23, wherein each agent has access to each library corresponding to the agent.   The plurality of databases includes a goal database having the list of goals, each goal in the list of goals is generated by the platform and corresponds to at least one library, and the goal operates on the corresponding library content. 23. The method of claim 22, used to perform.   26. The method of claim 25, comprising generating the list of goals to include a collection of libraries on which tasks are performed.   26. The method of claim 25, wherein each goal corresponds to at least one library and is used to perform at least one task on the contents of the corresponding library.   26. The method of claim 25, wherein the platform uses the goal to read a plurality of libraries of the plurality of agents.   26. The method of claim 25, wherein the platform comprises using the goal to identify at least one of inconsistencies and differences in the content accessible to the plurality of agents.   30. The method of claim 29, wherein the platform comprises using the goal to generate at least one task to remove at least one of inconsistencies and differences identified in the content.   The method of claim 1, wherein the plurality of databases comprises a work database.   32. The method of claim 31, wherein the platform comprises generating a work database corresponding to each agent, the work database including tasks performed by the corresponding agent.   The method of claim 32, wherein the work database includes a description of each task performed by a corresponding agent and information needed by the agent to perform the task.   The method of claim 1, wherein the plurality of databases includes a capability database having capability information of the plurality of agents.   The method of claim 1, wherein the plurality of databases includes a synchronization database that includes and maintains records of the plurality of agents.   36. The method of claim 35, wherein the synchronization database includes a file status record of content of the plurality of agents.   The synchronization database includes an entry for each file, the entry includes a content hash corresponding to the file, the content hash having a hash of a list of blobs representing the content of the file, and each of the lists of blobs 38. The method of claim 36, wherein a blob has a hash of a fragment of a file in the content, and the fragment is a component of the file.   The synchronization database includes an entry for each file, the entry includes a content hash corresponding to the agent's file, the content hash having a hash of a list of blobs representing the content of the file, the list of blobs 38. The method of claim 36, wherein each blob of has a hash of a fragment of a file in the content, the fragment being a component of the file.   The method of claim 1, wherein the platform includes controlling the transfer of content between client devices using the plurality of agents.   The platform includes controlling the synchronization of the content between client devices using the plurality of agents, wherein the synchronization synchronizes the content in response to changes in the content; 40. The method of claim 39, comprising:   40. The method of claim 39, wherein the transfer of content comprises a block level and unordered transfer of content.   The transfer of content includes transferring a first block of content from a second client device to a first client device, and transferring a second block of content from a third client device to a first client device. 42. The method of claim 41, further comprising:   The method of claim 1, wherein the plurality of tasks includes a scan task.   44. The method of claim 43, wherein the scan task includes the agent recursively monitoring a library corresponding to the agent and reporting changes to the library to the platform.   The reporting comprises placing the library file containing changes in a local database of the client device hosting the agent and providing the metadata of the local database to the platform. 45. The method of claim 44.   The method of claim 1, wherein the plurality of tasks includes a write task.   The write task includes the agent copying at least one blob of a file from a first location to a second location, each of the first location and the second location connected to the grid 48. The method of claim 46, corresponding to a designated client device.   48. The method of claim 47, wherein the at least one blob has a hash of a fragment of a file in the content, and the fragment is a component of the file.   The method of claim 46, wherein the write task has a condition for completion.   The completion condition includes at least one of obtaining the blob corresponding to the write task and identifying that the blob to be overwritten during the write task corresponds to a correct file. 50. The method of claim 49, comprising:   The method of claim 1, wherein the plurality of tasks comprises an upload task that includes copying the content file from a device accessible to the agent to a remote storage device.   The agent reporting to the platform the addition of a file to the content accessible to the agent and responsively assigning a task to the agent for the platform to upload the file; 52. The method of claim 51.   In response to the task, the agent determines whether the file exists on the remote storage device and uploads the file to the remote storage device if it is determined that the file does not exist; 53. The method of claim 52.   52. The method of claim 51, wherein the upload task includes the agent copying at least one blob of a file from the device accessible to the agent to the remote storage device.   55. The method of claim 54, wherein the at least one blob has a hash of a fragment of the file, and the fragment is a component of the file.   The method of claim 1, wherein the plurality of tasks includes a delete task.   The method of claim 1, wherein a task has a plurality of phases including at least one of waiting, incomplete, and completion, and wherein the platform tracks the phase of each task of the plurality of tasks.   The method of claim 1, wherein each agent comprises maintaining a task assigned to the agent locally at the client device.   59. The method of claim 58, wherein the agent comprises polling the platform periodically to identify assigned tasks.   The method of claim 1, wherein the agent has a hierarchy for finding task information required to complete the task and located at the remote device.   61. The method of claim 60, wherein the hierarchy comprises the agent searching a local database of the client device that hosts the agent.   62. The method of claim 61, wherein the hierarchy comprises the agent communicating with at least one peer agent of the plurality of agents to find the task information.   64. The method of claim 62, wherein the agent has an ID of a peer agent possessing the task information.   64. The method of claim 62, wherein the hierarchy comprises the agent obtaining the task information from a remote storage device.   68. The method of claim 64, wherein the agent comprises receiving peer agent IDs from the platform in an ordered list and searching the task information according to the ordered list.   The method of claim 1, wherein each agent includes a plurality of components executing in parallel.   68. The plurality of components includes a provider component that obtains a task specified for the agent from the platform and stores the obtained task in a task database local to the client device hosting the agent. the method of.   68. The method of claim 67, wherein the plurality of components comprises a task execution component.   The plurality of components monitor the task database, acquire each task from the task database, supply the acquired task to the task execution component, and instruct the acquired task to have an incomplete status 69. The method of claim 68, comprising a component.   70. The method of claim 69, wherein the task execution component executes the task and reports a status of task execution to the runner component.   71. The method of claim 70, wherein the runner component comprises reporting the status to the task database.   72. The method of claim 71, wherein the plurality of components includes an update component that monitors the task database for the task having a completion status and reports status information of completed tasks to the platform.   The method of claim 72, wherein the platform comprises updating the plurality of databases in response to the status information.   The method of claim 1, wherein the plurality of agents comprises generating the metadata, wherein the metadata generated for an agent corresponds to the content accessible to the agent.   75. The method of claim 74, wherein the agent comprises generating the metadata by scanning the contents of each file accessible to the agent.   76. The method of claim 75, wherein the agent comprises generating the metadata by dividing the content of the file into a plurality of fragments, each fragment having a variable size component of the file.   77. The method of claim 76, wherein the variable size fragment is between a pre-specified minimum and maximum length.   77. The method of claim 76, wherein the agent comprises generating the metadata by generating a plurality of blobs representing the plurality of fragments, each blob representing a fragment.   Said agent generating said plurality of blobs using a blob hash, said blob hash comprising: performing a hash algorithm on said content component for each byte of said content; 77. The method of claim 76, wherein is set to identify a specified pattern of data.   80. Generating the blob comprises generating a description of the blob that includes a hash value at a location where the fragment represented by the blob is separated from the rest of the file. the method of.   81. The method of claim 80, wherein generating the blob includes generating an offset value based on a fragmentation position of the fragment represented by the blob.   82. The method of claim 81, wherein generating the blob comprises generating a hash of the complete content of the blob, wherein the hash of the complete content of the blob is an identifier of the blob.   83. The method of claim 82, wherein generating the blob includes generating a list of blobs that represent the contents of the file.   84. The method of claim 83, wherein generating the blob includes generating a content hash having a hash of the list of blobs representing the content of the file, the content hash being an identifier of the file. .   85. The method of claim 84, wherein generating the blob includes generating a name hash having a file name hash corresponding to the file.   86. The method of claim 85, wherein generating the blob includes generating a file hash having a hash of a combination of the content hash and a name hash.   87. The method of claim 86, wherein generating the blob includes generating a path hash having a hash of the file name and file path corresponding to the content of the file.   90. The method of claim 87, wherein generating the blob includes generating a metadata hash having a file metadata hash of the file.   90. The platform of claim 88, comprising generating a record for the file, the record comprising a blob hash of the file, the content hash, the file hash, the path hash, and the metadata hash. Method.   90. The method of claim 89, wherein the plurality of databases includes a library database, the library database having records.   80. The method of claim 79, wherein generating the blob includes generating a size of the blob.   80. The method of claim 79, wherein the file is described as a list of blobs that make up the file.   80. The method of claim 79, wherein each agent comprises storing a blob locally on the client device hosting the agent and transferring an unreported blob to the central storage of the platform.   The method of claim 1, wherein the platform comprises a policy database and a rules engine.   The method of claim 1, comprising analyzing each file of the content.   96. The method of claim 95, comprising defining each file of the content as at least one of business content and personal content.   99. The method of claim 96, comprising determining a security level for each file of the content for the operational content.   99. The method of claim 96, comprising scanning each file of the content to extract key terms.   99. The method of claim 98, comprising comparing the key terms across companies to generate a list of company specific key terms.   99. The method of claim 98, wherein determining the security level comprises identifying a particular key term among the key terms.   98. The method of claim 97, wherein generating the metadata includes generating the metadata to include the sensitivity level of corresponding content.   98. The method of claim 97, comprising controlling access to each file of the content based on the security level.   103. The method of claim 102, wherein controlling the access comprises controlling browsing of each file of the content.   103. The method of claim 102, wherein controlling the access comprises controlling downloading each file of the content.   104. The method of claim 102, wherein controlling the access comprises controlling access to each file of the content.   104. The method of claim 102, wherein controlling the access comprises controlling copying each file of the content.   103. The method of claim 102, wherein controlling the access comprises controlling delivering each file of the content.   96. The method of claim 95, comprising clarifying each file of the content based on inferences from the content.   96. The method of claim 95, comprising clarifying each file of the content based on the syntax of the content.   Identifying the operational content of the content across the location and at least one of the plurality of agents, wherein identifying the operational content includes: document name, document size, proximity, keyword comparison, and hash 96. The method of claim 95, comprising using at least one of the comparisons.   The method of claim 1, comprising controlling collaboration between a plurality of devices accessing the content file.   111. The method of claim 111, comprising controlling editing of content between a plurality of devices that edit the content file simultaneously.   111. The method of claim 111, comprising tracking activity on the content file.   111. The method of claim 111, comprising tracking a modification history of the content file.   The method of claim 1, comprising mobile device management comprising monitoring and controlling access to the content through a mobile device.   116. The method of claim 115, wherein the mobile device management comprises tracking a mobile device location.   117. The method of claim 116, wherein the mobile device management comprises controlling access to the content based on the location.   116. The method of claim 115, wherein the mobile device management comprises tracking mobile device data.   The method of claim 1, comprising connecting a console to the platform, the console including automated activity monitoring, inspection, and reporting associated with the content.   120. The method of claim 119, comprising controlling activity related to the content through the console.   120. The method of claim 119, comprising controlling the location through the console.   Controlling the activity includes controlling browsing of each file of the content, controlling downloading of each file of the content, controlling accessing each file of the content, 121. The method of claim 120, comprising at least one of controlling copying each file of the content and controlling delivering each file of the content. Connecting a platform having a processor to multiple databases;
Forming a grid having a plurality of agents connected to the platform, wherein each agent is an agent of the platform running on a client device;
Each agent generates metadata by hashing a plurality of fragments of content accessible to the agent, wherein the metadata is accessible to a plurality of client devices corresponding to the plurality of agents Corresponding to the content of the memory,
Providing the metadata to the platform instead of the content;
Generating a plurality of tasks including a task for controlling at least one of storing, transferring, and processing of the content using the metadata, wherein the task is executed by an agent responsible for the task; Processing operations performed on accessible content,
Assigning the plurality of tasks to the plurality of agents. Connecting a platform having a processor to multiple databases;
Forming a grid having a plurality of agents connected to the platform, wherein each agent is an agent of the platform running on a client device;
Each agent of each client device provides metadata of the contents of memory accessible to the client device to the platform;
Using the metadata to locate the content;
Using the metadata to generate a goal representing an action for maintaining the state of the content;
Generating a plurality of tasks corresponding to the goal using the metadata;
Assigning each task to an agent accessible to the content corresponding to the task, wherein the task is a processing operation instructed by the platform and executed by the agent on the content accessible to the agent And the metadata provided by the plurality of agents includes information of the plurality of tasks executed by the plurality of agents. Establishing a coupling between a platform having a processor and a plurality of databases;
Connecting a plurality of agents to the platform to form a grid, wherein each agent of the plurality of agents is an agent of the platform running on a client device;
Each agent generates metadata representing the contents of memory accessible by a plurality of client devices corresponding to the plurality of agents, wherein the generating is a plurality of fragments representing a plurality of fragments of the contents Hashing the plurality of fragments to generate a blob of
Providing the metadata to the platform instead of the content;
Generating a plurality of tasks using the metadata, wherein the task is a processing operation executed on content accessible to the agent responsible for the task, and the plurality of tasks include the task Including tasks that are at least one of content monitoring, storage, transfer and processing;
Assigning the plurality of tasks to the plurality of agents.