GB2588817A - Validating distributed source content - Google Patents

Abstract

Described herein is a machine-implemented method for generating a data image from a plurality of network devices, comprising: identifying a plurality of network devices each having an instance of said data image; requesting return of a portion of data and a hash tree path from each identified device of said plurality, each said hash tree path being derived starting from leaf data nodes of said data image; producing at least one compiled subtree of a hash tree from a plurality of returned said hash tree paths; verifying at least one said portion of data according to said at least one compiled subtree; and forming a data image comprising a plurality of verified said portions.

Description

VALIDATING DISTRIBUTED SOURCE CONTENT

The present technology relates to methods of validating delta updates for software images, including firmware images, stored on electronic devices. More particularly, the technology relates to methods for validating the data from distributed sources that is to be used to construct a delta update from a precursor image that has not been centrally saved.

In the past, information processing environments were typically isolated from the "real world", secured from interference by physical barriers and lack of electronic connections, and under the control of dedicated professionals with detailed knowledge of system operation, data integrity and system security. Such installations were once kept behind locked doors and tended by trained operators and system programmers; they were often only accessible from dedicated terminal devices which were themselves often kept in secure areas of a plant or office. Updates to software were usually conducted by selected professionals whose interactions with the systems were filtered through access control lists and passwords, and were often subject to checks and balances such as "buddy-checking", managerial sign-offs, and sometimes long periods of testing and parallel operation to ensure that the correct and secure functioning of the systems was maintained.

In recent years, by contrast, more and more devices are becoming networked and provided with local processing capability; these devices are typically in unprotected environments, open to the world through Internet connections, and under the control of people without any particular training in system operation, integrity and security. Very recently, devices from home computers to vehicles and light-bulbs have begun to acquire these additional functions and to be connected together through the Internet of Things (IoT). With this massive proliferation of networked devices, maintaining and updating stored data, including executable instructions such as software and firmware, poses increasingly complex difficulties.

To address the difficulties inherent in updating software images (including firmware images) in distributed devices, the present technology provides a machine-implemented method for validating the data from distributed sources that is to be used to construct a delta update for a software image.

To address the difficulties of validating data from distributed sources that is to be used to construct a delta update, the present technology provides a machine-implemented method of operating a data collector server attachable to a network, comprising: receiving an instruction requiring access to a data image not stored at the server; responsive to receiving the instruction, identifying a plurality of network devices each having an instance of the data; requesting return of a portion of data and a hash tree path from each identified device of the plurality, each hash tree path being derived starting from leaf data nodes of the data; producing at least one compiled subtree of a hash tree from a plurality of returned hash tree paths; verifying at least one portion of data according to the compiled subtree; and forming a data image comprising a plurality of verified said portions.

There is further provided a system comprising a server operable to perform the method and at least one of the plurality of devices operable to receive a request from a server for a portion and a hash tree path; responsive to the request, to derive the hash tree path; and to send the portion and the hash tree path to the server.

"Portions" of data, as the term is herein used, may be any digital data that can be handled in a structured or unstructured manner. The "hash tree" described herein may also be known as a Merkle tree.

In a hardware approach, there may be provided an electronic apparatus, such as a server, comprising logic elements operable to implement the methods of the present technology. In another approach, the computer-implemented method may be realised in the form of a computer program product, tangibly stored in a non-transient storage medium, and operable in use to cause a computer system to perform the process of the present technology.

Embodiments will now be described with reference to the accompanying figures in which: Figure 1 shows an apparatus or arrangement of apparatus at least part of which is operable according to an embodiment of the present technology; Figure 2 shows a method of operation of an update server or the like according to an embodiment of the present technology; Figure 3 shows a very simple example of a Merkle tree according to embodiments of the present technology; and Figure 4 shows a method of operation of a verifier according to embodiments of the present technology.

In the following description software may be configuration data or executable software and refers to a collection of data and/or instructions which inform a device how to configure/operate. Since firmware is a type of software, the following methods equally apply to firmware updates.

As described above, more and more devices are being connected via the internet, as part of the Internet of Things (IoT). The devices may be relatively large devices having large capacity and capabilities, such as laptop computers, tablets, smart phones etc, or may be relatively constrained devices having limited capacity and capabilities, such as temperature sensors, healthcare monitors, electronic door locks etc. All the devices are likely to require some form of software update during their lifetime, the update being downloaded via the internet and installed at the device. Large devices tend to have a storage capacity capable of storing the current level of software and the new level of software until they are ready to switch over to the current update. However, smaller devices (such as JOT sensors and the like) often do not have the capacity to store two versions of the software at the same time, and so some form of update in place is needed. Conventionally, this has been achieved by overwriting an old level of the software image with a complete new level under the control of a bootstrap control program and then restarting the system. However, this is not always feasible in systems in which communications bandwidth and device memory may be constrained.

In such systems, it may be preferable to use delta updates. A delta update is a software update that only requires a device to download the code that has changed, not the whole program. Delta updates are often used to reduce network traffic, to reduce the amount of storage required by an update and to save time.

When a new version of a software is available, generally an update server will determine a delta update for each device/group of devices that wants/needs the update. The update server knows which version of software each device is currently running and determines the delta update for each device based on the current version at each device and the new version of the software. The code and data which is required for updating the software can be compressed easily and efficiently using delta algorithms, in order to keep the delta update as small as possible. The delta update may then be sent to the devices which require the update.

An additional difficulty arises in mature systems of distributed devices, in which the versions or levels of software installed in various devices may diverge over time -some devices may be operating at one level while others are operating at a later update level. In addition, at the software distributor end of the system, it may be (for various reasons) that versions or levels of software have not been retained in any central storage location. This may be the result of some error, whereby a saved image of a level of software has been erroneously deleted, or it may be the result of a policy decision not to store every version or level of the software, perhaps because there are too many such levels for convenient storage and handling, or perhaps as a measure to prevent a "rogue insider", such as a disgruntled employee, from tampering with images that may be subsequently used for updates that will be widely distributed -such tampered-with images might represent a serious problem for end users, either by failing to operate correctly or by operating in a malicious manner.

Delta updates are typically created by calculating the difference between two binary files, one binary file being an image of the data at the earlier level and the other being an image of the data at the later level, and sending only the corresponding small delta update to the device that is to be updated. The delta update thus consists of data to be added or modified and instructions and location references for data to be added, modified, moved or deleted. To create a delta update requires the server to have an instance of the earlier level of the data to act as the precursor image in the generation of the delta update. This is straightforward enough in controlled and limited environments in which all the devices can be centrally controlled and monitored. This was the case, for example, with corporate in-house computer networks -all attached devices were centrally monitored and controlled by teams of professional system administrators, and software levels could be similarly centrally monitored and controlled. In the case of an update to software, the system administrators could with ease check which devices were up-to-date and which needed to be updated, and could then schedule updates from a central point.

In a world of massive connectivity and of the pushing down of data handling capabilities into very large numbers of lower and lower level devices, such as consumer appliances, this level of control is much more difficult to achieve. For example, the number of software variants deployed in the field is typically larger than would be the case in a contained and controlled in-house environment, and maintaining all the software variants deployed is not always possible. This may be due to human error causing loss of the original software, or it may be that it is not desirable to have those versions permanently stored on the server -for example, it may be necessary for business or technical reasons to maintain the server as stateless with respect to updates. In some circumstances, it can be very inconvenient if a delta update to IoT devices is not possible as the download capacity of devices and their networks can be very limited, making a full update package impracticably unwieldy. The absence of a precursor image makes generation of a delta update impossible, and this could potentially prevent updates altogether if a full update image is too large to be applied.

To address these difficulties and to compensate for the lack of centrally-stored precursor images, it would thus be desirable to provide a server that can assemble a precursor image from the deployed instances of the appropriate level of the image -for example a software or firmware image that is deployed on a plurality of devices attached to the network. This would also require each of the distributed devices to be operable to respond to requests from the server to send a portion of its currently installed software image version back to the update server.

The server would need to have information as to the size of the population of deployed devices that were equipped with the software that needed to be updated, and this could typically be determined from the response to a simple query canvassing the installed level identifier from each device in the network. Once the deployed population is determined, the size of the installed image and therefore the size of the portions and the number of request targets could be determined according to an algorithm that would achieve a sensibly low impact on the normal, day-to-day processing needs of the devices and the network connections. In another embodiment, in which the image size is not centrally-known, a relative portion size (rather than an exact size) may be given to the request targets as a fraction relative to the overall image size as known to each request target.

In this technology, each device would be asked to send only a small portion of its full software, thus reducing the impact of the request on its functioning to perform its normal, day-to-day processing tasks.

The data image is thus divided into portions. This can be done by specifying either the number of portions or the size of the portions. The chosen specification can be a fixed parameter of the protocol, or can be determined by the server at runtime (in this latter case this parameter needs to be transmitted as the part of the request). For simplicity of explanation, the following example assumes this to be a fixed parameter of the protocol and the partitioning of the requests is determined by the size of the portions. In the example, the number of portions is denoted by N. The client devices can derive this from the portion size and the length of the firmware image if necessary.) Thus, in the present technology, the server requests portions of the firmware from some number of devices in the network. The server's strategy of choosing the devices to ask for the required portions may be influenced by geography, network topology, type, environment and physical security of the devices, as will be clear to one of ordinary skill in the art, and this matter thus needs no further discussion here.

Depending on the number of devices in the population, the request could be sent to, for example, 1000 devices running the same software version, and each of the 1000 devices would then deliver 1/1000th of its software image, thereby distributing the additional workload and transmission slot occupancy across the 1000 devices. The burden on any single device would thus remain low as the uploaded portion of the image would be as small as possible, so that providing it would not significantly affect normal operation of the device. Requesting the same portion from several devices at the same time can reduce the latency of the recovery, but increases overall resource usage. For the sake of simplicity in the worked example hereinbelow, the server may ask 128 devices for each portion or tree node it needs.

Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to one of ordinary skill in the art that the present teachings may be practiced without these specific details.

In other instances, well known methods, procedures, components and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

Turning now to Figure 1, there is shown an apparatus 100 or arrangement of apparatus operable according to embodiments of the present technology. In Figure 1, an update server 102 is operable in electronic communication with an update source 103, which may be connected by, for example, the Internet, or which may be located in a computing cloud. Update server 102 is further operable in electronic communication with network 104, which comprises nodes, including at least Node 1 106, Node 2 108 and Node 3 110.

Nodes, including at least Node 1 106, Node 2 108 and Node 3 110 typically represent end-user devices attached to the network 104. These end-user devices may include any device connected to a network 104. They may be conventional computing devices, mobile (cellular) communications devices with processing capabilities, or any other device that is equipped to store and execute software or firmware. For example, they may be smart lightbulbs, temperature sensors, light switches, music speakers, smart fridges, healthcare monitors, electronic door locks, thermostats etc. According to another embodiment, one or more of Node 1 106, Node 2 108 and Node 3 110 may be a virtual device, a virtual device being an isolated software module, for example a virtual machine or container. Update source 103, update server 102, Node 1 106, Node 2 108 and Node 3 110 may be connected to the network 104 using wired or wireless communication. For example, network 104 may use wireless communication such as WiFiTM, ZigbeeTM, BluetoothTM, 6LoWPAN etc., may use short range communication such as radio frequency communication (RFID) or near field communication (NFC), or may use cellular networks, such as 3G, 4G, 5G etc. The network 104 may connect numerous different devices, each device using different or the same methods of communication.

As will be clear to one of skill in the art, the arrangement depicted in Figure 1 is much simplified for ease of description, and in any meaningful real-world setting, there would be many servers and nodes arranged in complex structures, such as tree and mesh networks.

Update server 102 comprises components, which may comprise hardware components, such as logic gates and circuits, or which may comprise program code components, such as code fragments, procedures or callable library elements. Update server 102 comprises updated image receiver 112, which is operable to receive updated images, such as an updated program code image, from update source 103. Updated image receiver 112 is in electronic communication with portion size calculator 114, which is further in electronic communication with network interrogator 116 and is operable to use information from updated image receiver 112 and network interrogator 116 to calculate the size of the portion of the image that is to be requested from deployed nodes in network 104. Network interrogator 116 is operable to interrogate nodes, such as Node 1 106, Node 2 108, etc. to determine the software or firmware version installed at each node and subsequently to determine which nodes will need to be asked by portion requester 118 to supply a portion of the image of the installed image. In one embodiment, in which the image size is not centrally-known, a relative portion size (rather than an exact size) may be given to the request targets as a fraction relative to the overall image size as known to each request target.

Responsive to inputs from portion size calculator 114 and network interrogator 116, portion requester 118 is operable to request portions of the image from selected nodes in network 114. Example request messages are indicated by simple dashed arrowed lines from portion requester 118 to the nodes selected to supply portions of the image, in this case, Node 2 118 and Node 3 110. Responsive to the request messages, the selected nodes, Node 2 108 and Node 3 110, are operable to return portions of their installed images to portion accumulator 120. The return messages of the example are shown as dot-anddash arrowed lines from Node 2 108 and Node 3 110 to portion accumulator 120. Portion accumulator may, in one embodiment, incorporate decryption engine 123, which is operable to decrypt the received portions, if they have been encrypted at their originating nodes in network 104.

Portion accumulator 120 is operable to accumulate the requested portions as they are received by update server 102 from the nodes in network 104, cooperating with portion verifier 122 to request that each portion be verified, using, for example, a comparison of hash values. Portion verifier 122 verifies a plurality of portions for security and integrity purposes, returning the verified values to portion accumulator 120 when the verification is complete. The operation of portion verifier 122 will be described in greater detail hereinbelow. When a portion fails verification, portion verifier 122 is operable to notify portion accumulator 120, which in turn communicates with portion requester 118 to initiate a repeat request for that portion from a different one of the potential target nodes in network 104. Portion requester 118 is operable on receipt of such a request from portion verifier 122 to elicit information about potential request targets from network interrogator 116 and to act upon that information to request the replacement portion from the new target node in network 104.

When portion accumulator 120 has accumulated all the required portions and had them verified by portion verifier 122, it is further operable to supply the complete precursor image to delta generator 124. Delta generator 124 is operable to receive input from portion accumulator 120 in the form of a complete precursor image of the software or firmware to be updated. Delta generator 124 is then operable to acquire an updated image from updated image receiver 112 and to generate a delta update from a comparison between the precursor image and the updated image. Delta generator 124 supplies the resulting delta update to delta distributor 126, which is operable to distribute the delta update to the relevant nodes in network 104. In the example shown, Node 1 106, Node 2 108 and Node 3 110 are all relevant recipient nodes, and the delta update distribution is shown in the form of a solid arrowed lines from delta distributor 126 to the nodes in network 104.

The above described apparatus or arrangement of apparatus is thus operable to generate delta updates for software or firmware images stored on electronic devices, constructing the delta update from a precursor image verifiably recovered from a plurality of devices in a network.

Turning now to Figure 2, there is shown a method 200 of operation of an update server or similar apparatus according to an embodiment of the present technology. The method 200 starts at 202, and at 204, an updated image, such as an image of a new level of software, is received at a distributor's system, such as an update server 102 of Figure 1. At 206, the required precursor level of the image is identified, and at 208, the precursor image size may be determined. In one embodiment, in which the image size is not centrally-known, a relative portion size (rather than an exact size) may be given to the request targets as a fraction relative to the overall image size as known to each request target. The request targets may then use the known size of their installed images to calculate the required portion size.

At 210, the network is interrogated for information concerning the affected population of devices, including the number of devices in the population. At 212, the number of target devices and the required portion size to be requested are calculated, and the target devices are selected from the affected population. At 214, each of the target devices is requested to return its portion of the complete image, and at 216, the requested portions are received. In cases in which a byte-oriented structure is to be preserved, and where the portions may not comply with that byte structure, portions may be rounded up to the nearest byte boundary, with a view to the rounding being adjusted-for at the server.

In certain embodiments, the portions will be received in a form in which they have been encrypted by the device software, and will require decryption before they can be used. In other cases, for example, where the portions are transmitted using an inherently secure method, such as Transport Layer Security (TLS), no additional decryption needs to be applied by the embodiment of the present technology. These techniques are directed to ensuring that the data has not been tampered with in transit, but do not offer any assurance as to the integrity the original data sent by each device -further verification is required to detect, for example, false data sent by a compromised device.

After the portions have been received at 216 and decrypted if necessary, they are further verified at 218. The verification may take the form of comparing received hash values with hash values calculated by the update server, or it may take any other of the many integrity checking methods known in the art. If the verification at 218 succeeds, all the portions are verified, and the process continues to 222. If the verification fails at 218, replacement target devices are selected at 220, possibly using information derived at 210 and 212, and the failed portions are re-requested at 214. This iterative process continues through 214, 216, 218 and 220 until all the required portions have been received at 216 and verified at 218. The process continues at 222, when the portions are assembled to construct a precursor image of the software or firmware suitable as input to the delta update generation process. In some cases, the construction procedure at 222 may require that any rounding of the bits in the portion performed by the target device be adjusted to maintain correct byte structures in the precursor image. At 224, the delta update is generated using the precursor image constructed at 222 and the updated image that was received at 204. Once the delta update has been generated at 224, it is distributed at 226 to the affected population of devices identified by the network interrogation process at 210. The method 200 then completes at 228.

For a fuller understanding of the apparatus of Figure 1 and the method of Figure 2, it may be helpful to consider an example scenario, simplified to assist in understanding.

In one simple example scenario, the total size of the binary image in the device is 4,000,000 binary digits and the algorithm determines that the request for data is to be sent to 10,000 devices in the deployed population.

One selected device in the deployed population receives a request for portion 451. The request format in this case could be rendered in pseudo-code as: return piece 451/10000 Assuming that the program code in the device's flash memory starts from address 0, it can be seen from the parameters of the return instruction that the selected device needs to return a portion of its binary image of the program code from starting address +(4,000,000/10000)*451 and read 4,000,000/10,000 bits from that address. In one implementation, the device may round up to the nearest full byte to avoid loss of single bytes due to rounding errors -the update server side can then handle the processing of any overlapping bytes.

It will be immediately clear to one of ordinary skill in the art that a real-world network will be much larger than this simplified example, and that more target devices may be used to return portions of the software or firmware image, depending on the estimated total size of the precursor image and number of deployed devices in the population.

In the example scenario, the return package from the device to the update server may be formatted as: piece 451 range 180400-180800 u8 payloarlf400] The selected device thus delivers the requested portion and after a time each device will have delivered its respective portion of the image. The portions are verified as described in further detail hereinbelow, and the server is then operable to assemble the full precursor image from the verified portions. If any portion is not returned or if a portion fails verification, a retry request for the missing portion may be issued to a different device.

For security and integrity reasons, the request and return messages could either travel securely via a Transport Layer Security (TLS) protected connection or be suitably encrypted by the device and decrypted at the server. In some implementations, the request may be implemented using the Hypertext Transport Protocol (HTTP) or Constrained Application Protocol (CoAP) using Lightweight Machine-to-Machine (LWM2M) observed resources.

When the server has acquired a complete precursor image assembled from the verified image portions received from the selected deployed devices, the delta update is generated using the precursor image and the updated image. The delta update can then be made available to all the relevant devices in the deployed device population.

The descriptions of Figures 1 and 2, and the example scenario sketched 15 above show features of the general embodiments of the present technology. Figures 3 and 4 will now be referred to in an expansion of the detail of the verification process briefly mentioned above.

Turning now to Figure 3, there is shown a very simple example of a Merkle tree according to embodiments of the present technology. In Figure 3, tree 300 has been generated starting from data portions X 302, Y 304, Z 306 and A 308. A hash algorithm is applied to each data portion, thus generating hash values H(X) 310, H(Y) 312, H(Z) 314 and H(A) 316. At the next higher level of the tree, the hash values H(X) from 310 and H(Y) from 312 are concatenated and hashed at 318, and the hash values H(Z) from 314 and H(A) from 316 are concatenated and hashed at 320. The hash values from 318 and 320 are further concatenated and hashed to generate the root value at 302. As will be known to one of ordinary skill in the art, a single incorrect value at a leaf of such a tree generates an incorrectly-valued branch all the way up to the root.

Embodiments of the present technology provide machine-implemented methods to rebuild a Merkle tree based on potentially untrusted data retrieved from nodes in the network and to use that tree to recover the data needed to construct a precursor firmware image. Merkle trees are typically used to verify integrity of data sets and to use the structure to update the data and the checksum or to prove membership in the dataset efficiently. The current approach differs from those in that in the present technology, the verifier does not necessarily already have the whole Merkle tree; and the main purpose of the algorithm is not merely to perform a pass/fail integrity check, but to set up the conditions for a successful search for verified data in the potential presence of an adversary who has tampered with the data.

The present technology provides for three cases: the server might have the root of the Merkle tree for the image it wants to recover; it might have the image hash for the whole image; or it have neither. All of these cases can be dealt with using variants of the present technology, but for the simplicity of the presentation of the algorithm and the protocol, the worked example below assumes that the server has the root of the Merkle tree. Some alternatives and variant scenarios will be discussed hereinbelow.

Similarly, the Merkle tree may be binary or non-binary, each of which has advantages and disadvantages. For example, the more leaves a node can have the more resource is consumed by the recovery on the server side, but in turn the client-side computations and the transmission bandwidth will be reduced. For the simplicity of the presentation of the algorithm and the protocol, the following worked example assumes that the Merkle tree is binary.

If at least the hash of the image is available to a server, it is possible to recover the image even if the compromised nodes are in majority. Again, for the simplicity of the presentation of the algorithm and the protocol, the worked example assumes that the honest devices are in majority.

The following notation is used in the following description: I I Concatenation of bit strings H() A collision free cryptographic hash function ai'b Exponentiation of a to the power of b log \_a(c) Base a logarithm of c A worked example of a recovery procedure according to the present technology, and with the assumptions described above, is as follows: 1. The server selects a set of clients to act as source devices and assigns a portion number to each of them in such a way that it receives each portion from 128 distinct clients. Each client sends only one portion.

2. The server sends out the request to each individual client.

3. Clients build a Merkle tree of their image and send the server the portion that was requested from them along with a branch of their tree leading to the requested portion.

4. The server verifies the portions and recovers the firmware image.

The verification in step 4 operates on the basis that the server has a parent node that has already been verified and then proceeds to verify the children of that parent node. Since, in the worked example, the server knows the root of the tree, the verification algorithm may be recursively called for the children until the whole tree is verified. As will be clear to one of skill in the art, there are known in the art alternative, for example non-recursive, implementations of such verification procedures.

The recursively called tree verification procedure takes as input P --a parent node (previously verified and correct); and M all the candidates received from the clients for the children of P. (Each candidate appears in M as many times as the number of clients that sent it.) The recursively called verification procedure produces as output Cl and C2 --children of P such that P=H(C1I1C2).

The procedure is as follows: 1. Use M to determine a set of 128 values for Cl and to determine a set of 128 received values for C2.

2. Pick a new pair of candidates for Cl and C2.

3. Output ERROR if all pairs have been tried and no new pair could be chosen.

4. Check if P=H(C1I IC2); and a) if P=H(C1I1C2) then output Cl and C2 b) go to step (2) otherwise.

Note that step (1) is always feasible. The server received values for Cl from at least 128 clients and values for C2 from at least 128 other clients.

Therefore we can always choose 128 candidates for each of them. It is not feasible to come up with Cl. and C2 for a given P such that P=H(C1I1C2), because H is collision free. Since P has already been verified and therefore known to be correct, from P=H(C1I1C2) we can deduce that Cl and C2 are correct.

In the worst case, all 128 values are different for both Cl and C2. In this case 16384 hash values need to be checked, which does not present a problem for a server. If the algorithm outputs ERROR, then at least one of the set of 128 values comes from a compromised node. If a single honest node exists in both sets, then the correct values have been found. With the assumption of having a majority of honest nodes, this means a negligible probability of contamination of at most 2^-128.

The verified Merkle tree construction procedure takes as input: a root node in the Merkle tree or a subtree of a Merkle tree (previously verified and correct); and A all the responses received from the clients.

The verified Merkle tree construction procedure produces as output the correct Merkle tree for the data to be recovered.

The construction procedure is as follows: 1. Use A to construct M, the set of candidates received for the children of R. Each candidate appears in M as many times as the number of clients that sent it.

2. Determine the children Cl and C2 using the child verification procedure described above for (R,M) 3. Determine the left subtree by using the tree verification procedure described above for (C1,A) 4. Determine the right subtree by using the tree verification procedure described above for (C2,A) 5. Set Cl and C2 as the children of R and output the tree.

Having shown the worked example, and now turning to Figure 4, a method 400 of operation of a verifier according to embodiments of the present technology is shown. In Figure 4, the process begins at start 402, and at 404, the server (such as the update server described hereinabove) receives an instruction that requires access to an image of data that is not available locally. The server identifies clients that can serve as source devices for retrieval of the data at 406. At 408, the server requests a portion of data and a corresponding hash tree path (or branch) from each client that has been selected to act as a source device. At 410, the server runs a compilation process to produce one or more compiled subtrees of the hash tree according to the hash tree paths supplied to it by the clients acting as data sources, applying the verification procedures described hereinabove to the data portions according to the compiled subtrees. Any ERROR returns are determined at test step 414 (any such ERROR returns indicating that a portion of data has not passed verification), and at 416, one or more replacement clients are selected as alternative data sources for that portion and the procedure returns to 408. ERROR is returned not only when a portion has not passed verification, but also when a node in the tree could not be recovered. In this latter case when requesting a replacement portion, the server may choose any portion in the subtree belonging to the missing node. Furthermore in this case the server might choose to request any number of portions (depending on the variant of the algorithm it implements and the strategy most suitable for the circumstances), not just one for every missing portion.

When all portions have been verified, the required data image is assembled at 418 and the process completes at 420. As will be clear to one of ordinary skill in the art, any such End step as 420 may indicate only the completion of one iteration of a process, and a return to Start 402 may initiate a further iteration of the process as required.

In one variant implementation, the number of clients used as source devices may be reduced. This optimisation may not contribute greatly in a typical case (thousands of portions with millions of clients), but may be significant if there are relatively few clients in the network. With asking 128 clients for the same portion, the probability of failure is negligible. This also means that most of the time it is enough to ask many fewer clients.

Also, since every client is sending a branch (a Merkle tree path), some clients sending different portions are also sending the same node higher up in the tree. For example the root of the tree is sent by every client independently of which portion they were asked to return. To minimise the overall load on the clients, we can ask fewer clients for the same portion in the first round and modify the child verification procedure to request new pairs from some clients and add them to M in step 3) instead of returning with an error.

Since every client sends a branch (a hash tree path), the input from a single client is involved at several levels in the algorithm and thus compromised nodes can be identified if they haven't sent the correct value. Ignoring such client responses reduces the computations needed to be done on the server side and blacklisting these clients when asking for new portions improves the likelihood of asking honest nodes and thus reduces the total number of requests needed.

In circumstances that are different from those assumed by the worked example above -for example, if the server does not have the root of the tree, but does have the firmware hash, the verification may be performed for each possible root value received, and the firmware hash be checked in the end. If the hash is correct, then the image is accepted. Starting with values sent by the greatest number of clients can speed up the process. If the server does not have the root or the firmware hash, a new request may be added to the protocol, whereby the clients could send the firmware hash or the root of the tree (whichever they have access to without doing the computations). The server then sends this request to all clients on the network and accepts the hash or root returned by the greatest number of clients.

As will be clear to one of ordinary skill in the art, the above-described "vote-counting" method may fail if the compromised nodes are in the majority and there is not much that can be done about that algorithmically. Since the ratio of honest and compromised devices is unknown, there exists a residual risk in this technique. In mitigation, even if the compromised nodes are in the majority, they might be under the control of several different actors and thus not working in a coalition, or they may lack the sophistication to use delta updates to propagate their contamination. In some cases the inspection of the delta update derived from the retrieved image data can discover the tampering (e.g., changes in the certificates and credentials may raise suspicions).

In a further variant, the Merkle tree of the present technology may be implemented as a non-binary tree -that is, with parent nodes having more than two children. With a binary tree, each client needs to calculate 2N hashes to build the tree and needs to send log \_2(N) hashes along with the requested portion. (The number of hashes sent can be reduced by one if we remove the leaf hashes and instead use the data portions directly.) If every node in the tree has k children, then the client has to calculate (1-N*k)/(1-k) nodes and send only log \_k(N) hashes. (This is because the number of nodes at each level is a geometric series, and thus there are log\_k(N) levels.) For example, if N=1024, 10 hashes must be sent for a binary tree and only 2 if every node in the tree has 32 children. Also instead of calculating 2048 hashes to build the tree the client only needs to calculate 1057 hashes. This reduces the battery and bandwidth consumption for the individual source devices (the clients), but may be burdensome in computational terms at the server. Thus, in the non-binary tree case, the server needs to consider k-tuples instead of pairs in the child verification algorithm. This means that instead of 128A2 = 16384 hash computations it would need to do 128A32 = 2A224, which is quite infeasible to do.

To address this burden at the server, one possible technique is to order the values received by their multiplicity (how many clients have sent it) and pick the first two in the order of multiplicity for all k children. Now the server has to verify only 2Ak hashes instead of 2A(7*k). In order to make this fail the adversary needs to have a 2/3 majority at one of the children, therefore raising the number of nodes asked to 192 should restore the probability of failure to 2^128.

In summary, using shorter trees reduces the battery and bandwidth consumption on individual devices, at the cost of affecting more clients and potentially increasing the computational cost on the server or introducing additional computational complexity in mitigation.

In certain circumstances, the honest devices may not be in the majority, and this introduces further considerations. If the compromised devices are in majority, the image still can be recovered. However, the server needs to request the same portion from more clients than otherwise to achieve the above-described negligible probability (2^-128) of not having a single honest device sending a particular portion. For example, if 60% of the devices are compromised then the original algorithm needs to be modified to send request to 173 devices for each portion. The number of requests is 308 in case of a 75% majority of compromised devices, 842 in case of 90% and 1729 in case of 95%. In the case of non-binary trees the threshold is lower and therefore the server needs to send 50% more requests per portion.

As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, the present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. It follows that, where the term "component" is used, it may refer to hardware components, such as logic gates and circuits, or it may refer to program code components, such as code fragments, procedures or callable library elements.

Furthermore, the present techniques may take the form of a computer program product embodied in a non-transient computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object-oriented programming languages and conventional procedural programming languages.

For example, program code for carrying out operations of the present techniques may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as VerilogTM or VHDL (Very high speed integrated circuit Hardware Description Language).

The program code may execute entirely on a user's device, partly on the user's device and partly on a remote server or entirely on the remote server. In the latter scenario, the remote computer may be connected to the user's device through any type of network. Code components may be embodied as procedures, methods or the like, and may comprise sub-components which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

It will also be clear to one of skill in the art that all or part of a logical method according to the preferred embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the method, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

In one alternative, an embodiment of the present techniques may be realized in the form of a computer implemented method of deploying a service comprising steps of deploying computer program code operable to, when deployed into a computer infrastructure or network and executed thereon, cause said computer system or network to perform all the steps of the method.

In a further alternative, the preferred embodiment of the present techniques may be realized in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the method.

It will be clear to one skilled in the art that many improvements and modifications can be made to the foregoing exemplary embodiments without departing from the scope of the present techniques.

Claims (15)

1. CLAIMS1. A machine-implemented method for generating a data image from a plurality of network devices, comprising: identifying a plurality of network devices each having an instance of said data image; requesting return of a portion of data and a hash tree path from each identified device of said plurality, each said hash tree path being derived starting from leaf data nodes of said data image; producing at least one compiled subtree of a hash tree from a plurality of returned said hash tree paths; verifying at least one said portion of data according to said at least one compiled subtree; and forming a data image comprising a plurality of verified said portions.
2. 2. The machine-implemented method according to Claim 1, said data image comprising a software or firmware image.
3. 3 The machine-implemented method according to Claim 1 or Claim 2, each said instance of said data image comprising an image at the same update version.
4. 4 The machine-implemented method according to any preceding claim, said identifying and requesting being responsive to receipt of an instruction requiring access to a data image not stored at said server.
5. The machine-implemented method according to Claim 4, said instruction requiring access to a data image not stored at said server comprising an instruction to use said data image as a precursor image for generating an image update delta.
6. 6. The machine-implemented method according to claim 5, further comprising generating an image update delta on completion of said assembling.
7. 7 The machine-implemented method according to any preceding claim, further comprising, if said verifying fails for a said portion, requesting said portion from a further device.
8. 8 The machine-implemented method according to any preceding claim, further comprising, responsive to said receiving at least one said respective portion in encrypted form, decrypting said at least one said respective portion.
9. 9 The machine-implemented method according to any preceding claim, further comprising requesting a same portion from a plurality of said identified devices.
10. 10. The machine-implemented method according to any preceding claim, said verifying comprising cross-comparisons of nodes of said hash tree.
11. 11. The machine-implemented method according to any preceding claim, said verifying comprising recursive verification of plural said portions.
12. 12. A data collector server apparatus, operable to form a data image, and comprising logic operable to perform all the steps of the method according to any of claims 1 to 11.
13. 13.A system comprising the data collector server apparatus according to claim 12 and at least one of said plurality of devices operable to: receive a request from a server for a portion of said data image; responsive to said request, identify said portion and generate a hash tree from its instance of said data; and send an identified said portion and a respective hash tree path derived from said generated hash tree to said server.
14. 14. The system of claim 13, further comprising when said request specifies a size of said portion as a fraction relative to a size of said software image, determining by said one of said plurality of devices an actual size of said portion.
15. 15.A computer program comprising computer program code to, when loaded into a computer system and executed thereon, perform all the steps of the method according to any of claims 1 to 11.