DE102020118411A1 - METHODS AND SYSTEMS FOR TRANSMITTING AN IMAGE IN Sentences - Google Patents

Abstract

Embodiments relate to techniques for improving the speed and rendering quality of an image (e.g., a JPEG) and are particularly useful in an end-to-end encrypted environment. The image can be analyzed on the side of the sending client and the image data is divided into several categories. In one embodiment, the image data is divided into four sets, each set (in conjunction with previous sets) can be rendered to provide increasingly high levels of quality (e.g., a thumbnail set, a medium quality set, a high quality set, and a full quality set). The image is uploaded to a blob storage and a message is sent to the receiving client with image information. On the receiving end, the JPEG image data is retrieved from the blob memory and each sentence is played back with progressively increasing quality levels. As a result, images are reproduced faster and the quality improves over time.

Description

RELATED REGISTRATIONS

This application claims priority of the provisional U.S. Patent Application 62,874,554 , filed July 16, 2019 and entitled "Progressive JPEGs". The contents of the above-mentioned application are hereby incorporated by reference.

BACKGROUND

Some communication services, such as messaging services and social networking services, allow users to exchange communications with other users. The communications can include images (such as a JPEG image). In particular, if the image is of a relatively large size, it can take some time to receive and load on the recipient's device. For example, a common phenomenon when requesting images over a network is that the upper half of an image is initially loaded after a certain period of time, with the lower half remaining empty. As more data is received, the rest of the image will eventually be filled in.

SUMMARY OF THE INVENTION

Accordingly, the present invention discloses a method according to the appended claims. The method includes: receiving, via an end-to-end encrypted session between a sending device and a receiving device, a message including a reference to an image; Establishing a connection to a blob storage, wherein the blob storage contains an encrypted thumbnail of the image, first additional encrypted data which, when combined with the encrypted thumbnail, results in a low quality version of the image, and second additional encrypted data which when combined with the encrypted thumbnail and the first additional encrypted data, resulting in a high quality version of the image; Retrieve the encrypted thumbnail from the blob storage, decrypt the thumbnail, and display the thumbnail; Receiving a request to display the image; Retrieving the first additional encrypted data from the blob memory, decrypting the first additional data, and merging the decrypted first additional data with the thumbnail; and displaying the low quality version of the image.

In an embodiment of the method according to the present invention, the blob memory can transmit information to the receiving device in a data stream and the method can further include: receiving a network quality parameter; Comparing the network quality parameter to a threshold associated with the high quality version of the image; Determining that the network quality parameter does not exceed the threshold; and terminating the data stream to cancel the transmission of the second additional encrypted data.

In another embodiment of the method according to the present invention, the retrieval of the encrypted thumbnail view from the blob memory can be carried out automatically in direct response to the receipt of the message.

In yet another embodiment of the method according to the present invention, the encrypted thumbnail, the first additional encrypted data and the second additional encrypted data can be stored together in an image structure and the message including the reference to the image can include: a first location in the image structure that separates the encrypted thumbnail from the first additional encrypted data, and a second location in the image structure that separates the first additional encrypted data from the second additional encrypted data.

In a further embodiment of the method according to the present invention, the method can further include: receiving a request to forward the image to a second recipient; and retransmitting the first and second locations in the image structure while renouncing transcoding of the image is dispensed with.

In another embodiment of the method according to the present invention, the retrieval of the encrypted thumbnail view from the blob memory, the retrieval of the first additional encrypted data from the blob memory and the retrieval of the second additional encrypted data from the blob memory in response to a single request for the image referenced in the message.

In another embodiment of the method according to the present invention, retrieving the encrypted thumbnail may include: downloading a data set that includes at least a portion of the encrypted thumbnail; Receiving a keyed-hash message authentication code (HMAC) in connection with the data set; Computing an HMAC over the downloaded data set; and comparing the calculated HMAC to the received HMAC to verify that the encrypted thumbnail has not been changed.

The present invention further discloses a non-transitory computer readable medium that stores instructions configured to cause a processor to: Receive, via an end-to-end encrypted session between a sending device and a receiving device, a message which includes a reference to an image; Establishing a connection to a blob storage, wherein the blob storage contains an encrypted thumbnail of the image, first additional encrypted data which, when combined with the encrypted thumbnail, results in a low quality version of the image, and second additional encrypted data which when combined with the encrypted thumbnail and the first additional encrypted data, resulting in a high quality version of the image; Retrieve the encrypted thumbnail from the blob storage, decrypt the thumbnail, and display the thumbnail; Receiving a request to display the image; Retrieving the first additional encrypted data from the blob memory, decrypting the first additional data, and merging the decrypted first additional data with the thumbnail; and displaying the low quality version of the image.

In one embodiment of the non-transitory computer readable medium according to the invention, the blob storage can transmit information to the receiving device in a data stream and the non-transitory computer readable medium can further store instructions for: receiving a network quality parameter; Comparing the network quality parameter to a threshold associated with the high quality version of the image; Determining that the network quality parameter does not exceed the threshold; and terminating the data stream to cancel the transmission of the second additional encrypted data.

In another embodiment of the non-transitory computer-readable medium according to the invention, the retrieval of the encrypted thumbnail from the blob memory can be performed automatically in direct response to the receipt of the message.

In yet another embodiment of the non-transitory computer readable medium according to the invention, the encrypted thumbnail, the first additional encrypted data and the second additional encrypted data can be stored together in an image structure and the message comprising the reference to the image can include the following : a first location in the image structure that separates the encrypted thumbnail from the first additional encrypted data, and a second location in the image structure that separates the first additional encrypted data from the second additional encrypted data.

In another embodiment of the non-transitory computer-readable medium according to the invention, the medium can further store instructions for: receiving a request to forward the image to a second recipient; and retransmitting the first and second locations in the image structure while renouncing transcoding of the image is dispensed with.

In another embodiment of the non-transitory computer readable medium according to the invention, the retrieval of the encrypted thumbnail from the blob memory, the retrieval of the first additional encrypted data from the blob memory and the retrieval of the second additional encrypted data from the blob memory can be combined in response to a single request for the image referenced in the message.

In yet another embodiment of the non-transitory computer readable medium according to the invention, retrieving the encrypted thumbnail may include: downloading a data set comprising at least a portion of the encrypted thumbnail; Receiving a key hash based message authentication code (HMAC) associated with the data set; Computing an HMAC over the downloaded data set; and comparing the calculated HMAC to the received HMAC to verify that the encrypted thumbnail has not been changed.

In another aspect, the present invention discloses a device according to the appended claims. The facility includes: a network interface configured to: receive, via an encrypted end-to-end session between a sending device and a receiving device, a message including a reference to an image; and connecting to a blob storage, wherein the blob storage contains an encrypted thumbnail of the image, first additional encrypted data that, when combined with the encrypted thumbnail, results in a low quality version of the image, and second additional encrypted data that, when combined with the encrypted thumbnail and the first additional encrypted data result in a high quality version of the image, holds; a processor configured to: retrieve the encrypted thumbnail from the blob memory, decrypt the thumbnail, and display the thumbnail; Receiving a request to display the image; and retrieving the first additional encrypted data from the blob memory, decrypting the first additional data, and merging the decrypted first additional data with the thumbnail; and a display configured to display the low quality version of the image.

In one embodiment of the apparatus of the present invention, the blob memory can transmit information to the receiving device in a data stream and can further include: receiving a network quality parameter; Comparing the network quality parameter to a threshold associated with the high quality version of the image; Determining that the network quality parameter does not exceed the threshold; and terminating the data stream to cancel the transmission of the second additional encrypted data.

In another embodiment of the device according to the present invention, the encrypted thumbnail, the first additional encrypted data and the second additional encrypted data can be stored together in an image structure and the message comprising the reference to the image can include: a first Location in the image structure that separates the encrypted thumbnail from the first additional encrypted data, and a second location in the image structure that separates the first additional encrypted data from the second additional encrypted data.

In yet another embodiment of the device according to the present invention, the processor can further be configured to: receive a request to forward the image to a second recipient; and retransmitting the first and second locations in the image structure while renouncing transcoding of the image is dispensed with.

In a further embodiment of the device according to the present invention, the retrieval of the encrypted thumbnail from the blob memory, the retrieval of the first additional encrypted data from the blob memory and the retrieval of the second additional encrypted data from the blob memory can be combined in response on a single request for the image referenced in the message.

In yet another embodiment of the apparatus according to the present invention, retrieving the encrypted thumbnail may include: downloading a data set comprising at least a portion of the encrypted thumbnail; Receiving a key hash based message authentication code (HMAC) associated with the data set; Computing an HMAC over the downloaded data set; and comparing the calculated HMAC to the received HMAC to verify that the encrypted thumbnail has not been changed.

Figure list

• 1A-1 E illustrate the same image at different quality levels according to one embodiment.
• 2 Fig. 10 illustrates an exemplary communication system suitable for transmitting an image in an encrypted end-to-end manner.
• 3 Figure 13 is a block diagram illustrating example computing systems suitable for implementing example embodiments.
• 4A FIG. 11 illustrates an exemplary data structure suitable for storing image data according to an embodiment.
• 4B Figure 3 illustrates an example message structure according to an embodiment.
• 5 Fig. 3 is an exemplary data flow diagram illustrating the exchange of information in computing systems in a first embodiment.
• 6A-6B 12 are flow charts illustrating example logic according to the first embodiment for encoding and sending an image from a sending client and receiving and decoding an image at the receiving client, respectively.
• 6C Figure 13 is a flow diagram illustrating exemplary image routing logic on the receiving client side for relaying a received image to another client.
• 7th is an exemplary data flow diagram illustrating the exchange of information in computing systems in a second embodiment.
• 8A-8B 12 are flow charts illustrating exemplary logic according to the second embodiment for encoding and sending an image from a sending client and receiving and decoding an image at the receiving client, respectively.
• 9 Figure 13 is a block diagram illustrating an example of a system for a messaging service;
• 10 Figure 3 is a block diagram illustrating an exemplary computing device suitable for use with embodiments;
• 11 illustrates an exemplary communication architecture; and
• 12 Figure 13 is a block diagram illustrating an exemplary multi-carrier communication device.

DETAILED DESCRIPTION

Often on a network, images such as JPEGs are loaded top-down while data is being received. This is often due to the fact that such images are typically packetized and transmitted from a beginning to an end of the image data structure, which means that the image data, which form the first part of the image, completely precede the image data, the subsequent parts of the image form, be transmitted. As a result, while the image is being loaded, the user may not have a particularly good feel for the image, as initially no image data is available from the lower portion of the image.

To overcome this problem, some communication networks allow their users to automatically download images when communications including the images are received. In this way, the user can be provided with quick access to the full quality version of the image if he does not immediately open the message that includes the image. The problem with this is that in many cases a user never looks at the downloaded image, which means that the network bandwidth used to transmit this message has been wasted.

Another potential solution is to create a smaller "thumbnail" of the image. A user can view the thumbnail to see a preview of what the image will look like, and then select the thumbnail to download the full size image. This solution allows a user to preview the entire image and also has the advantage that if a user does not want to download the full image, the amount of information transmitted over the network can be reduced. If the user previews and decides they do not want to download the entire image, they can forego it and avoid the need to broadcast the full quality image - in this scenario only the smaller thumbnail was sent. In fact, if a user wishes to download the entire image, using a thumbnail in this way increases the total amount of data transmitted over the network, since both the full quality image and the smaller thumbnail have to be downloaded. Also, if the full-size image is loaded, it may still load from top to bottom. Using a conventional miniature view will therefore not cope with the problem that the full size image is loaded incrementally.

Another potential solution is to use a progressive JPEG. When an image is transmitted over a network, the full image is typically uploaded to an intermediate server which enables the transfer. The intermediate server scans the image multiple times using appropriate transcoding parameters to generate multiple versions of the image data. When these versions are transmitted to a recipient, the recipient can play them back in a way that is first blurry and then progressively clearer. This is an improvement over the top-down approach of non-progressive JPEGs.

However, the transmission of a progressive JPEG generally depends on the intermediate server that has to process the JPEG. With a progressive JPEG, the multiple scans of the image are performed according to a scan script that includes transcoding parameters that define the data that is encoded during each scan. A JPEG typically has three channels: a luma channel (Y) and two chroma channels (Cr and Cb). Each channel can contain 64 coefficients that provide the data for the channel: one DC coefficient and sixty-three AC coefficients. Some of the coefficients for certain channels affect the image more than others and so progressive JPEGs can send the most important, influential, and less data-intensive information in the first scan, then subsequently compensate for the effect of the rest of the data on the image with the amount of data necessary to achieve this effect in subsequent scans. To determine which of this data is included in which scan, the intermediate server analyzes the image, selects appropriate transcoding parameters, and generates or applies a scan script to define the various scans of the image. The image is then scanned multiple times according to the scanning script to generate appropriate image data which can then be streamed to a recipient device to allow the recipient device to progressively display the JPEG.

While this works well in traditional communication systems, there has recently been a trend towards end-to-end encrypted environments (E2EE). In an E2EE environment, the body of a communication is encrypted at the sending client and decrypted at the receiving client, in a way that ensures that no other actors can view the message. Therefore, any intermediate servers that handle the communication cannot view the communication content. If the message includes an image, the intermediate server cannot determine the transcoding parameters, making it difficult to efficiently transmit the image.

Embodiments overcome these and other problems related to image transmission and reception. The techniques described improve the speed and rendering quality of an image (such as a JPEG) and are particularly well suited for an end-to-end encrypted environment.

According to exemplary embodiments, the image can be analyzed on the side of the sending client and the image data can be divided into several categories or segments. In one embodiment, the image data is divided into two sets: a thumbnail set and a full quality image set. The full quality picture set depends on the thumbnail set so that the picture data from the thumbnail set is reused when the full picture is displayed using the full quality picture set.

The sending client determines the records, identifies the record sizes, and calculates hash-based message authentication codes (HMACs) on each of the records. The image data can be encrypted and uploaded to a blob memory and a notification message can be sent to the receiving client with the particular image information and an appropriate decryption key. The receiving client can first download the thumbnail set and use it to play a thumbnail; in some embodiments, the thumbnail set can be automatically downloaded when a message is received. The user on the receiving side can view the automatically downloaded thumbnail and determine if they want to download the full screen with high quality.

Accordingly, the low quality version of the image can be used as a thumbnail. However, this approach has several disadvantages over traditional thumbnails, which are typically smaller images that are independent (e.g., stored as a separate file) from the full quality images they represent. In the embodiments described herein, the low quality version includes data that can be combined with the medium quality data to generate a medium quality version (and then a high quality version) of the image. As a result, images are played back faster and the quality improves over time (instead of top-down loading like a traditional JPEG).

In addition, the sentences are entirely defined and processed at the clients, without the aid of the blob memory (which only holds the encrypted sentences) or the intermediate server (which simply transmits the encrypted message from the sender to the client). The intermediate server therefore does not have to process the image or prepare any transcoding parameters.

In another embodiment, the image data can be divided into more than two sets, each set being reproducible (in conjunction with previous sets) to provide increasingly high levels of quality. For example, if the image data is divided into four sets, the set may include a thumbnail set, a medium quality set, a high quality set, and a full quality set. At the receiving end, the image data can be retrieved from the blob memory and decrypted, and each sentence can be played back with progressively increasing quality levels. In some embodiments, clients on low quality networks can only request the sentences that result in a medium quality rendering, thereby consuming less data.

This brief overview is intended to serve as a non-limiting introduction to the concepts discussed in greater detail below. However, before discussing further exemplary embodiments, a brief note on data protection is provided first. A more detailed description of the privacy settings and authentication is addressed in conjunction with the following figures.

A note on data protection

Some embodiments described herein use training data or metrics, which may include information that is voluntarily provided by one or more users. In such embodiments, data protection can be implemented in a number of ways.

For example, the user may be asked to consent to data collection before user data is collected or used. The user can also be given the opportunity to decline data collection. Before the user agrees to the data collection, a description of the way in which the data will be used, how long the data will be retained and what security measures are in place to protect the data from disclosure.

Information that identifies the user from whom the data was collected can be deleted from or disassociated from the data. In the event that identifying information must be retained (e.g. to meet regulatory requirements), the user can determine the collection of the identifying information, the uses made with the identifying information and the period of time over which the identifying information is retained. Information that specifically identifies the user can be removed and replaced, for example, with a generic identification number or some other non-specific form of identification.

Once captured, the data can be stored in a secure data storage location that includes safeguards to prevent unauthorized access to the data. The data can be stored in an encrypted format. Identifying information and / or non-identifying information can be deleted from the data storage after a predetermined period of time.

While particular data protection techniques are described herein for purposes of illustration, one of ordinary skill in the art will recognize that data protection may be implemented in other ways. Further details regarding data protection are discussed below in the section describing network embodiments.

Assuming a user's privacy policy is met, embodiments may be used in a variety of messaging systems, including messaging on a social network or on a mobile device (e.g., via a messaging client application or via a short messaging service), among other possibilities . Next, an overview of exemplary logic and processes for engaging in synchronous video calls in a messaging system is provided

To aid understanding, a number of examples are presented first, before detailed descriptions of the underlying implementations are given. It should be noted that these examples are only intended to be illustrative and that the present invention is not limited to the embodiments shown.

Referring now to the figures, like reference numbers are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the same. However, the novel embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate description thereof. It is intended to cover all modifications, equivalents, and alternatives in accordance with the claimed subject matter. In the figures and the accompanying description, the terms "a" and "b" and "c" (and similar identifiers) are intended to be variables that represent any positive integer. Thus, for example, if an implementation sets a value for a = 5, then a complete set of components 122, illustrated as components 122-1 through 122-a, may include components 122-1, 122-2, 122-3, 122-4 and 122-5. The embodiments are not limited in this context.

Images with different qualities

1A-1E illustrate various examples of an image transcoded with different qualities according to an embodiment.

This in 1E The final image illustrated is the full quality image encoded with 100% of the information making up the image. In the case of a JPEG, for example, this image may include data for all 64 coefficients for all three data channels (Y, Cr and Cb).

The first in 1A-1 B The illustrated image includes only a small amount of image data (e.g. approximately 20-45% of the image data, depending on the image). This image data can for example include all but the last two bits for the DC coefficients for all three image channels (Y, C and Cb). At full size ( 1A) this picture is clearly of a fairly low quality. However, if the size is reduced ( 1B) , the quality of the image may be sufficient to serve as a reasonable thumbnail.

The second in 1C Illustrated image includes additional data that corresponds to the data that the thumbnail of 1A form, supplement (e.g. approximately 36-67% of the image data, depending on the image). The low quality image data may include data from multiple scans; in one example, this may include the remaining bits from the DC coefficients for all three picture channels, as well as other data (e.g. the AC coefficients for both chroma channels and all but the last two bits of the AC coefficients of the Luma Canal). This image may be suitable for use as a low quality image. The resulting image will be of higher quality than the thumbnail, but the difference between this low quality image and the full quality image can be easily seen.

The third in 1D The illustrated image includes additional data that supplements the data making up the thumbnail and low quality image (e.g., approximately 56-81% of the image data, depending on the image). The medium quality image may include data from one or more samples; in one example, this may include the penultimate bit of the AC coefficients for the luma channel. This image may be suitable for use as a medium quality image. In reality, in many applications it can be difficult for the human eye to tell the difference between the mean quality image from 1D and the full quality image of 1E to make out. Therefore, especially when a user's network connection is not sufficiently fast or stable, the user's client device may choose to download only up to the medium quality image, thereby avoiding the need to transmit the rest of the data that will tell the difference between the medium quality version and the full quality version.

Finally, the full quality image ( 1E) include the remaining image data not captured by the thumbnail, low quality, and medium quality image data. In the example above, this would include the final bit of the AC coefficients from the Luma channel.

The four qualities in 1A-1E are just one example of how an image can be encoded with different qualities. In exemplary embodiments, images with only two qualities (e.g. a thumbnail version, as in 1A illustrated, and a full quality version as in 1E illustrated), three qualities (e.g. a thumbnail corresponding to the image of 1A corresponds to a low quality version similar to the image of 1C or 1D and a high quality version that matches the image of 1E corresponds to) or can be coded with more than four qualities.

Next, an exemplary system for distributing images such as those described above will be described with reference to FIG 2 described.

Exemplary communication environment

As in 2 As shown, embodiments may include communication services (potentially including end-to-end encryption) and / or social network services between a sending client 202 and a receiving client 212-i , made possible by an intermediate server 204 , provide. Some embodiments described herein use server-side broadcasting to carry group messages and broadcasts. In the case of a server-side distribution process, the sending client transmits 202 a single copy of the message to the server 204 who then sends a copy of the update to the appropriate recipient clients 212-i sends. The intermediate server 204 can for this purpose client information in a client database 206 store, thereby distributing messages to the client devices 212-i becomes possible.

In order to transfer media and / or attachments, the communication system can also use a blob server 208 include that of a blob memory 210 hosts (this can be part of the intermediate server 204 hosted or can be deployed separately). When a message is sent that includes a media item (such as a picture or video), the media item can be sent to blob storage 210 when the message is uploaded to the receiving client 212-i is pushed. Optionally, the receiving client 212-i a low quality thumbnail can be provided while the original media item is in blob storage 210 is saved. When the receiving clients 212-i the receiving client can attempt to access the media element 212-i the media item from the blob server 208 request the requested media item from blob storage 210 and it to the receiving client 212-i sends. This can be done by establishing a streaming connection with the receiving client 212-i and transmitting the data as part of the stream.

The use of the blob storage 210 to transmit content (in particular in connection with the sending of an encrypted message), the content can be made available to the receiving clients 212-i enable, especially if a client is offline when the update is sent and / or if different clients 212-i try to access the content at different times.

In operation, embodiments may place the image processing logic that processes the images, divides them into scans, organizes the data image data in sets suitable for streaming, calculates key-hash-based message authentication code codes, routes the images, on the sending and receiving clients. The intermediate server 304 may be configured to forward only picture messages that include a reference to the picture that are then sent by the receiving clients 212-i from the blob server 208 is retrieved. Such logic is referred to below with reference to 3 described in detail.

Exemplary computing systems

3 Figure 13 is a block diagram illustrating an exemplary response recommendation system suitable for use with embodiments.

One client 300 can be used to capture an image, record an audio / video, display an image, and / or play an audio / video feed. The client 300 For example, it can be a computer, mobile device, special purpose media creation device, or any other suitable client device.

The client 300 can be an audio input device 302 such as a microphone. The audio input device 302 may record audio in the vicinity of the client device and pass the recorded audio to audio input logic 312 forward for processing. The audio input logic 312 can process the recorded audio, potentially with the aid of sound analysis logic to extract information from the audio (e.g. a textual transcript). The audio input logic 312 can for example perform speech-to-text processing, analyze tone and / or pitch in the audio, provide the audio directly to a machine learning system for analysis, etc.

The client 300 can also be a text input device 304 such as a physical keyboard or an on-screen keyboard for entering text.

The client 300 can be an image input device 306 such as a camera capable of recording still images and / or video. An image input logic 314 can process incoming graphic content, for example from the client-side image processing logic 600 and / or the client-side multi-sample logic 700 to be used.

The audio input logic 312 and / or the image input logic 314 can also or alternatively on the server 330 lie.

The client 300 can be an ad 308 such as a touch screen or other display device. The ad 308 can display a communication interface or social network interface including the in 1A-1E illustrated images.

The client 300 can be an audio output device 310 such as a speaker or an audio output jack. The audio output device 310 can play audio associated with an update, such as the audio track included in a video. The audio output device 310 can output the audio via an audio output logic 316 received in connection with the operating system 318 and / or an application 338 can work to provide output audio.

Any or all of the audio input devices 302 , the text input device 304 , the video input device 306 , the ad 308 and the audio output device 310 can / can in the client 400 be integrated or can / can outside of the client 300 deployed and with the client 300 be connected in a wired or wireless manner. These devices can also be used to support a communication application, such as a video calling application, a messaging application, or a social networking application.

An operating system 318 can the client 300 support and can use the audio input device 302 who have favourited the text input device 304 who have favourited video input device 306 , the ad 308 , the audio output device 310 , a memory 320 , an application 338 and / or a network interface 326 coordinate and / or control.

The memory 320 (e.g. RAM, ROM etc.) can contain data structures for use by the application 338 and / or the operating system 318 to save. The memory 320 can for example save a message made by the application 338 was received, the message including graphical content. In the memory 320 can take a picture 322 sent by the image input device 306 is captured and / or transferred from a sending client, as well as a miniature view 324 of the picture 322 if one is retrieved from the blob server, saved.

The client 300 can be an application 338 capable of sending and receiving communications on a network. These skills can also be used between multiple applications 338 be divided. The application 338 For example, it can be a communication application, a social networking application, or any other type of application that includes update and response capabilities.

The application 338 can include image and / or video capture logic 340 capable of capturing, processing, analyzing and displaying graphical content. The image / video capture logic 340 for example, may support interface logic configured to generate a social network interface or messaging interface.

The application 338 can also use an E2EE session setup logic 340 that is configured to establish an end-to-end encrypted session from a sending client to a receiving client over a communications network. The E2EE session establishment logic 340 For example, it can be configured to send an E2EE setup message from a sending client to a receiving client, the setup message including parameters (such as a decryption key) that allow the sender and recipient to encrypt and decrypt mutual messages so that Intermediate devices (such as the server 330 ) cannot read the message content.

For this purpose, the application 338 an encryption and decryption logic 342 that is configured to support messages sent over the E2EE connection encrypt and decrypt. The encryption and decryption logic 342 may employ any suitable encryption and decryption algorithm, such as the Advanced Encryption Standard (AES) algorithm, Diffie-Hellman algorithms and their variants, or any other suitable encryption algorithm.

The application 338 can also have picture coding / picture decoding logic 344 support that for processing an image 322 suitable is. The picture coding / decoding logic 344 can convert the image to a thumbnail and a full screen, as in the ones referring to 5-6B described embodiments, or in several intermediate image qualities, as in those with reference to FIG 7-8B described embodiments, split.

The application 338 can also include message transfer logic 346 support that is configured to generate a message containing information about the image 322 may include and the message to the server 330 to be transmitted to be retransmitted to a recipient client.

The application 338 , the memory 320 and the I / O devices can be accessed through the operating system 318 with a network interface 325 for communication in a network 328 such as the Internet. The network interface 326 For example, it may be a network interface card (NIC) or other suitable hardware and / or software for regulating network transmissions and receptions.

The client 300 can through the network interface 326 with a corresponding network interface 332 on a server 330 communicate. The server 330 can support for the application 338 and can provide messages from the client for this purpose 300 and receive them using message forwarding logic 334 forward to one or more receiving clients. The server 330 can also be a blob memory 336 to save an encoded and encrypted version of the image 322 support (or the blob storage 336 can be stored on a separate device).

Any or all of the logic described above can be executed via instructions on one or more hardware processors. The instructions can be stored on a physical, non-transitory, computer-readable storage medium.

It should be noted that although the logic and data structures in 3 be illustrated in a special configuration (with some on the client 300 and others on the server 3 30 ), other configurations are contemplated in connection with the present invention.

Exemplary data structures

4A illustrates a data structure used to store image data 404 in encrypted user data 402 suitable is. The image data 404 can come in several qualities 404-i which can be determined and constructed by the sending client device. The qualities 404-i may each comprise one or more image samples and may be included in the image data 404 be presented in a sequential order so that the image quality increases when of a quality ( 404-i) going to the next (404-i + 1). For example, the first includes quality in 4A Data that is of thumbnail quality 404-1 while the second quality includes data that is of low quality 404-2 when this in conjunction with the data of the thumbnail quality 404-1 be reproduced.

The amount of image data that each quality 404-i may vary. In this example, thumbnail quality includes 404-1 for example eight blocks of data, for one data length 406-1 of 8. The low quality data 404-2 comprise twenty four blocks of data, for one data length 406-2 of 24. A first divisor 408-1 after eight blocks of data, there is a conceptual split point (it may or may not correspond to actual information stored in the data structure) between the thumbnail data 404-1 and the low quality data 404-2 represents. Likewise represents a second divisor 408-2 after thirty-two blocks of data, a conceptual split point between the low quality data 404-2 and the medium quality data 404-3 represent.

When the image data 402 are transmitted from the blob storage to the receiving client, they can be transmitted as part of an encrypted data stream. Such data streams can be transmitted in sets, with each set size being a multiple of the AES block size (e.g. 16 bytes). However, there is no guarantee that the dividers will 408-i between the different qualities 404-i will correspond to the division points between sentences. In the in 4A In the example illustrated, the first sentence has an endpoint 410-1 after 16 blocks, while the second set has an end point 410-2 after 32 blocks. Therefore, when the first sentence is transmitted, it becomes all of the thumbnail data 404-1 and one third of the low quality data 404-2 include. The second set becomes the rest of the low quality data 404-2 include.

The HMACs for the encrypted data stream are generally calculated over each record. Accordingly, the sending client can determine and perform the scans of the image and can also determine where the division points will be between the various sets. The sending client can calculate the HMACs using these records. The sending client can upload the image data to the blob storage and can transmit an image message containing information about the image to the receiving client. An example of such a picture message 450 is in 4B illustrated.

The picture message 450 can be the recipient or recipients 452 identify for the message. This information can be used by the server to route the message to the appropriate receiving client devices.

The picture message 450 can also encrypt message data 454 included by the encryption / decryption logic 342 on the client device 300 can be encrypted / decrypted. The payload 454 may include various encrypted message data. For example, a message body 456 Message information to be displayed on an interface of a messaging application. The message body 456 for example, may include text to be displayed in the message. If the message includes (or consists of) an image, the message body may 456 a reference 458 on the image, which can be stored remotely in blob memory. The reference 458 For example, it may include a pointer to the image in the blob memory or an identifier for the image so that the image can be retrieved from the blob memory.

The payload 454 may further include information describing how the image is broken down into different qualities so that it can be sequentially reconstructed by the receiving device. For example, the payload 454 the length (s) 460 of the data corresponding to each quality. In the case where the image data is divided into thumbnail and full image data, the length (s) 460 may be a single value that is the length of the thumbnail data 404-1 corresponds. If the data is divided into intermediate qualities, the length (s) 460 can be represented as an array or list comprising the length of the data corresponding to each image quality.

Using this information, the receiving client can determine where within each set any dividing lines between image data of different qualities may be. The receiving client can use this information to display the image with the appropriate quality and / or to end the data stream before downloading the full image if the network quality is relatively low.

Alternatively or additionally, the user data 454 include a list of where each scan of a progressive scan script begins. Accordingly, the image can be loaded as a progressive image (e.g. a progressive JPEG) by updating the UI each time a new sample is received. This can allow further intermediate stages within a particular image quality if the image quality is composed of several scans (see e.g. the low quality example in Table 1 below).

The payload 454 can also use the HMACs calculated by the transmitter 462 include for each image set. After receiving a sentence, the receiver can calculate its own HMAC for the sentence and use this with the HMAC 462 in the picture message 450 to compare. If the HMACs are the same, the receiver can verify the integrity of the received data. If the HMACs differ, then the data was either corrupted or manipulated during transmission. In this case, the recipient can take appropriate action, such as requesting a new copy of the data or terminating the transmission.

Embodiment for providing a thumbnail and a full image

5-6B provide data flow diagrams and flow charts describing a first embodiment in which an image is divided into a thumbnail and a full quality image. It is intended that the in 5A-6B illustrated embodiment in conjunction with any of the above thumbnail / full-screen embodiments described can be used and with the in 7-8B illustrated multi-sampling embodiments can be expanded.

As in 5 shown, the sending client can first send an end-to-end encryption setup message (E2EE setup message) 502 send to the intermediate server. The intermediate server can send the E2EE setup message 502 forward to the receiving client. The receiving client can send an E2EE setup response 504 and generate the answer 504 transferred to the server. The server can answer 504 to the sending client. The setup message 502 and the answer 504 may include all the information necessary to establish an E2EE session between the sending client and the receiving client.

At the sending client, a user can work with a communication application to compose a message (process 506 ). The message can include an image. In response to receiving a request to transmit the message, the image can be viewed using the in 6A process described are encoded and encrypted. The encrypted image data 508 can be uploaded to the blob server that comes with a place 510 of the image in the blob server can respond.

It can be a picture message 512 generated comprising encrypted message content; the encrypted message content can have the lengths of each set of data corresponding to different image qualities (see 4A-4B) , which include HMACs from each record in the image data and a reference to the image. The reference can be the image location received from the blob memory 510 be or be derived from it. The picture message 512 can be transmitted to the server and then transmitted from the server to the receiving client.

The receiving client can view the image message using the information from the E2EE setup and response messages 502 , 504 decrypt. The receiving client can recognize that the decrypted message includes a reference to an image and can determine whether automatic downloading of image thumbnails is enabled. For example, automatic downloading can be activated or deactivated in a setting of the user's communication application and automatic downloading can be deactivated in certain contexts (e.g. if the user is on a cellular data network, if the network is unstable or slow, etc.). If automatic download is enabled, the receiving client can make a request 514 transferred to the blob memory for downloading the thumbnail portion of the image data. This can build a data stream with the blob storage, initiate a download 516 of the image data referenced in the image message and terminating the data stream as soon as enough image data has been received to display the thumbnail 518 to reproduce.

The communication application can request 520 that the image is retrieved from the blob memory. This requirement 520 can take place in response to a user requesting the full picture (e.g. by clicking the thumbnail) if automatic downloading has been activated and performed. Alternatively or additionally, the image can be downloaded in response to a thread including the message being opened in the communication application.

In response to the request 520 the blob server can use a data stream for the image data 522 transmitted to the receiving client. The image data can be streamed sequentially so that the thumbnail data is transmitted first and the frame data is transmitted second. If the thumbnail data is at 516 have already been transmitted, then only data can be sent in response to the subsequent image request 520 that have not yet been retrieved. After the thumbnail data at 516 downloaded, they can be saved in a thumbnail directory so that they remain available to refer to with 522 retrieved data to be united. If automatic download is not activated, the entire image (first the thumbnail data, then the full-screen data) can be downloaded in response to the individual image request 520 can be accessed.

While the data is being received, it can be played back on the receiving client. In the case of a thumbnail / full screen embodiment, the thumbnail can first be shown at 524 and then the UI can be updated when the frame is available at 526.

At some point in time, the receiving user can request that the receiving client forward the image. If the receiving client still has a copy of the image reference, the data lengths and HMACs in its memory, as in the encrypted message 512 received, saved, this information can simply be forwarded to the next client 528 without re-encoding the image.

These messages can be generated in accordance with the flowcharts of 6A (on the transmitter side) and 6B (on the receiver side) illustrated processes are generated and transmitted. The example logic can be implemented in hardware, software, or a combination of hardware and software (e.g., implemented at least in part in hardware). For example, the logic may be included as digital logic that may be implemented at least in part in hardware and that includes instructions for a processor circuit to perform the steps described below. Even though 6A-6B (and Figures 8A-8B) illustrate a particular arrangement of logical elements in a particular order, it should be understood that the configurations illustrated are merely an example. In other embodiments, more elements can be provided and / or some elements can be omitted, some elements can be performed in parallel, and / or elements can be performed in a different order. In some embodiments, some of the elements illustrated may be performed in different systems.

In the image processing logic on the side of the sending client 600 can processing at block 602 begin with the sending client establishing an E2EE session with a receiving client. The sending client can establish the E2EE session in response to receiving a request to transmit a message from the sending client to the receiving client. Establishing the E2EE session can include, for example, the exchange of encryption and decryption keys between the sending client and the receiving client.

At 604 the sending client can receive a request in an interface of a communication application in order to compose a message for the receiving client (block 604 may in some embodiments the block 602 precede). The message can include an attachment, such as a picture, that can be stored in memory or storage of the sending client. Accordingly, the communication application at Block 606 retrieve the image from memory or memory. Alternatively, the image can be saved remotely and the communication application can block 606 request a copy of the image from the remote location.

At block 608 the system can determine whether multi-sampling is enabled. In some embodiments, multi-scanning can be used to scan the image with a thumbnail quality and with at least one additional intermediate quality between the thumbnail quality and the full-frame quality. Multi-sampling can be enabled by default, based on user preferences or in certain contexts (e.g. if the image is expected to be downloaded from an unreliable or slow network). If applicable, processing can be done with Block 802 continue (see 8A) .

If multi-scanning is not activated, then the image can be displayed at Block 610 can be scanned with two qualities. For example, the image can be scanned as a progressive JPEG using a scanning script. An example of a scan script that provides two or four image qualities is as follows:

In this example, the first sample is the initial sample for the DC coefficients of the Y, Cr, and Cb channels, minus the lowest two bits. This information is suitable for use as a thumbnail for the image.

The next five scans produce a low quality image. Of these, the first two (samples 2 and 3) capture the penultimate and last bit, respectively, of the DC coefficients for the Y, Cr, and Cb channels. The third (sample # 4) captures the AC coefficients for the Cr chroma channel, while the fourth (sample # 5) captures the AC coefficients for the Cb chroma channel. The fifth (sample no. 6) captures all bits of the AC coefficients for the Y luma channel except for the last two.

Typically, the AC coefficients for the Luma channel comprise most of the data and are downloaded last. The next sample (sample # 7), which can be combined with the previous samples to produce a medium quality image, therefore reads in the penultimate bit of the AC coefficients for the Y luma channel. The final scan (scan # 8) produces the high quality frame (when merged with the previous scans) by reading in the lowest bit of the AC coefficients for the Y luma channel.

It should be understood that this particular scan script configuration is exemplary only and uses particular data that has been found in practice to adequately balance the size of each respective scan with the quality obtained from rendering the scan in conjunction with previous scans results. For example, the first scan that produces a progressive thumbnail tends to provide a relatively large amount of information in a relatively small amount of data. The quality is not sufficient for this information to represent a medium quality image, but is sufficient for use as a thumbnail. In practice, the low quality samples tend to take in about half of the image data (around 36-67%, with an average of around 48%), while the medium quality samples tend to take 56-81% of the image data, with an average of around 66 %. In many cases, the medium quality images are indistinguishable for a human from the original full quality image.

In embodiments that provide a thumbnail and full image (ie, with no intervening low quality and medium quality rendition), the second through eighth samples can be treated as full image data. It should be noted that in both the multi-scan and thumbnail / full-frame embodiments, the image can still be processed as a progressive JPEG. However, the clients can be configured to treat the first sample (or samples) as thumbnail data and can use the other layers (low quality and medium quality) to perform other processing. For example, depending on the network quality and available bandwidth, the receiving client may choose to receive only the amount of data that results in a low quality or medium quality image. In such an embodiment, limits can be defined for various network quality parameters (e.g. throughput, bit rate, packet dropping rate, etc.). If the network quality is above a first (low) threshold level, then the low quality image is received and the receiving client can access the data stream after receiving the samples corresponding to the Match the low quality image, exit. If the network quality is above a second (intermediate) threshold level, then the medium quality image is received and the receiving client can terminate the data stream after receiving the samples corresponding to the medium quality image. If the network quality is above a third (high) limit level, then the full image can be received and the receiving client can terminate the data stream after receiving all available samples.

The system can customize the scan script or it can use a different scan script for various reasons. For example, the use of different transcoding parameters can result in a smaller file size, and such parameters can be selected to reduce the amount of data transmitted over the network. Additionally, since each scan of the image consumes processing and other computational resources, and since these resources may be more limited on an end-user client device than on a central server, depending on the quality of the hardware on the end-user client device, the system may choose to perform fewer scans .

As a result of scanning the image at block 610 image data is generated, the image data including data corresponding to the thumbnail scan (s) and the remainder of the image data. At block 612 the system can calculate the length of the thumbnail scan (s). This can be done, for example, using a progressive JPEG parser which identifies the location in the image data that corresponds to the point at which a particular scan ends and the next scan begins.

At block 614 the system can encrypt the generated image data using a suitable encryption algorithm. At block 616 the system can divide the encoded image data into sets and blocks in accordance with an encrypted data streaming algorithm 618 the HMAC can be calculated for each rate.

At block 620 the encrypted image data can be uploaded to the blob memory and the blob memory can tell the sending device about the location in the blob memory where the image is stored (or about the identification of the image that can be used to identify the image in the blob memory).

Using the at block 612 , 618 and 620 The system can block the generated information 622 generate a picture message. The picture message may have a reference to the picture, which may include the location and / or the ID retrieved from the blob memory at block 620 are output, the thumbnail length, which is the block 612 is calculated (and / or other lengths that are specified in Block 810 are calculated), as well as the record HMACs that are used in Block 618 calculated include. The content of the picture message can be encrypted and by block 624 the picture message can be transmitted to its intended recipient (s).

The in 6A (8 and 8A) can be performed in parallel with or in conjunction with a process executed in the receiver-side device. 6B (and FIG. 8B) is a flow diagram illustrating example logic 650 (and 850 ) which is performed by a system such as a receiving client to request and display a progressive image.

At block 652 the receiving client can establish an E2EE session. block 652 can be the counterpart of block 602 and may include similar or equivalent steps.

At block 654 the receiving client can use the in block 652 receive a picture message when the E2EE session has been established. The picture message may include encrypted content such as an encrypted reference to a picture, data lengths, and HMACs.

At block 656 For example, the system may retrieve the lengths of at least some of the data corresponding to the respective image qualities included in image data of the referenced image (such as the length of the thumbnail data). As previously mentioned, the data lengths may not exactly match the record lengths of the E2EE data stream, and thus the system can use this information to determine how many records it will take to obtain the data corresponding to each respective image quality.

At block 658 allows the system to determine whether automatic thumbnail downloading is enabled. If not applicable, processing continues with block 660 and the system waits for a request to display the picture message before downloading the thumbnail data. Such a request may be in the form of a request to load a thread containing the picture message or to scroll to a point in the thread where the picture can be viewed. Alternatively, if automatic downloading is activated, processing can be carried out directly with Block 662 Continue.

At block 662 the system retrieves the image data corresponding to the thumbnail from the blob memory. This may include initiating a data stream with the blob memory to retrieve the image data corresponding to the reference in the original image message. The system can download one or more sentences that match the thumbnail.

At block 664 the system can retrieve the thumbnail data according to the parameters of the block 652 decrypt the established E2EE session. At block 666 the system can calculate an HMAC using the encrypted data (block 666 may in some embodiments block 664 to verify that the data is not corrupted and unchanged. Assuming this is the case, the system can at Block 668 view the decrypted thumbnail. Once the thumbnail data has been downloaded, the system can optionally pause the download of the rest of the data and / or end the data stream with the blob memory.

At block 670 the system can receive a request to display the full screen. This can be done in response to a thumbnail selection indicating that the user would like to view a higher quality version of the image. Alternatively or additionally, the system can simply continue to download the image data after the thumbnail has been downloaded.

At block 672 the system can initiate a data stream with the blob memory and at block 674 transmit a request to the blob memory to download the image data (or alternatively may continue to download data over an existing data stream). In some embodiments, only one request is needed to download all of the image data with the various qualities with which the image was encoded (as the image data continues to sequentially stream additional data that can be cumulatively applied to render the image at increasing levels of quality) .

At block 676 the system determines whether multi-sampling capabilities are enabled (similar to Block 608 of 6A) . If applicable, processing continues with block 852 away (see 8B) . Otherwise processing continues with block 678 continues and becomes a set of data on the streaming connection from block 672 retrieved.

At block 680 the system can decrypt the received sentence and with block 682 verify the sentence HMAC (block 680 and 682 can be reversed in some embodiments). At block 684 the system updates the UI based on the newly received image data (assuming the most recent set included data up to the end of one of the image samples; otherwise, block 684 be omitted in this iteration of the loop).

At block 686 the system determines whether there are more records to download from the blob memory. If so, processing returns to the block 678 back and the next sentence can be called up. If not applicable (ie all sets have been downloaded), processing can be stopped because the image was reproduced at full quality.

After the receiving client has received the image data, the receiving user can request that the image be forwarded to another user. 6C Figure 3 is a flow diagram showing example logic 690 for forwarding the image to a second receiving user.

At block 692 the system can receive a request to forward a previously downloaded image to the receiving client. The request can be to forward the image to a second receiving client with which the first receiving client will block an E2EE session 694 can build.

Since the image data is already in the blob memory and since the various scans and image qualities have already been processed and the HMACs have been calculated using the relevant sentences, it is not necessary to perform this processing again (assuming that the receiving client still has a copy of the relevant data lengths and HMACs from the original sending client).

Accordingly, the system generates at block 696 a new image message for the second receiving client comprising the data / sample lengths from the original sender and the HMACs from the original sender. The reference to the picture from the original sender can also be included in the new picture message.

The new picture message can be sent to Block 698 forwarded to the second recipient. Accordingly, the second recipient can use the information in the new picture message to download the picture from the blob memory in the same manner as for the first recipient.

Embodiment for providing multiple samples

7th , 8A and 8B complement the in 5 , 6A and 6B in order to provide a system that generates intermediate image qualities between thumbnail quality and full image quality.

The data flow representation of 7th is the in 5 illustrated similar in many respects. For the sake of brevity, in 5 and 7th Similar message exchanges are identified by the same reference numbers and a description of those similar steps is omitted here for brevity. A difference between the in 5 measure taken and the in 7th The measure taken is that if the image is encoded and encrypted at 506, it can be encoded with several different qualities, as in FIG 8A set out. Accordingly, there will be a new process 702 added to reproduce the image with intermediate qualities as in connection with 8B described.

Also the expiration 528 For forwarding the image to a new recipient, this procedure (described in 6C ) can be used in connection with the in 7-8B may be used in the same manner as described above for the thumbnail / full quality embodiment (except that the data lengths corresponding to the intermediate image qualities, not just the thumbnail quality, are loaded into the new image message).

Regarding 8A describes a sequence for storing the image data so that the image can be displayed in intermediate qualities between the miniature view and the full image. The logic 800 starts at block 802 which on block 610 of 6A follows. At block 608 For example, the system can divide the image samples that are used to generate the image data (see e.g. Table 1) into N grades. The number N of qualities may depend on the number of intermediate stages desired by an administrator of the communication service. In general, for example, at least two image qualities are used (the thumbnail quality and the full image quality). At least some intermediate image quality may also be desirable for use over network connections that may be unstable or relatively slow. Other intermediate grades can be used to provide more granularity - for example, low quality for very bad connections and intermediate or medium quality for better connections that are nonetheless suboptimal.

Optionally, the system can choose between different scan profiles available, or can adapt a scan script to provide different numbers or different configurations of scans. This can be done based on available network and / or client hardware information, as well as based on which scan scripts / configurations result in the lowest overall image data (or the lowest image data for certain image qualities). The system can assign the scans, which are designated in the scan script, between the different N image qualities.

At the blocks 804-806 the system can iteratively scan the image until all scans in the scan script have been performed. At block 808 the system can determine the size of the data according to each quality level. For example, the system can use a progressive JPEG parser to identify where in the image data the dividing lines lie between successive scans. Using this information and the assignment of certain samples to different image qualities in block 802 the system can determine how much data corresponds to each respective sample. These quantities can be in an array (or in some other suitable structure) at block 810 get saved. This array can be used to represent the data lengths in the picture message.

Regarding 8B a procedure for reproducing an image using the in 8A stored intermediate qualities. The logic 850 starts at block 852 that on block 676 of 6B follows. At block 852 the system determines whether the data corresponding to the currently processed quality level has been downloaded in full. For example, at the start of the sequence, the system will download image data that corresponds to the thumbnail quality. If more data remains to be downloaded with the current quality level, processing with Block 854 Continue. The blocks 854-864 generally correspond to the blocks 678-686 of 6B , and the description of these blocks is omitted here to avoid repetition. These blocks are repeated until the downloading of image data for the current quality level via the data stream is stopped.

As soon as you stop downloading data for the current quality level (“Yes” to Block 852 ), processing continues with block 866 away. Here the system receives parameters that describe the status of the network connection (e.g. network throughput, bandwidth, bit rate, rate of dropped packets, etc.). At block 868 the system compares these parameters with previously defined network quality limits for the next quality level. For example, if the system has just stopped processing for the thumbnail quality, the system can evaluate whether the network connection is good enough to download the low quality image data.

If applicable, processing continues with block 854 and the system continues to download records from the image data until the next level of quality is reached. If not applicable, processing continues with block 870 and the system can end the data stream with the blob server. As a result, the system refrains from downloading additional image data for the higher qualities of the image. The system will therefore display the image with the highest downloaded quality, but it can avoid large downloads if the network quality is insufficient.

The embodiments described above may be implemented by a messaging architecture, an example of which will be described next with reference to FIG 9 described.

Messaging architecture

9 illustrates one embodiment of a plurality of servers performing various functions of a messaging service 900 implement that are suitable for use with exemplary embodiments. It will be understood that different distributions of work and functions in different embodiments of a messaging service 900 can be used.

The messaging service 900 can be a domain name front end 902 include. The domain name front end 902 One or more domain names can be assigned to the messaging service 900 are associated in the Domain Name System (DNS). The domain name front end 902 can receive incoming connections and distribute the connections to servers that provide various messaging services.

The messaging service 902 can be one or more chat servers 904 include. The chat server 904 may include front end servers for receiving and transmitting user-to-user messaging updates, such as chat messages. Incoming connections can be made to the chat servers 904 through the domain name front end 902 allocated based on workload balancing.

The messaging service 900 can backend server 908 include. The backend server 908 can perform specialized tasks in support of the chat operations of the frontend chat server 904 carry out. There can be a variety of different types of backend servers 908 be used. It will be traceable to the assignment of the types of tasks to different backend servers 908 may vary in different embodiments. In some embodiments, some of the backend services provided by dedicated servers may be consolidated into a single server or set of servers each performing multiple tasks that, in the embodiments described herein, are shared between different servers. Likewise, in some embodiments, tasks from some of the dedicated backend servers as described herein may be shared between different servers from different server groups.

The messaging service 900 can have one or more offline storage servers 910 include. The one or more offline storage servers 910 can messaging content for messaging clients that are currently offline, which will be held until the messaging clients are reconnected.

The messaging service 900 can be one or more session servers 912 include. The one or more session servers 912 can maintain the session status of connected messaging clients.

The messaging service 900 can be one or more presence servers 914 include. The one or more presence servers 914 can provide presence information for the messaging service 900 care for. Presence information may correspond to user-specific information indicating whether a given user has an online messaging client and is available to chat, has an online messaging client, but is currently removed from it, does not have an online messaging client, and other presence states.

The messaging service 900 can be one or more push storage servers 916 include. The one or more push storage servers 916 can cache push requests and transmit the push requests to the messaging clients. Push requests can be used to wake up messaging clients, notify messaging clients that a messaging update is available, and otherwise perform server-driven interactions with messaging clients.

The messaging service 900 can be one or more group servers 918 include. The one or more group servers 918 can maintain lists of groups, add users to groups, remove users from groups, and receive, cache and forward group chat messages.

The messaging service 900 can be one or more block list servers 920 include. The one or more block list servers 920 can maintain user-specific block lists, the user-specific block lists for inputs for each user specifying the one or more other users who are prohibited from transmitting messages to that user. Alternatively or additionally, the one or more block list servers 920 Maintain user-specific block lists for outlets which specify for each user the one or more other users for whom this user is prohibited from transmitting messages. It will be understandable that blocking lists for inputs and blocking lists for outputs can be stored in a network, for example in a database, whereby the blocking lists for inputs and blocking lists for outputs represent different views of the same pool of blocking information.

The messaging service 900 can be one or more Last Seen Information servers 922 include. The one or more Last Seen Information servers 922 can receive, store, and maintain information including last seen location, status, messaging client, and other elements of a user's last connection with the messaging service 900 specify.

The messaging service 900 can be one or more key servers 924 include. The one or more key servers can host public keys for public / private key encrypted communication.

The messaging service 900 can have one or more profile photo servers 926 include. The one or more profile photo servers 926 can for the wide variety of users of the messaging service 900 Save profile photos and make them available for retrieval.

The messaging service 900 can have one or more spam logging servers 928 include. The one or more spam logging servers 928 can log known and suspected spam (e.g. unwanted messages, especially those with an advertising character). The one or more spam logging servers 928 may be operable to analyze messages to determine if they are spam and, in some embodiments, take repressive action against suspected spammers (users who send spam messages).

The messaging service 900 can be one or more statistics servers 930 include. The one or more statistics servers can provide statistical information related to the operation of the Messaging service 900 and the behavior of the users of the messaging service 900 compile and save.

The messaging service 900 can be one or more web servers 932 include. The one or more web servers 932 can participate in Hypertext Transport Protocol connections (HTTP connections) and Hypertext Transport Protocol Secure connections (HTTPS connections) with web browsers.

The messaging service 900 can have one or more chat activity monitoring servers 934 include. The one or more chat activity monitoring servers 934 can monitor the chats of users to detect unauthorized or improper behavior from users of the messaging service 900 to monitor. The one or more chat activity monitoring servers 934 can interact with the spam logging servers 928 and block list servers 920 operate, the one or more chat activity monitoring servers 934 Identify spam or other unapproved behavior and the spam logging servers 928 Spam information and the block list servers 920 provide blocking information if necessary.

The messaging service 900 can have one or more sync servers 936 include. The one or more sync servers 936 can use the messaging system 500 with contact information from a messaging client, such as an address book in a cell phone, to get contacts for a user in the messaging service 900 to determine.

The messaging service 900 can be one or more multimedia servers 938 include. The one or more multimedia servers can store multimedia (e.g. images, video, audio) that are in transit between messaging clients, multimedia that are cached for offline endpoints, and can perform transcoding on multimedia.

The messaging service 900 can be one or more payment servers 940 include. The one or more payment servers 940 can process payments from users. The one or more payment servers 940 may be connected to external third-party servers to process payments.

The messaging service 900 can be one or more registration servers 942 include. The one or more registration servers 942 can new users of the messaging service 900 to register.

The messaging service 900 can be one or more voice delivery servers 944 include. The one or more voice delivery servers 944 can convey Voice over Internet Protocol (VoIP) voice communication between messaging clients for making VoIP calls.

The methods described above can be contained as instructions in a computer-readable medium or as part of a computing architecture. 10 illustrates one embodiment of an exemplary computing architecture 1000 which is suitable for implementing various embodiments as previously described. In one embodiment, the computing architecture 1000 an electronic device such as a calculator 1001 , contain or be implemented as part of such. The embodiments are not limited in this context.

As used in this application, the terms “system” and “component” are intended to denote a computer-related entity, either hardware, a combination of hardware and software, software or software in execution, examples of these being given by the exemplary computer architecture 1000 to be provided. A component can be, for example, a process running on a processor, a processor, a hard disk drive, multiple storage drives (with optical and / or magnetic storage media), an object, an executable element, an execution thread, a program and / or a computer . For purposes of illustration, both an application running on a server and the server can be a component. There can be one or more components in a process and / or execution thread and a component can be located in a computer and / or distributed between two or more computers. Furthermore, components can be communicatively coupled to one another through various types of communication media in order to coordinate operations. The coordination can include the unidirectional or bidirectional exchange of information. For example, the components can communicate information in the form of signals that are communicated over the communication media. The information can be implemented as signals allocated to different signal lines. With such assignments, every message is a signal. However, other embodiments may alternatively employ data messages. Such data messages can be sent over various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.

The computing architecture 1000 includes various common computing elements, such as one or more processors, multi-core processors, coprocessors, memory units, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input / output components (I / O components), power supplies etc. However, the embodiments are not intended to be implemented by the computational architecture 1000 limited.

As in 10 shown includes the computing architecture 1000 a processing unit 1002 , a system memory 1004 and a system bus 1006 . The processing unit 1002 can be any of various commercially available processors including, without limitation, Athlon®, Duron®, and Opteron® processors from AMD®; ARM® application, embedded and secure processors; DragonBall® and PowerPC® processors from IBM® and Motorola®; IBM and Sony® cell processors; Celeron®, Core (2) -Duo®, Itanium®, Pentium®, Xeon® and XScale® processors from Intel®; and similar processors. Dual microprocessors, multi-core processors and other multi-processor architectures can also be used as processing units 1002 can be used.

The system bus 1006 provides an interface for system components including, but not limited to, system memory 1004 with the processing unit 1002 . The system bus 1006 may be any of several types of bus structure that may be further connected to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using a number of commercially available bus architectures. Interface adapters can be connected to the system bus 1006 be connected via a slot architecture. Sample slot architectures can be used without restriction Accelerated Graphics Port (AGP), Card Bus, (Extended) Industry Standard Architecture ((E) ISA), Micro Channel Architecture (MCA), NuBus, Peripheral Component Interconnect (Extended) (PCI (X)), PCI Express, Personal Computer Memory Card International Association (PCMCIA), and the like.

The computing architecture 1000 may include or implement various articles of manufacture. An article of manufacture may include a computer readable storage medium for storing logic. Examples of computer readable storage medium can include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of logic can be executable computer program instructions implemented using any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like. Embodiments can also be implemented, at least in part, as instructions contained in or on a non-transitory computer-readable medium that can be read and executed by one or more processors to enable the operations described herein to be performed.

The system memory 1004 may include various types of computer readable storage media in the form of one or more high speed storage devices such as read-only memory (ROM), random access memory (RAM), dynamic RAM (DRAM), dual data rate DRAM (DDRAM) ), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, polymer memory such as ferroelectric polymer memory, Ovonic memory, phase change or ferroelectric memory, silicon oxide-nitride-oxide-silicon memory (SONOS memory), magnetic or optical cards, an array of devices such as redundant-array-of-independent-disks drives (RAID Drives), solid-state storage devices (e.g., USB memory devices, solid-state drives (SSD), and other types of storage media that are suitable for storing information are. In the illustrated in 10 The embodiment shown can be the system memory 1004 a non-volatile memory 1008 and / or volatile memory 1010 include. A basic input / output system (BIOS) can be stored in the non-volatile memory 1008 get saved.

The computing architecture 1000 may include various types of computer readable storage media in the form of one or more low speed storage devices, including an internal (or external) hard disk drive (HDD) 1012, a magnetic floppy disk drive (FDD) 1014 for reading from or writing to a removable magnetic floppy disk 1016 and an optical disk drive 1018 for reading from or writing to a removable optical disk 1020 (e.g. a CD-ROM or DVD). The HDD 1012 , FDD 1014 and optical disk drive 1020 can with the system bus 1006 via an HDD interface 1022 , an FDD interface 1024 or an optical drive interface 1026 be connected. The HDD interface 1022 for external drive implementations, it may include at least one or both of the Universal Serial Bus (USB) and IEEE-694 interface technologies.

The drives and associated computer readable media provide volatile and / or non-volatile storage of data, data structures, computational instructions, and so on. For example, a number of program modules can be in the drives and storage units 1008 , 1012 including an operating system 1028 , one or more application programs 1030 , other program modules 1032 and program data 1034 . In one embodiment, the one or more application programs 1030 , other program modules 1032 and program data 1034 for example the various applications and / or components of the messaging system 500 include.

A user can enter commands and information into the calculator 1001 Enter via one or more wired / wireless input devices, for example a keyboard 1036 and a pointing device such as a mouse 1038 . Other input devices can be microphones, infrared remote controls (IR remote controls), radio frequency remote controls (RF remote controls), gamepads, input pens, card readers, copy protection plugs, fingerprint readers, gloves, graphics tablets, joysticks, keyboards, retina readers, touch screens (e.g. capacitive, resistive, etc.), track balls, track pads, sensors, pens, and the like. These and other input devices are often used with the processing unit 1002 via an input device interface 1040 connected to the system bus 1006 but can also be connected via other interfaces, such as a parallel port, an IEEE-694 serial port, a game port, a USB port, an IR interface, etc.

One monitor 1042 or another display device is also on the system bus 1006 through an interface such as a video adapter 1044 , connected. The display 1042 can be inside or outside the computer 1001 be. In addition to the monitor 1042 A computer typically includes other peripheral output devices such as speakers, printers, etc.

The computer 1001 can be in a networked environment using logical connections over wired and / or wireless communications with one or more remote computers, such as a remote computer 1044 , work. The remote computer 1044 may be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other common network node and typically includes many or all of the elements relating to the computer 1001 will be described, although for brevity only a storage / storage device 1046 is illustrated. The illustrated logical connections include wired / wireless connectivity to a local area network (LAN) 1048 and / or larger networks, for example a wide area network (WAN) 1050 . Such LAN and WAN network environments are common in offices and companies and enable company-wide computer networks, such as intranets, which can all be connected to a global communication network, for example the Internet.

When used in a LAN network environment, the calculator will 1001 with the LAN 1048 via a wired and / or wireless communication network interface or adapter 1052 connected. The adapter 1052 can have wired and / or wireless communications with the LAN 1048 enable that also has an access point arranged therein for communicating with the wireless functionality of the adapter 1052 includes.

When used in a WAN network environment, the computer can 1001 a modem 1054 include, or it is with a communication server in the WAN 1050 connected, or it has means for establishing communications over the WAN 1050 on, such as via the Internet. The modem 1054 that can be inside or outside and a wired and / or wireless device is with the system bus 1006 via the input device interface 1040 connected. In a networked environment, program modules relating to the computer 1001 illustrated, or sections thereof, in the remote storage / storage device 1046 get saved. It will be understood that the network connections shown are exemplary and that other means of establishing a communication connection between the computers can be used.

The computer 1001 is operable to communicate with wired and wireless devices or entities using the IEEE 802 suite of standards, such as wireless devices operable in wireless communication (e.g., IEEE 802.13 over-the-air modulation techniques) . This includes at least the wireless technologies WiFi (or Wireless Fidelity), WiMax and Bluetooth ™. The communication can therefore be a predefined structure, as in a conventional network, or simply an ad hoc communication between at least two devices. WiFi networks use radio technologies called IEEE 802.13x (a, b, g, n etc.) to provide secure, reliable, fast wireless connectivity. A WiFi network can be used to connect computers to each other, to the Internet, and to wired networks (which use IEEE 802.3-related media and functions).

11 Fig. 3 is a block diagram showing an exemplary communication architecture 1100 10 that is suitable for implementing various embodiments as previously described. The communication architecture 1100 includes various common communication elements such as transmitters, receivers, transceivers, radio, network interface, baseband processor, antenna, amplifier, filter, power supplies, etc. However, the embodiments are not limited to implementation by the communication architecture 1100 limited.

As in 11 shown includes the communication architecture 1100 one or more clients 1102 and server 1104 . The clients 1102 can the client device 510 to implement. The servers 1104 can use the server device 526 to implement. The clients 1102 and the servers 1104 are with one or more respective client data stores 1106 and server data stores 1108 operably connected, which can be used to provide information locally to the respective clients 1102 and servers 1104 to store such as cookies and / or associated contextual information.

The clients 1102 and the servers 1104 can share information using a communication framework 1110 communicate. The communication framework 1110 can implement any well-known communication technique and protocol. The communication framework 1110 may be a packet-switched network (e.g., public networks such as the Internet, private networks such as a corporate intranet, etc.), a circuit-switched network (e.g., the public switched telephone network), or a combination of a packet-switched network and a circuit-switched network (with suitable gateways and translators).

The communication framework 1110 can implement various network interfaces arranged to accept, communicate with, and connect to a communication network. A network interface can be viewed as a special form of input / output interface. Network interfaces can use connection protocols, including without limitation Direct Connect, Ethernet (e.g. thick, thin, twisted pair -10 / 100/1000-Base-T and the like), Token-Ring, wireless network interfaces, cellular network interfaces, IEEE 802.11ax network interfaces, IEEE 802.16 network interfaces, IEEE 802.20 network interfaces, and the like. In addition, multiple network interfaces can be used to work with different communication network types. For example, multiple network interfaces can be used to allow communication over broadcast, multicast, and unicast networks. Should processing requirements require greater speed and capacity, distributed network controller architectures can be used in a similar manner to those of clients 1102 and the servers 1104 bundle the required communication bandwidth, load balance these or otherwise increase them. A communications network can be any and combination of wired and / or wireless networks including, without limitation, a direct connection, a secured custom connection, a private network (e.g. a corporate intranet), a public network (e.g. the Internet), a Personal Area Network (PAN), a local area network (LAN), a broadband telecommunications network (Metropolitan Area Network, MAN), an Operating-Missions-as-Nodes-on-the-Internet (OMNI), a wide area network (WAN), a wireless network, a cellular network, and other communication networks.

12 illustrates one embodiment of an apparatus 1200 for use in a multi-carrier OFDM system such as the messaging system 500 . The device 1200 can for example Software components 1202 as with regard to messaging component logic 600 , the intent setting logic 700 and the group selection logic 800 described, implement. The device 1200 can also be a logic circuit 1204 to implement. The logic circuit 1204 may include physical circuitry in order for the messaging system 500 perform the operations described. As in 12 shown, the device 1200 a radio interface 1206 , Baseband switching technology 1208 and a computing platform 1210 although embodiments are not limited to this configuration.

The device 1200 may include some or all of the structure and / or some or all of the operations for the messaging system 500 and / or the logic circuit 1204 implement in a single computing entity, such as entirely within a single device. Alternatively, the device 1200 Sections of the structure and / or operations for the messaging system 500 and / or the logic circuit 1204 a distributed system architecture across multiple computing entities, such as using a client-server architecture, a 3-level architecture, an N-level architecture, a tightly coupled or grouped architecture, a peer-to-peer architecture, a Master-slave architecture, a shared database architecture, and other types of distributed systems. The embodiments are not limited in this context.

In one embodiment, the radio interface 1206 comprise a component or combination of components that are adapted to transmit and / or receive modulated single-carrier or multi-carrier signals (e.g. including complementary code keying symbols (CCK symbols) and / or orthogonal frequency symbols Division Multiplexing (OFDM) symbols, although the embodiments are not limited to any specific over-the-air interface or modulation scheme. The radio interface 1206 can for example be a recipient 1212 , a transmitter 1214 and / or a frequency synthesizer 1216 include. The radio interface 1206 can have bias controls, a crystal oscillator and / or one or more antennas 1218 include. In another embodiment, the radio interface 1206 Use external voltage-controlled oscillators (VCOs), surface acoustic wave filters, intermediate frequency filters (IF filters, IF = Intermediate Frequency) and / or RF filters as required. Because of the variety of potential RF interface designs, an expansive description of these is omitted.

The baseband switching technology 1208 can with the radio interface 1206 communicate to receive and / or transmit signals and may, for example, use an analog-to-digital converter 1220 for down-converting received signals and a digital-to-analog converter 1222 for up-converting signals for transmission. Furthermore, the baseband switching technology 1208 a baseband processing circuit or a physical layer (PHY) processing circuit 1224 for PHY link layer processing of respective receive / transmit signals. The baseband switching technology 1208 for example a processing circuit 1226 for media access control (MAC) processing / data link layer processing. The baseband switching technology 1208 can be a memory controller 1228 to communicate with the processing circuit 1226 and / or a computing platform 1210 , for example via one or more interfaces 1230 , include.

In some embodiments, the PHY processing circuit 1224 a frame construction and / or a detection module, in combination with additional box technology, such as a buffer memory, to construct and / or deconstruct communication frames, such as radio frames. Alternatively or additionally, the MAC processing circuit 1226 share the processing for certain of these functions or these processes independently of the PHY processing circuit 1224 carry out. In some embodiments, MAC and PHY processing can be integrated into a single circuit.

The computing platform 1210 can be a computing functionality for the device 1200 provide. As shown, the computing platform 1210 a processing component 1232 include. Additionally or alternatively to the baseband switching technology 1208 can the device 1200 Processing operations or logic for the messaging system 500 and the logic circuit 1204 using the processing component 1232 To run. The processing component 1232 (and / or the PHY 1224 and / or MAC 1226 ) can contain various hardware elements, software elements, or a combination of both. Examples of hardware elements can include devices, logic devices, components, processors, microprocessors, circuits, processor circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices ( PLD), digital signal processors (DSP) freely programmable gate array (FPGA), memory units, logic gates, registers, a semiconductor device, chips, microchips, chipsets, etc. Examples of software elements can be software components, programs, applications, computer programs, application programs, system programs, software development programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, processes, processes, software interfaces, application program interfaces (API), instruction sets, arithmetic code, computer code , Code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether to implement an embodiment using hardware elements and / or software elements will vary according to a number of factors as desired for a given implementation, such as desired computational rate, performance levels, thermal tolerances, processing cycle budget, input data rates, output data rates, memory resources, Data bus speeds and other design or performance limitations.

The computing platform 1210 can also use other platform components 1234 include. Other platform components 1234 include common computing elements, such as one or more processors, multi-core processors, coprocessors, memory units, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input / output components (I / O components) (e.g. Examples of storage devices may include, without limitation, various types of computer-readable and machine-readable storage media in the form of one or more high-speed storage devices, such as read only memory (ROM), random access memory (RAM), dynamic RAM (DRAM), dual data rate -DRAM (DDRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, polymer memory such as ferroelectric Polymer storage, Ovonic storage, phase change or ferroelectrisc memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, magnetic or optical cards, an array of devices such as redundant array-of-independent disks (RAID drives), solid-state storage devices (e.g. B. USB storage devices, solid state drives (SSD), and other types of storage media that are suitable for storing information.

The device 1200 For example, an ultra-mobile device, a mobile device, a fixed device, a machine-to-machine (M2M) device, a personal digital assistant (PDA), a mobile computing device, a smartphone, a telephone, a digital telephone, a cellular telephone User device, eBook readers, handset, one-way pager, two-way pager, messaging device, calculator, personal calculator (PC), desktop calculator, laptop calculator, notebook calculator, netbook calculator, handheld calculator, tablet calculator , a server, a server array or a server farm, a web server, a network server, an Internet server, a workstation, a mini-computer, a mainframe computer, a supercomputer, a network device, a web device, a distributed computing system, multiprocessor systems, processor-based systems, consumer electronics, programmable consumer electronics , Game devices, a television set, a digital television set, a set-top box, a wireless access point, a base station, a node B, an evolved node B (eNB), a subscriber station, a mobile subscriber center, a radio network controller, a router, a hub, a gateway, a bridge, a switch, be a machine or a combination thereof. Accordingly, functions and / or specific configurations of the device described herein 1200 in different embodiments of the device 1200 be included or left out as desired. In some embodiments, the device 1200 configured to be compatible with protocols and frequencies associated with one or more of the 3GPP LTE specifications and / or IEEE 1402.16 standards for WMANs and / or other wireless broadband networks as cited herein, although the Embodiments are not limited in this regard.

Embodiments of the device 1200 can be implemented using single input single output architectures (SISO architectures, SISO = Single Input Single Output). However, certain implementations may have multiple antennas (e.g. antennas 1218 ) for transmission / or reception using adaptive antenna techniques for beam shaping or spatial division multiple access (SDMA) and / or using MIMO communication techniques.

The components and features of the device 1200 can be implemented using any combination of discrete circuitry, application specific integrated circuits (ASICs), logic gates, and / or single chip architectures. Furthermore, the features of the device 1200 using microcontrollers, programmable logic arrays, and / or microprocessors, or any combination of the foregoing, where appropriate. It will pointed out that hardware, firmware and / or software elements may be referred to herein collectively or individually as “logic” or “circuit”.

It will be understandable that the block diagram of 12 exemplary device shown 1200 can be a functionally descriptive example of many potential implementations. Accordingly, division, omission, or inclusion of block functions illustrated in the accompanying figures does not imply that the hardware components, circuits, software, and / or elements for implementing these functions are necessarily divided, omitted, or included in embodiments.

At least one computer-readable storage medium 1236 may include instructions that, when executed, cause a system to perform any of the computer-implemented methods described herein.

General notes on terminology

Some embodiments may be described using the phrase “a single embodiment” or “an embodiment” and its derivatives. These terms mean that a particular feature, structure, or property described in connection with the embodiment is included in at least one single embodiment of the invention. The phrase "in a single embodiment" in various places throughout this specification does not necessarily refer to the same embodiment. In addition, unless otherwise stated, it can be seen that the features described above can be used together in any combination. Therefore, any separately discussed features may be used in combination, unless it is stated that the features are incompatible.

With general reference to the notations and nomenclature used herein, the detailed descriptions herein may be presented in terms of program flows that are executed on a computer or network of computers. These process descriptions and illustrations are used by those skilled in the art to most effectively convey the content of their work to others skilled in the art.

A process is here and generally considered to be a self-consistent sequence of operations that lead to a desired result. These operations are those that require physical manipulations of physical quantities. Usually, but not necessarily, these quantities are in the form of electrical, magnetic, or optical signals that can be stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be noted, however, that all of these and similar terms are intended to be associated with corresponding physical quantities and are merely practical names for these quantities.

Furthermore, the manipulations performed are often referred to using terms such as adding or comparing, which are commonly associated with mental operations performed by a human operator. Such ability of a human operator is in most instances not necessary or desirable in the operations described herein that form part of one or more embodiments. Rather, the operations are machine operations. Useful machines for performing the operations of various embodiments include general purpose digital computers or similar devices.

Some embodiments may be described using the terms “coupled” and “connected” and their derivatives. These terms are not necessarily meant to be synonyms for one another. For example, some embodiments may be described using the terms “connected” and / or “coupled” to indicate that two or more elements are in direct physical or electrical contact with one another. The term “coupled” can, however, also mean that two or more elements are not in direct contact with one another, but nevertheless cooperate or interact with one another.

Various embodiments also relate to devices or systems for performing these operations. This facility may be specially designed for the required purpose or may include a general purpose computer selectively operated by a computer program stored in the computer activated or reconfigured. The procedures presented herein do not inherently relate to any particular computer or other device. Various general purpose machines may be used with programs written in accordance with the teachings described herein, or it may be convenient to build more specialized equipment to perform the required procedural steps. The required structure for a number of these machines will be apparent from the description given.

It is emphasized that the abstract of the disclosure is provided to allow a reader to quickly determine the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, it can be seen from the present detailed description that, for the purpose of streamlining the disclosure, various features are combined into a single embodiment. This method of disclosure is not to be interpreted as an intention that the claimed embodiments require more features than are expressly cited in each claim. Rather, as the following claims show, the inventive subject matter is present in less than all features of a single disclosed embodiment. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment. In the appended claims, the terms “comprising” and “for which” are used as simple equivalents of the respective terms “including” and “where”. In addition, the terms “first”, “second” and “third” etc. are only used as identifiers and are not intended to impose any numerical requirements on your objects.

What has been described above includes examples of the disclosed architecture. It is of course not possible to describe every conceivable combination of components and / or methodologies, but one of ordinary skill in the art will recognize that many other combinations and permutations are possible. Accordingly, the novel architecture is intended to embrace all variations, modifications, and variations that come within the spirit and scope of the appended claims.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 62874554 [0001]

Claims (9)

A process that includes: Receiving, via an end-to-end encrypted session between a sending device and a receiving device, a message comprising a reference to an image; Establishing a connection to a blob storage, wherein the blob storage contains an encrypted thumbnail of the image, first additional encrypted data which, when combined with the encrypted thumbnail, results in a low quality version of the image, and second additional encrypted data which when combined with the encrypted thumbnail and the first additional encrypted data, resulting in a high quality version of the image; Retrieve the encrypted thumbnail from the blob storage, decrypt the thumbnail, and display the thumbnail; Receiving a request to display the image; Retrieving the first additional encrypted data from the blob memory, decrypting the first additional data, and merging the decrypted first additional data with the thumbnail; and displaying the low quality version of the image. Procedure according to Claim 1 wherein the blob store transmits information to the receiving device in a data stream, and further includes: receiving a network quality parameter; Comparing the network quality parameter to a threshold associated with the high quality version of the image; Determining that the network quality parameter does not exceed the threshold; and terminating the data stream to cancel the transmission of the second additional encrypted data. Procedure according to Claim 1 wherein the retrieval of the encrypted thumbnail from the blob memory is performed automatically in direct response to receiving the message. Procedure according to Claim 1 wherein the encrypted thumbnail, the first additional encrypted data, and the second additional encrypted data are stored together in an image structure, and the message comprising the reference to the image includes: a first location in the image structure representing the encrypted thumbnail of dividing the first additional encrypted data, and a second location in the image structure that divides the first additional encrypted data from the second additional encrypted data. Procedure according to Claim 4 further comprising: receiving a request to forward the image to a second recipient; and retransmitting the first and second locations in the image structure while renouncing transcoding of the image is dispensed with. Procedure according to Claim 1 wherein retrieving the encrypted thumbnail from the blob storage, retrieving the first additional encrypted data from the blob storage, and retrieving the second additional encrypted data from the blob storage together in response to a single request for the image referred to in the message. Procedure according to Claim 1 wherein retrieving the encrypted thumbnail includes: downloading a data set that includes at least a portion of the encrypted thumbnail; Receiving a key hash based message authentication code (HMAC) associated with the data set; Computing an HMAC over the downloaded data set; and comparing the calculated HMAC to the received HMAC to verify that the encrypted thumbnail has not been changed. A non-transitory computer readable medium that stores instructions configured to cause a processor to perform the method according to any one of the Claims 1 to 7th or perform: receiving, via an end-to-end encrypted session between a sending device and a receiving device, a message including a reference to an image; Connect to a blob storage, the blob storage having an encrypted thumbnail of the image, first additional encrypted data that when using the encrypted thumbnail merged, resulting in a low quality version of the image and holding second additional encrypted data which, when merged with the encrypted thumbnail and the first additional encrypted data, results in a high quality version of the image; Retrieve the encrypted thumbnail from the blob storage, decrypt the thumbnail, and display the thumbnail; Receiving a request to display the image; Retrieving the first additional encrypted data from the blob memory, decrypting the first additional data, and merging the decrypted first additional data with the thumbnail; and displaying the low quality version of the image. A device comprising: a network interface configured to perform the method according to any one of the Claims 1 to 7th or perform: receiving, via an end-to-end encrypted session between a sending device and a receiving device, a message including a reference to an image; and connecting to a blob storage, wherein the blob storage contains an encrypted thumbnail of the image, first additional encrypted data that, when combined with the encrypted thumbnail, results in a low quality version of the image, and second additional encrypted data that, when combined with the encrypted thumbnail and the first additional encrypted data result in a high quality version of the image, holds; a processor configured to: retrieve the encrypted thumbnail from the blob memory, decrypt the thumbnail, and display the thumbnail; Receiving a request to display the image; and retrieving the first additional encrypted data from the blob memory, decrypting the first additional data, and merging the decrypted first additional data with the thumbnail; and a display configured to display the low quality version of the image.

Images