JP2016531498A - Voice communication using real-time status notification - Google Patents

Abstract

A system, apparatus, graphical user interface, and method for electronic voice communication using status notification are provided. A user of the first portable device executing the communication application selects a multi-function control mechanism that automatically initiates audio recording, and when the control mechanism is released, it is directed to another participant in the active communication session. Audio recording is automatically sent. The same multifunction control mechanism can also be used for sending text messages. The representation of the recording is displayed on the user's device with an indicator that changes the appearance when other participants play the recording. In addition, when one user starts an audio recording that is automatically sent to another user, the other user's device is notified of the start of the recording and the status is displayed to the other user.

Description

  The present invention relates to the field of computers and electronic communications. More particularly, a system, apparatus, graphical user interface, and method for electronic voice communication using status notification are provided.

  Wireless phones become an indispensable accessory and accompany most people wherever they go. Thus, mobile phone users can be contacted almost always via some form of communication. The ability to perform live real-time voice communication is still the main reason that many people carry phones, but many other people can use the phone for non-voice communication rather than real-time voice communication.

  In particular, many people use wireless phones to exchange short text messages via SMS (Short Message Service). SMS messages are typically limited to 140 characters of text and are carried over the control channel of the wireless telephone service provider's voice network. Thus, wireless telephone users are limited in the amount they can communicate at one time, regardless of whether they are sending text messages to one person or many. MMS (Multimedia Messaging Service) was developed to allow media (eg, images, video) to interact with wireless phones, but still relies on SMS protocols to deliver notifications to recipients Yes.

  However, the ability of a wireless telephone user to exchange recorded voice communications with another wireless telephone user is cumbersome even when using MMS. In particular, initiating voice recording (or other audio recording) with a typical wireless telephone requires multiple actions, such as operating multiple control mechanisms in the correct order. For example, a user opens a recording application, starts recording, ends recording, possibly naming it for storage, opens a communication application, creates a message to the desired recipient, and adds attachments You may need to select an option for and then find and attach the record. Even if the source and destination users are running a common communication application, the caller still opens the menu, finds and selects an option to record, starts recording, and stops recording Then it may be necessary to select an option to send it to the recipient.

  In some embodiments of the present invention, systems, devices, graphical user interfaces, and methods for voice communication using real-time status notification are provided. In these embodiments, a user of one portable device can easily create an audio recording quickly and send it to a user of another device. Either or both users may receive real-time notifications of other users' status or records.

  In some embodiments, with one operation of the multifunction control mechanism, a user of one portable electronic device creates an audio recording and sends it to another user. In different situations, the same control mechanism allows the user to send a text message separate from the audio recording. In some implementations, the control mechanism may change the appearance depending on circumstances such as whether the user has entered text into a text window or dialog box.

  In some embodiments, the sender of the audio recording receives a real-time notification that the receiver has started and / or completed playback of the audio recording. In some implementations, representations of audio recordings (eg, in a graphical representation of a user's conversation) or other user representations change appearance to indicate this change in recording status.

  In some embodiments, the recipient of the audio recording may be informed that the sender has started the audio recording. For example, when recording is initiated, a signal can be issued to the recipient's device, and the recipient can view the status before the sender completes the recording.

FIG. 2 is a block diagram of a system that facilitates recorded voice communication using real-time status notifications according to some embodiments of the present invention. FIG. 4 illustrates a user interface of a device that performs voice communication using real-time status notifications according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. FIG. 4 illustrates a graphical user interface for performing voice communication using real-time status notification according to some embodiments of the present invention. 6 is a flowchart illustrating a method for supporting voice communication using real-time status notification according to some embodiments of the present invention. FIG. 2 is a block diagram of a portable device that provides recorded audio communication using real-time status notifications according to some embodiments of the present invention.

  The following description is presented to enable any person skilled in the art to make and use the invention. Various modifications of the disclosed embodiments will be readily apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments and applications without departing from the scope of the present invention. Accordingly, the present invention is not intended to be limited to the embodiments shown.

  In some embodiments of the present invention, systems, devices, user interfaces, and methods for providing voice communication using real-time status notification are provided. In different embodiments, one-touch audio recording, dynamic real-time notification to the communication partner of the start of audio recording, reliable delivery of the recording to the other party, real-time notification of the recording reproduction by the communication partner, and an output device for reproducing the audio recording One or more of a plurality of complementary features, such as automatic selection of

  Embodiments of the present invention allow text, images, audio recordings, and / or other electronic information to be exchanged over a data communication channel that conforms to the IEEE 802.11 standard (eg, Wi-Fi® connection). Implemented within a communications environment. In these embodiments, the same type and / or different depending on the data communication channel, rather than relying on traditional wireless voice communication channels that provide limited support for transmission of instant or real-time communications other than live voice conversations. Allows unlimited exchange of almost any type of communication and media between types of devices.

  Exemplary embodiments of the present invention are described as being implemented as part of a one-to-one communication session (eg, a chat session), but can also be applied to a group communication session (eg, a group chat). It is.

FIG. 1 is a block diagram of a system that facilitates recorded voice communication using real-time status notifications according to some embodiments of the present invention.
In these embodiments, a user of the communication system 120 may use the same or different operating system (e.g., Android (R), BlackBerry (R), iOS (R), Nokia S40 (Nokia). S40), Symbian (registered trademark), Windows Phone (registered trademark)) operating various portable communication devices. Thus, the devices 102, 104 are operated by different users, but may be configured to be similar or very different. Communication session 110 between device 102 and device 104 includes separate connections between device 102 and system 120 and between device 104 and system 120.

  The system 120 is a data center, computer network, or other coordinated collection of computer resources that hosts communication applications or services (or server portions of communication applications) that can be used simultaneously by multiple users. In some implementations, an application can send text from a user's communication device (eg, device 102) to one or more other users operating other communication devices (eg, device 104), text, Provide users with audio recordings, photos, videos, and multimedia instant messaging.

  The communication application hosted by the system 120 corresponds to the devices 102, 104 and includes client logic that runs under the device's native operating system. A user operating a certain type of device and subscribing to a communications server provider can operate other types of devices and exchange messages with users subscribing to other communications service providers. Individual messages can be sent to any number of recipients, and can provide, for example, group forms of communication (eg, chat) in which multiple users interactively exchange instant communication.

  The communication server 122 is a front-end component of the system 120 that hosts user connections with the system and facilitates the exchange of communications between users. Server 122 may be referred to as a “chat” server in that it receives real-time messages and chat session communications from individual users for sending to one or more other users.

  Illustratively, as long as the user is online to the system 120 and the application is running on the user's device, a connection or session may be maintained between the device and the communication server 122. When the user closes the application on the device, the user's connection may last for a period of time (eg, 5 minutes, 15 minutes).

  The state server 124 maintains the state or status of the user and / or the user's device. Thus, the state server stores some or all of the user's profiles, including applicable preferences (eg, preferred audible and / or visual alerts, status messages), device type / model, etc. Some status information maintained by the state server for the user may only persist for the duration of the user's connection with the system 120. As the user's device operates in different ways or models, goes offline, goes online, etc., the state server may capture those changes.

  However, in some embodiments of the invention, the status of the user's device or system connection is not retained. However, in these embodiments, changes in the status of a user or user device may be reported to the system by the device and other users engaged in a communication (eg, chat) session with the user and / or the user. May relay to a user who does not have an open communication session with, but subscribes to status updates for that user.

  In these embodiments, identifiers of other users associated with the user may be identified as part of the user profile maintained on the state server. These other users are notified of some or all changes in the user's status (or changes in the user's device status), such as when they are online and when they are offline. As described below, other status changes can also be communicated to the associated user.

  The message server 126 stores communications that cannot be immediately delivered to the target or destination user. New messages (eg, text chat messages) received for users who are currently online to the system 120 can be delivered immediately (ie, through the user's open connection to the communication server 122). However, a message received for a user who is currently offline (ie, does not have an open connection with the communication server 122) may be stored in the message server 126 until it can be delivered.

  In addition to messages that may be stored on the message server 126, the media server 128 stores media files or content that are awaiting delivery to the destination user. For example, a message from one user to another message can be or include a photo, video, audio recording, or some other media. The media is stored on the media server 128 until it can be delivered and delivery can occur automatically when the destination user is online or only when the destination user requests the media (eg, It can also be initiated (by clicking on a link in a text message delivered to the destination user to inform the destination user of the media content).

  Information identifying the end point of the online user's connection with the system 120 may be stored on the communication / chat server 122, the state server 124, and / or other entities (eg, a session server (not shown in FIG. 1)). . For example, each communication server may maintain a table or other data structure that identifies a user whose connection with the system is terminated at that server or hosted by that server. Another entity (eg, state server 124, session server) may store a larger data set that identifies which communication server is holding the connection for every user connected to the system.

  Thus, when a communication server receives a new communication from a connected user to one or more other users, it performs a lookup in the local data and / or data stored in another system entity to The user's status can be determined, for example, whether it is connected, busy communication, etc., and if so, where it can be identified. In some implementations, a local data table (eg, a routing table) is first searched and a query is issued to another component only if the status of some or all destination users is not found locally.

  Any number (eg, one or more) of communication servers, state servers, message servers, media servers, and / or other servers may be deployed within the system 100. The functionality of the servers 122, 124, 126, 128 may be distributed differently in other embodiments, such as by being integrated into a smaller number of servers or being divided into additional servers.

  The server of system 120 is a separate physical computer system comprising hardware resources (eg, processor, storage, communication interface) and software resources (eg, program code, data) suitable for performing the functions belonging to them. It can be. In some alternative embodiments, any or all servers may be virtual computing machines that share hardware and / or software resources. Accordingly, a plurality of software server modules that perform different functions may exist in one server computer.

  The network 150 comprises a communication path between the devices 102, 104 and the system 120 and may include a data channel and / or a voice communication channel. To carry data communications, the network 150 may include the Internet, one or more intranets, a LAN (Local Area Network), a WAN (Wide Area Network), and / or point-to-point or distributed connections. And / or may be wireless in nature. In order to carry voice communication, the network 150 is managed by different communication service providers and includes the Global System for Mobile Communications (GSM), Enhanced Data Rate for GSM Evolution (EDGE), Long Term Evolution (LTE), It may include wired and / or wireless voice communication links that implement almost any wireless technology such as code division multiple access (CDMA), evolution data optimization (EV-DO), and the like.

  Accordingly, the network 150 may establish, maintain, switch, or otherwise establish routers, gateways, switches, base stations (eg, wireless access points, Node B, eNode B), and / or data and / or voice connections. 1 may include various entities not shown in FIG. 1, such as other devices for performing the operations.

  FIG. 2 illustrates a user interface of a device that performs voice communication using real-time status notification according to some embodiments of the present invention. In these embodiments, device 202 may be a smartphone or portable computing device installed with a communication application hosted by system 120 of FIG. Thus, the operator of device 202 can exchange communications with other users of the application in real time. Those communications may include text, images, audio recordings, and / or other media. Device 202 includes a touch screen display, one or more microphones, and one or more speakers, and may also include other components not referenced herein.

  In FIG. 2, the user or operator of the device 202 has opened a communication application and is presented with a user interface 204 including a session thread 210. Each session thread 210 relates to a separate communication session with one or more users. Some or all threads may include a thumbnail image associated with the user if another user involved in the session provided the thumbnail image 212a of the thread 210a. A thread is the time or date of the last communication exchanged with the user (eg, time 214a of thread 210a) and an excerpt of the last communication (if the communication included text content) or if the communication is short enough Can also identify all communications (eg, excerpt 216a of thread 210a).

  A session thread is for any active communication session in which the user of device 202 is participating and for any session in which the user has participated within an elapsed time period that may or may not be configurable by the user. Enumerated. The user may need to scroll through the thread if the display component of device 202 cannot display all at once. Thread 210 is the most active session (eg, the session with the highest number of messages or the highest average communication rate overall) so that the thread associated with the newest active communication session is at the top. As above, the people selected by the user can be sorted so that they are listed first, or some other way.

  Individual sessions within session thread 210 may be identified by the name of another user participating in the communication session (in the case of a one-to-one session) or in the case of a group communication session. The group session thread can also identify the member who submitted the most recent communication and identify which members of the group are currently online and / or provide other information.

  In addition to the session thread 210, the interface 204 of FIG. 2 includes a footer 220 and a header 240, one or both of which may provide soft control to perform various functions. Exemplary functions include searching (eg, finding threads associated with a particular user or group), communicating (eg, adding a contact, starting a conversation / communication thread with a contact), setting (eg, a user Adjustment of operating parameters of the communication application or device to change the appearance of the interface), status (e.g. setting status to be displayed to the communication partner and / or leisure user), creation of a new group, etc.

  The hard control of device 202 and other soft controls not shown or noted in FIG. 2 may have other functions, such as device type, device manufacturer, device operating system, And / or may depend on other factors. These other functions are not listed here and do not obscure descriptions of features relevant to the present invention.

FIGS. 3A-3H illustrate the use of a graphical user interface for voice communication using real-time status notifications according to some embodiments of the present invention.
In FIG. 3A, a graphical user interface (GUI) 304 is displayed on the touch screen display of the device 302 when the user selects a session thread 210 (eg, thread 210a) from the interface 204 of FIG. The user interfaces 204, 304 can be different parts of the same interface.

  The interface 304 features a conversation area or area 310 that displays recent communications 312 transmitted as part of the current communications session. In particular, if the user previously exchanged any communication during this session or thread, those conversations (or at least the most recent conversation) are displayed or represented in the conversation area 310. The entire text message (or only the initial portion of the long message) is displayed, and audio recordings, images, videos, and / or other communications representations sent or received may also be displayed, as shown in the following figures. If the user has not talked before (or within a certain period of time) or if this is a newly created conversation, region 310 may lack communication.

  The GUI 304 of FIG. 3A features a footer 320, which in the illustrated embodiment includes a multi-function control mechanism 322, a text entry window 324, and an emotion icon tool 326. Emotion icon tool 326 may be activated to obtain a palette of emotions for selection and input into a text message. Text input window 324 may be activated by the user to initiate or resume text message input.

  The multi-function control mechanism 322 of the GUI 304 is presented and operates differently depending on the situation, as further described below. In one situation, such as when the user opens a current communication session as shown in FIG. 3A, the control mechanism 322 displays in one aspect or form, such as a microphone that can be selected (eg, pressed, clicked) Is done.

  In this situation, the user can easily generate a voice or audio recording and send it to other users participating in the session. In particular, by pressing the control mechanism 322 and keeping the pressed state as it is, the microphone of the device 302 is activated and recording starts. Thereafter, when the control mechanism is released, the recording is automatically sent to other participants in the conversation. In particular, the record is sent to the system hosting the application (eg, system 120 of FIG. 1) and subsequently delivered to the recipient. In other situations, the multi-function control mechanism 322 allows text messages to be sent, as described below.

  In the case of a one-to-one session, the interface 304 of FIG. 3A returns thumbnail images 342 of other users participating in the communication session or associated with other users to the header 340 and the interface 204 of FIG. A return 344 is also included.

  In FIG. 3B, the graphical user interface 304 is displayed on the touch screen display component of the device 302 so that it can be viewed after selection of the multifunction control mechanism 322. In the illustrated implementation, selection of the control mechanism includes pressing and maintaining that state (eg, using a finger, stylus, or some other object). When control mechanism 322 is selected, GUI footer 320 is modified to include hint 334, timer 336, and indicator 338.

  As described above, selection of the multifunction control mechanism when presented with the appearance shown in FIG. 3A automatically pulls in the microphone and audio recorder components of device 302. The start of recording can be slightly delayed (eg, 1/2 second, 1 second) in case the control mechanism 322 is accidentally or overselected, and the recording process can be performed without sending anything to the correspondent 306. Be able to give up quickly.

  The hint 334 plays a role of reminding the user that recording can be canceled by horizontally sliding the footer with an object whose control mechanism 322 is pressed. Timer 336 informs the user of the length recorded so far, and indicator 338 provides a visual clue that the recording is in progress. The indicator 338 may be colored (eg, red) and may blink or otherwise move so that recording is in progress and the relative intensity of the audio being captured by the device's microphone. Etc. In some implementations, the multifunction control 322 takes a slightly altered appearance when pressed, such as a darker (or lighter) shade.

  FIG. 3C shows the GUI 304 of FIG. 3B after the operator of the device 302 releases the multifunction control mechanism 322 and the interface returns to a state similar to that shown in FIG. 3A. A communication 312a is added to the conversation area 310 to represent the audio recording. In addition, when the operator deselects the multifunction control 322 (eg, by releasing it), the recording is closed and the audio recording file is communicated to the graphical user interface 304 for delivery to the recipient. Uploaded to the communication system hosting / supporting the application.

  The communication 312a includes a control mechanism for playing back (and alternately pausing) the recording, a time field identifying the duration of the recording, and a slider for quick access to the desired portion of the recording (during operation of this control mechanism, The time field may include various control mechanisms associated with audio recording, such as (which may indicate the current time position within the recording).

  However, the communication 312a also accompanies the record creator's thumbnail or representation 314a. In a one-on-one chat or communication session such as that shown in FIG. 3C, each communicator typically appears in the conversation area 310 and records that he did not create must have been created by other communicators. know. However, in a group communication session, the sender representation 314a may allow a group member to easily identify who originated the associated record. In addition, the representation 314a may change over time based on the status of the recording (eg, in addition to or as an alternative to the status indicator changes described immediately below).

  The communication 312a is also accompanied by a status indicator 316a and a delivery indicator 318a. Delivery indicator 318a indicates when communication 312a has been delivered to other members of the communication session. For illustration purposes, a single check mark (or other symbol) indicates that communication has been dispatched from device 302 to the communication system. A pair of check marks (or other symbols) indicates that the communication has been delivered to the other device, as shown for the previous communication 312.

  However, the status indicator 316a allows the operator of the device 302 to quickly identify whether the recipient has played the recording. When the recipient begins playing the recording, the recipient's device notifies the system, the system notifies device 302, and device 302 changes or replaces indicator 316a. In some implementations, the indicator 316a changes color (eg, from gray to blue), and in other implementations it changes size or shape (eg, becomes a speaker instead of a microphone). In some other implementations, it disappears. In still other implementations, the status indicator 316a can be changed in other or multiple ways, the sender representation 314a can be changed, and / or other changes can be made to the communication 312a, and the recording can be played back by the recipient. You can let them know.

  In a group communication session, the status indicator 316a may change appearance for any number of times a group member plays the recording (eg, to indicate the number or percentage of recipients that have played the recording), and a separate status indicator May be used for different members, the operator may select an indicator to view a list of group members who have played (or have not played) the recording, etc.

  Updates to status indicator 316a (and / or other indicators and status described herein) may or may not be reliable. In other words, when a recipient of an audio recording associated with communication 312a plays the recording, the action notification is buffered in the communication system, even if device 302 is offline or turned off. Can be delivered to the device 302 so that the indicator can be updated when the device 302 is brought back online.

  FIG. 3D shows the GUI 304 of FIG. 3A when the operator or user of the device 302 selects the text entry window 324 to create a text message. The conversation area 310 contracts to fit the appearance of the keyboard 350 used when typing a message, and the cursor is placed in the text entry window. The multi-function control mechanism 322 retains the initial or initial appearance and can be used as described above to create and send an audio recording as an alternative or addition to sending a text message.

  FIG. 3E shows the GUI 304 of FIG. 3D when the operator of the device 302 presses one or more keys on the keyboard or keypad 350. In addition to displaying text corresponding to the user's input in the text input window 324, the multi-function control mechanism 322 changes to the second form or appearance, and at this point, the text containing the text in the window 324 Indicates that it can be used to send a message.

  In different embodiments of the invention, the multi-function mechanism 322 may take virtually any form or appearance (eg, one or more symbols in place of the word “send”) and / or depending on the situation being displayed. Can provide different functions. In the illustrated example, the first situation is characterized by no text entered by the operator of the device, while the second situation is characterized by the presence of such text.

  However, in some embodiments, the control mechanism 322 may still be usable to create an audio recording, even after the user has entered text into the text entry window 324, changing its appearance. It may or may not be changed. In these embodiments, a brief selection of the control mechanism 322 (eg, pressing and releasing the control mechanism 322) may cause the text in the text entry window to be sent as a text message, after which the user interface can It may return to a state such as the state shown in FIG. 3A or 3D (with a new text message displayed in area 310).

  However, in these embodiments, the control mechanism 322 is selected longer (eg, depressing the control mechanism 322 and keeping that state), thereby recording after a predetermined time period (eg, 1 second, 2 seconds). Let it begin. Release of the control mechanism can cause the record to be forwarded to the recipient, as described above, and the text in the text entry window to be sent as a text message. Alternatively, after the recording is sent, the interface returns to the state before the generation of the audio recording (eg, as shown in FIG. 3E), except that a new communication of the audio recording appears in area 310. So that the user can complete the text entry. In these embodiments, during recording creation, the user can still cancel the recording, such as by sliding from the control mechanism 322 horizontally, vertically, or in other directions while the control mechanism 322 is selected. It may be possible.

  Returning to the discussion of the embodiment of the invention shown in FIG. 3E, if the user activates the control mechanism 322 after creating a text message in the window 324, the message is sent and displayed in the conversation area 310. (Or a message representation is displayed). The keyboard 350 can be closed or can remain open, allowing the generation of additional text.

  FIG. 3F shows the GUI 304 of FIG. 3D when the operator of the device 302 activates the multi-function control mechanism 322 without or before pressing a key on the keyboard 350. As described in conjunction with FIG. 3D, audio recording begins, but can be canceled as advised by hint 334.

  FIG. 3G shows the GUI 304 of FIG. 3F after the control mechanism 322 has been deselected and a new audio recording has been transferred to the recipient. A new communication 312b is added to the conversation area 310 and is accompanied by or includes a sender representation 314b, a status indicator 316b and a delivery indicator 318b, which are similar to the purposes described above in connection with FIG. 3C. Serve purpose. In this case, a double check mark on indicator 318b may indicate that the audio recording has been delivered to the recipient's device.

  FIG. 3H shows the GUI 304 of FIG. 3C (or FIG. 3G) after the recipient has started playing the communication 312a (or communication 312b) and a real-time notification of the action has been sent to the device 302. In those embodiments of the invention, the status indicator 316a changes color to alert the operator of the device 302 that the recording has been played or is playing. A color change (eg, from gray to blue) may not be obvious in FIG. 3H due to the black and white nature of the figure, but is noticeable in the color display of device 302. Alternatively, some other characteristic (eg, size, shape, fill) of status indicator 316a may be changed.

  4A-4D illustrate a graphical user interface for real-time status notification and voice communication according to some embodiments of the present invention. In these embodiments, the device 402 is operated by other participants in the communication session reflected in FIGS. 3A-3H, in other words, recipients of an audio recording sent by the operator of the device 302. A smartphone or portable communication device. Device 402 includes a touch screen display and a microphone and may be similar to device 302 in that it executes the same communication application. The devices may or may not be the same type or model, or may or may not run the same operating system.

  The graphical user interface 404 displayed on the touch screen display of the device 402 includes a multi-function control mechanism 422, a text entry window 424, and an emotion icon tool 426, which may function as described above. However, interface 404 also includes status 448 of the other party to the conversation in FIG. 4A, here “Recording Audio ...”. Thus, the operator of device 302 is currently recording (or was recording until just before) the audio to be transmitted to device 402.

  In the illustrated embodiment of the present invention, when one member of a communication session initiates audio recording (eg, via the multi-function control mechanism provided herein), the other party's behavior is real-time. To be notified. Before changing to status 448 indicating that the operator of another device is recording audio, the status was “online” or “last seen at <specific time>”, etc. You can show that.

  Status 448 may remain displayed until a different status is known. For example, if an operator at another device completes the recording (and the recording is being sent to device 402), the status may change accordingly. The device user may or may not be able to select messages or information displayed as status 448 on other users' devices or create a custom message.

  FIG. 4B shows the GUI 404 of FIG. 4A when the audio recording that was recorded by the communication partner of the operator of the communication 412a, device 402 has been downloaded. In some implementations, the recording may be downloaded automatically when the device 402 is online. In other implementations, the operator of device 402 may be notified that the recording is available (eg, via a link displayed in conversation area 410 identified in FIG. 4A), so download it. To act (for example, select a link). It may be noted that the status 448 has been changed to reflect the current status of other users.

  The communication 412a may include symbols and / or control mechanisms that indicate the status of the download, such as a slider that reflects the progress of the download, the duration of the recording, a control mechanism that cancels the download, and the like. The communication 412a also includes a sender representation 314a and a status indicator 416a that indicate whether the recording was played on the receiving device.

  In some implementations, the appearance of status indicators such as indicators 316a, 416a may differ between the sender and receiver of the record and between different device platforms. For example, status indicators can have different colors, different shapes, animations, and so on. However, the status indicator is presented and changes in some way when the recipient begins or ends playback of the associated recording.

  FIG. 4C shows the GUI 404 of FIG. 4B after the download of the audio recording represented by the communication 412a is complete and can be played (eg, using a control mechanism embedded within the communication 412a). The status indicator 416a has not been changed since the operator of the device 402 has not yet played back the recording.

  In FIG. 4D, the operator of device 402 is playing (or at least starting playing) an audio recording. Therefore, the status indicator 416a changes its appearance by changing its appearance, in this case by changing its color. In other implementations, the indicator 416a may undergo more significant changes as described above. The change in color (from green to blue) reflected in FIG. 4D may not be clear due to the black and white nature of the diagram.

  FIG. 4E shows the GUI 404 of FIG. 4D when the user of the device 402 selects the communication 412a (eg, when pressed and kept in that state). If this action is detected, a menu 460 is displayed on the touch screen display to allow the user to take some action regarding the communication (and associated records). For example, as shown in FIG. 4E, an option to delete or transfer the record may be provided. If the user wants to respond to the communication 412a with his record, as described above, he can activate the multi-function control mechanism 422, and the record is automatically created and sent with a single touch on the control mechanism 422. Is done.

  FIG. 5 is a flowchart illustrating a method for supporting voice communication using real-time status notification according to some embodiments of the present invention. In the illustrated method, a user of device 102 engages in a chat session with another person, the user of device 104. The illustrated method may be modified to support voice communication within a group communication session without exceeding the scope of the present invention.

  In operation 500, the user of device 102 opens a communication application on the device and opens (or creates) a conversation with the user of device 104. A user of device 102 presses a multi-function control mechanism (eg, control mechanism 322 in FIG. 3A) to initiate voice or other audio recording. It should be noted that some text related to audio recording can also be entered into the text input window, and the text will be recorded before audio recording (eg, before entering recording), during audio recording (eg, Sent with recording) or after audio recording (eg, input after recording complete).

  When recording is started, a communication application running on the device 102 issues a temporary signal to the communication system 120. This signal differs from a normal chat message sent by an application as part of a communication session in that it is not delivered to the device 104 as a message. Instead, it is intended to be consumed by application logic as described herein.

  In operation 502, the communication system receives a real-time status notification regarding the start of recording and relays the notification to the device 104. In some implementations, the notification may be retained in the system 120 for a length sufficient to relay to the device 104 (or until its acknowledgment is received from the device 104) and then discarded. In other implementations, the status may be retained until replaced with a new status of device 102. In the embodiment of the invention reflected in FIG. 5, the current conversation (i.e., audio recording was initiated) even if the user of device 102 has an active or open communication thread or session with other users. Only the user involved in the conversation is notified of the start of recording.

  In operation 504, the device 104 receives a notification of the audio recording at the device 102 and updates the display of the associated conversation to indicate the status. For purposes of illustration, the view of the conversation at device 104 may be changed to indicate “recording audio”, “recording”, or some other indication of the action being performed at device 102.

  If no conversation is currently displayed on the device 104, but the communication application is running, the status notification may or may not be shown. For example, if the device 104 is currently displaying multiple conversation thumbnails (eg, as shown in FIG. 2), the conversation thumbnails (or icons or other parts of the thumbnails) will flash, change color, Or it may have an altered appearance in other ways to indicate audio activity associated with the conversation. Or, if the device 104 is currently displaying another conversation, the header or other part of the conversation identifies the user of the device 102 and indicates that the user of the device 102 is recording audio ( (E.g., using printed notifications, tones, icons). For example, the ticker display may scroll the header, some of which may be temporarily changed to display the status of other conversations, other user thumbnails may be displayed with audible alerts, and so on. However, if the user of device 104 is currently recording audio or is playing audio from another conversation, an audible alert may not be sounded, or all of the status of device 102 is displayed. It does not have to be.

  If the communication application is not currently running on device 104, the notification may be discarded. If the application is running in the background, the notification may be consumed, but the user of the device 104 will only see the application if it brings the application to the foreground.

  In operation 510, the user of device 102 records audio (eg, voice, music, sound effects) and releases the multifunction control mechanism. When the control mechanism is released, the device 102 (ie, the communications application) issues a new temporary signal to the system 120 to report the status change.

In operation 512, the communication system 120 receives the notification, processes it in the same manner as the notification in operation 502, and relays it to the device 104.
In operation 514, the device 104 receives the notification and updates the display to reflect the new status if possible (eg, if the application is operating in the foreground of the device). For illustration, if the user is currently viewing a conversation with the user of the device 102, a status indicator (e.g., display) to "Online", "Idle", "Writing", or something else Change in the header).

  In operation 520, the device 102 uploads the audio recording to the communication system 120 (eg, media server). The device may also send any text entered by the user of the device 102 before and / or after the creation of the record. The recording may be transcoded, compressed, and / or otherwise processed before or after upload.

  In operation 522, the system 120 may store the record, report or provide it to the device 104, and deliver any text that may accompany the record. In some implementations, the system may attempt to deliver the record immediately if the device 104 is online. In other implementations, the system may deliver a message that appears in a conversation displayed on the device 104, the message may include any text associated with the recording, and a user of the device 104 may initiate the recording download. Like that. When the recording is downloaded to the device 104, a control mechanism is inserted into the conversation to start its playback. Thus, the user of the device 104 can start playback by pressing one icon or control mechanism.

  In operation 530, the user starts playback of the recording at device 104. Thus, the communication application sends a temporary signal to the system 120 to report this status change.

In operation 532, the system receives the notification, processes it similar to the notification from device 102, and relays it to device 102.
In operation 534, the device 102 receives the notification and updates the status of the user's conversation if possible (eg, if the application is running in the foreground of the device 102). For example, an identifier or a portion of a thumbnail (eg, an icon) representing a user of device 104 can change color, change to a different shape, a check mark or other symbol can be displayed, and so forth.

  In optional operation 540, the user of device 104 moves the device closer to or away from the user during playback of the recording, and playback is advanced through different output components. For example, a first speaker may be used, such as a speaker that is activated when the device is in speakerphone mode of operation while the device is relatively far from the user. If the device is located close to the user (eg, when the user makes a phone call using the device), the audio output is automatically directed to the speaker used when the user makes the phone call instead of the speakerphone mode. Can be switched automatically.

  In optional operation 550, the user of device 102 creates a text message to the user of device 104, such as a follow-up message to the audio recording, and the same multiple activated to create and send the audio recording. Send using functional tools. Thus, in some embodiments of the present invention, a single context sensitive control mechanism is used to transmit both text and recorded audio within a communication session.

FIG. 6 is a block diagram of a portable device that performs audio communication using real-time status notifications according to some embodiments of the present invention.
Device 602 may include one or more processing units or processors 602, touch screen display 604, soft and / or hard input control mechanism 606 (eg, keypad, keyboard, mouse, trackball), auxiliary storage 608, wireless transmission 610 (coupled to antenna 610 a), optional additional input / output ports 612, and memory 618, all coupled via bus 614. The wireless transceiver 610 may have a permanent communication connection to a wireless communication service provider (eg, a wireless telco network).

  Memory 618 stores programs and / or other instructions executed by processor 602. These programs include an operating system 630 that manages basic device services and performs hardware dependent functions and tasks.

  Memory 618 also stores a plurality of program modules or processor-executable instruction sets that support and / or implement a method for exchanging voice communications using real-time notifications. In particular, the memory 618 includes a communication program 620 that performs real-time communication via text and / or media. Program 620 may be provided by an operator of a communication system such as system 120 of FIG.

  As part of or in addition to communication application 620, memory 618 stores graphical user interface 622, notification logic 624, communication 626 (including audio recording 628), and the like.

  The GUI 622 displays and manages various content on the touch screen display 604, including a multi-function control mechanism that can initiate one or both of recorded audio messages and text messages, change the appearance in different situations, and inform the function. . The GUI 622 changes the appearance when the associated audio recording is played, and a communication status indicator that may further change the appearance for other reasons (eg, the recipient transfers or deletes the audio recording). In addition. Further, the GUI 622 includes a communication expression for displaying a communication partner's status (for example, indicating when the partner is recording audio), a thumbnail image of the communication partner, etc. in the conversation area of the display 604.

  Notification logic 624, when executed by processor 602, issues and / or consumes real-time notifications. For example, logic 624 may issue a notification to another device when an operator of device 602 records audio to be sent to a user of another device, and sends another notification when recording is complete. In addition, if the operator of device 602 plays a recording received from another device, another notification may be sent.

  Communication 626 includes at least some of the communication exchanged between an operator of device 602 and an operator's communication / chat session participant. Other (eg, older) communications may be stored in auxiliary storage 608 until needed (eg, when the device operator scrolls through communications).

  The environment in which some embodiments of the invention are implemented may incorporate a general purpose computer or a dedicated device, such as a handheld computer or a communication device. In particular, a device referred to herein (eg, a portable telecommunications device) may be a smartphone, laptop or tablet computer, or some other portable computing and / or communication device. Some details of such devices (eg, processor, memory, data storage, display) may be omitted for clarity. A component such as a processor or memory to which one or more tasks or functions belong may be a general purpose component that is temporarily configured to perform a specified task or function, or a task or function It may be a specific component manufactured to perform The term “processor” as used herein refers to one or more electronic circuits, devices, chips, processing cores, and / or others configured to process data and / or computer program code. Refers to the components of A wireless communication facility may provide support for one or more wireless networks (eg, Wi-Fi®, 3G cellular telephone).

  The data structures and program code described in this detailed description are typically stored on a non-transitory computer readable storage medium, which can store code and / or data used by a computer system. Device or medium. Non-transitory computer readable storage media include volatile memory, non-volatile memory, disk drives, magnetic tapes, CDs (compact disks), and DVDs (digital versatile or digital video disks), solid state drives and other magnetic and Including, but not limited to, optical storage devices and / or other non-transitory computer readable media currently known or later developed.

  The methods and processes described in the detailed description can be implemented as code and / or data, which can be stored on a non-transitory computer-readable storage medium, as described above. When a processor or computer system reads and executes code, and manipulates data stored on the medium, it executes methods and processes implemented as code and data structures and stored in the medium.

  The foregoing descriptions of embodiments of the present invention have been presented only for purposes of illustration and description. This is not intended to be exhaustive, i.e., to limit the invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in this art. The scope of the invention is defined by the appended claims rather than the foregoing disclosure.

Claims (40)

A wireless communication function, a touch screen display that supports a graphical user interface, a microphone that can be used for audio recording, a memory, and one or more that execute one or more programs stored in the memory A portable telecommunications device comprising:
The graphical user interface is:
A conversation area displayed on the touch screen display and filled in when communication occurs between the portable telecommunications device using the wireless communication function and another device;
A selectable control mechanism displayed on the touch screen display,
Selection by the user of the control mechanism in a first situation initiates transmission of an audio recording to the another device using the wireless communication function;
A portable telecommunication device wherein selection by the user of the control mechanism in a second situation initiates transmission of a text message to the other device using the wireless communication function.   The portable telecommunications device of claim 1, wherein the graphical user interface further comprises a header region configured to identify an operator of the another device. The graphical user interface further comprises a text entry window for creating the text message,
The portable telecommunications device according to claim 1 or 2, wherein a form of the control mechanism is automatically changed when text is entered into the text entry window. The graphical user interface further comprises a text entry window;
The first situation is characterized in that there is no text in the text entry window;
The portable telecommunications device according to any one of claims 1 to 3, characterized in that the second situation has text in the text entry window. The selection of the control mechanism initiates the creation of the audio recording using the microphone;
The portable telecommunications device according to any one of claims 1 to 4, wherein deselection of the control mechanism causes the audio recording to be automatically transmitted to the other device. Selection of the control mechanism includes pressing of the control mechanism;
The portable telecommunications device of claim 5, wherein deselecting the control mechanism includes releasing the control mechanism. Selection of the control mechanism over at least a predetermined duration causes audio to be recorded and automatically transmits the audio recording upon deselection of the control mechanism;
The portable telecommunications device according to any one of claims 1 to 6, wherein selection of the control mechanism in less than the predetermined duration causes the text message to be automatically transmitted upon deselection of the control mechanism.   The portable telecommunications device of claim 7, wherein the text message is entered via a text entry window of the graphical user interface prior to selection of the control mechanism in less than the predetermined duration. Further comprising representing the audio recording in the conversation area after transmitting the audio recording to the another device;
9. The representation of claim 1-8, wherein the representation has a first form prior to playback of the audio recording on the another device and a second form after playback of the audio recording on the another device. A portable telecommunications device according to any one of the preceding claims.   10. The portable telecommunications device of claim 9, wherein the representation is changed from the first form to the second form prior to completion of playback of the audio recording at the another device. The graphical user interface further comprises a status field configured to display a status of the another device;
11. The status is changed to indicate audio recording when a new audio recording addressed to the portable telecommunications device is initiated at the other device. A portable telecommunications device as described in. The graphical user interface further includes a visual alert regarding cancellation of the audio recording during selection of the control mechanism in the first situation;
12. A portable telecommunications device according to any one of the preceding claims, wherein movement away from the control mechanism during selection of the control mechanism cancels the audio recording. In a portable device with a touch screen display,
Displaying a conversation area on the touch screen display that reflects a conversation between an operator of the portable device and an operator of another device;
Displaying a selectable control mechanism on the touch screen display;
Initiating transmission of an audio recording to the other device when the control mechanism is selected in a first situation;
A method comprising initiating transmission of a text message to the another device when the control mechanism is deselected in a second situation.   14. The method of claim 13, wherein there is no communication exchange in the conversation area initially, and a communication exchange between an operator of the portable device and an operator of the other device is entered when a communication occurs.   15. A method according to claim 13 or 14, wherein the conversation area comprises one or more communication interactions between an operator of the portable device and an operator of the other device. The portable device further comprises a microphone;
Initiating transmission of the audio recording to the other device is
Activating the microphone to record the audio recording;
16. A method according to any one of claims 13 to 15, comprising automatically transmitting the audio recording to the another device when the control mechanism is deselected.   Further comprising displaying a representation of the audio recording in the conversation area after transmitting the audio recording, the representation including a first feature indicating that the audio recording is not being played on the other device. The method of claim 16. After displaying the expression,
Receiving a notification that playback of the audio recording has started on the other device;
The method of claim 17, further comprising modifying the first feature to indicate that at least a portion of the audio recording has been played on the another device. Starting to send a text message to the other device is:
19. A method according to any one of claims 13-18, comprising sending the text message towards the another device when the control mechanism is deselected. Before sending the text message,
Displaying a text entry window on the touchscreen display;
20. The method of claim 19, further comprising entering text corresponding to operation of a control mechanism by an operator of the portable device into the text entry window. Before sending the text message,
21. The method of claim 20, further comprising changing the appearance of the selectable control mechanism when the text is entered into the text entry window. In the first situation, no text is entered in the text entry window displayed on the touch screen display,
The method according to any one of claims 13 to 21, wherein in the second situation, text is entered in the text entry window.   23. The method according to any one of claims 13 to 22, further comprising displaying a header area on the touch screen display that identifies an operator of the another device. Displaying a status field configured to indicate a status of the another device on the touch screen display;
24. A method according to any one of claims 13 to 23, wherein when a new audio recording is initiated at the another device, the status is changed to indicate an audio recording. Receiving a new record from said another device;
In response to an operation of a playback control mechanism by an operator of the portable device, starting playback of the new recording via a first output component of the portable device;
25. A method according to any one of claims 13 to 24, further comprising switching playback of the new recording to a second output component in response to detecting proximity of the operator of the portable device.   26. A visual alert regarding cancellation of the audio recording is further displayed near the control mechanism of the touch screen display during selection of the control mechanism in the first situation. The method according to one item.   27. The method of claim 26, further comprising canceling the audio recording in response to detecting movement away from the control mechanism in the direction of the visual alert during selection of the control mechanism in the first situation. Method. A method of operating a portable device having a touch screen display and a microphone, the method comprising:
Displaying on the touch screen display a conversation area in which a communication exchange between an operator of the portable device and an operator of another device is filled in when communication occurs;
Displaying a text entry window on the touchscreen display;
Displaying a selectable control mechanism on the touch screen display;
When the selectable control mechanism is selected,
If the control mechanism is selected for a predetermined duration,
Starting audio recording using the microphone;
Transmitting the audio recording to the other device upon deselection of the control mechanism;
A method comprising sending a text message previously created in the text entry window to the another device if the control mechanism is selected in less than the predetermined duration.   29. A computer program comprising program instructions that, when executed by one or more processors of a communication device, perform all the steps of the method according to any one of claims 13-28.   30. The computer program of claim 29, comprising a graphical user interface. When executed by a portable device having a touch screen display,
Displaying a conversation area reflecting a conversation between an operator of the portable device and an operator of another device on the touch screen display;
Displaying a selectable control mechanism on the touch screen display;
Initiating transmission of an audio recording to the other device when the control mechanism is selected in a first situation;
A non-transitory computer readable medium having stored thereon instructions for executing transmission of a text message to the another device when the control mechanism is selected in a second situation. A portable device,
Touch screen display,
One or more processors;
A memory for storing instructions to be executed by the one or more processors, the instructions in the one or more processors,
Displaying a conversation between an operator of the portable device and an operator of another device on the touch screen display;
Displaying a selectable control mechanism on the touch screen display;
Initiating transmission of an audio recording to the other device when the control mechanism is selected in a first situation;
A portable device configured to cause a start of sending a text message to the another device when the control mechanism is selected in a second situation.   The portable device of claim 32, further comprising a microphone. The instructions are further configured to cause the one or more processors to execute displaying a text input window that creates the text message;
34. A portable device according to claim 32 or 33, wherein the form of the control mechanism is changed when text is entered into the text entry window. The instructions are further configured to cause the one or more processors to execute displaying a text entry window on the touch screen display;
In the first situation, there is no text in the text entry window,
35. A portable device according to any one of claims 32-34, wherein there is text in the text entry window in the second situation. Selection of the control mechanism initiates creation of the audio recording;
36. A portable device according to any one of claims 32-35, wherein deselecting the control mechanism causes the audio recording to be transmitted to the another device. Selection of the control mechanism over at least a predetermined duration causes audio to be recorded and automatically transmits the audio recording upon deselection of the control mechanism;
37. A portable device according to any one of claims 32-36, wherein selection of the control mechanism in less than the predetermined duration causes the text message to be automatically transmitted upon deselection of the control mechanism. The instructions are further configured to cause the one or more processors to execute displaying a representation of the audio recording;
38. The representation of claims 32-37, wherein the representation has a first form prior to playback of the audio recording on the another device and a second form after playback of the audio recording on the another device. The portable device according to any one of the above. The instructions are further configured to cause the one or more processors to execute displaying a status field on the touch screen display indicating a status of the another device;
39. A portable device according to any one of claims 32-38, wherein the status is changed to indicate an audio recording when a new audio recording is initiated at the other device. The instructions are further configured to cause the one or more processors to execute displaying a visual alert on the touch screen display regarding cancellation of the audio recording during selection of the control mechanism in the first situation. Configured,
40. A portable device according to any one of claims 32 to 39, wherein movement away from the control mechanism during selection of the control mechanism cancels the audio recording.