JP2004510217A - Electronic chat and instant messaging with ink data - Google Patents

Abstract

Disclosed are methods and apparatus for implementing ink data communication between multiple parties using a computing and / or communication device over a network. A hardware interface layer for receiving raw ink data from the input device, an ink management layer for processing the raw ink data into a form usable by the chat interface layer, and processed ink data A network interface layer for transferring data across a network to a chat interface layer of a remote computer.

Description

[0001]
(Field of the Invention)
The present invention relates to the field of communications using computing and / or communication devices. More specifically, the present invention relates to ink data communication between multiple parties using a computing and / or communication device over a network.
[0002]
(Background of the Invention)
Chat room and online services from Internet service providers provide a place for informal public meetings where multiple participants from around the world can talk on text, audio, or shared whiteboards. Is provided. The chat room may be maintained by a single server or multiple servers spread over a wide area connected by a network.
[0003]
A typical chat room service relies on two basic components: a transmission control protocol / Internet Protocol (TCP / IP), the underlying networking protocol of the Internet: a client and a server. The client runs software that allows a connection to the server. In other implementations, peer-to-peer chat communication is possible. In a client / server implementation, the server accepts connections from one or more clients simultaneously. Typically, multiple servers are interconnected. Clients can access conferences and users on other servers from one server. Multiple servers provide a support structure that allows the chat room service to operate. The server maintains information on currently available chat rooms. Each time a new room is created, information about it is passed to every other server on the network. The server also manages which clients are currently connected and which options and features they are setting up. All of this information is exchanged between servers as changes occur. A typical chat system, Internet Relay Chat (IRC), is described in Internet RFC 1459 “Internet Relay Chat Protocol”.
[0004]
Current chat room services do not provide satisfactory support for non-ASCII characters, and communication in languages other than English, such as ideographs, is difficult. Furthermore, current chat room services do not provide satisfactory support for sharing graphic or ink data messages between clients. Conventional ink data is a single set of information about the movement of a pen device through an electronic tablet.
[0005]
In the past, some services have allowed multiple clients to send graphic messages between clients using a shared whiteboard. Each client system was equipped with a video camera pointed to a whiteboard at the client site. The image captured by the video camera is sent to the server. The server superimposes the images received by the respective video cameras on each other and sends them to the respective clients for displaying the new images. One such system, Xerox PARC's Video Draw, is described in John C., published in CHI '90 Conference Proceedings. Tang and Scott L.S. Video Draw by Minneman: A Video Interface For Collaborative Drawing.
[0006]
There are several drawbacks to using a shared white board. One of the disadvantages of using a shared whiteboard is that linear conversations through shared graphic space cannot be easily performed. When multiple clients share a whiteboard, it is difficult for one client to determine which client is posting to the shared whiteboard, and the order in which each was posted Was difficult to track. Another disadvantage of using a shared whiteboard is that when the whiteboard is full, clients must wait for other clients to erase their content on the whiteboard and make other posts available Was that.
[0007]
One special case of chat use is conventional instant messaging, where one client can immediately send and receive messages to and from other clients. Typically, a short text message is immediately transferred between two networked users using instant messaging (IM). Long or complex messages or documents can be transferred between networked users in a less timely manner using conventional email. Conventional email systems support the transfer of documents in a variety of formats, including text, graphics, GIF, JPEG, bitmap, EXE, and many other formats. However, conventional IM and chat infrastructures are further limited to a few data formats, primarily ASCII text and / or audio formats.
[0008]
As the IM and chat user base grows, it becomes increasingly important to support data types other than simple ASCII text and / or voice. One such data type not currently supported by conventional IM or chat systems is the handwritten data type, such as ink data. The use of ink data is particularly important in applications where traditional text data is not very efficient. For example, languages other than English often use symbols that are more efficient for handwriting than for text-based keyboard input. As another example, some portable devices, such as personal digital assistants (PDAs), cell phones, handheld devices, pagers, etc., are more efficient to use pen input devices. Pen input devices are more efficient at inputting ink data than conventional text, unless handwriting recognition software is used. However, handwriting recognition software typically consumes a lot of system resources.
[0009]
Therefore, there is a need for an apparatus that integrates non-traditional data types, such as ink data, into traditional IM and chat infrastructure.
[0010]
There is a further need for a method and apparatus for performing ink data communication between a plurality of parties using a computer on a network.
[0011]
The present invention is illustrated by way of example, and not by way of limitation, in the accompanying figures wherein like reference numerals indicate like elements.
[0012]
(Detailed description)
Disclosed are methods and apparatus for performing ink data communication between multiple parties using a computing and / or communication device over a network. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that these specific details are not necessarily required to practice the invention. In other circumstances, well-known structures, materials, circuits, processes and interfaces have not been shown or described in detail so as not to unnecessarily obscure the present invention.
[0013]
Referring to FIG. 1, an exemplary computer system on which an embodiment of the present invention can be implemented is shown as 100. The computer system 100 includes a processing device for processing digital data, that is, a CPU 101. The processing unit 101 is a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processing unit that implements a combination of instruction sets, or It may be another processing device. Processing unit 101 is coupled to CPU bus 110, which transmits signals between processing unit 101 and other components in computer system 100.
[0014]
In the illustrated embodiment, memory 113 includes a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, or other memory device. The memory 113 stores information or other intermediate data while being executed by the processing device 101. Bridge memory controller 111 is coupled to CPU bus 110 and memory 113. Bridge memory controller 111 manages data traffic between processing unit 101, memory 113, and other components in computer system 100, and bridges signals from these components to high-speed I / O bus 120. I do.
[0015]
In the illustrated embodiment, the high-speed I / O bus 120 supports peripherals operating at high data throughput. Bus 120 may be a single bus or a combination of multiple buses. By way of example, bus 120 may include a Peripheral Components Interconnect (PCI) bus, a Personal Computer Memory Card International Association (PCMCIA) bus, or another bus. Bus 120 provides a communication link between components within computer system 100.
[0016]
Network interface 121 is coupled to bus 120. The network interface operates to link the computer system 100 to a network of servers or computers and provides communication between machines. Network interface 121 may be a telephone modem, cable modem, Integrated Services Digital Network (ISDN) connection, or other interface to a network. Display device controller 122 is coupled to high-speed I / O bus 120. Display device controller 122 is coupled to bus 120. Display device controller 122 allows coupling of the display device to the computer system and serves as an interface between the display device and computer system 100. The display device receives information and data from the processing device 101 via the display device control device 122 and displays the information and data to the user of the computer system 100. Graphics input device 124 is coupled to bus 120. Graphic input device 124 operates to input graphic images and / or handwritten or ink data into computer system 100. Graphic input device 124 may be, for example, a video camera and whiteboard or an electronic pen and tablet or other graphic input device. Graphic input device 124 may also include a conventional handwriting input device, such as a touch screen, digitizer, electronic tablet, mouse, electronic pen, light pen, and the like.
[0017]
In the illustrated embodiment, bus bridge 123 couples high-speed I / O bus 120 to I / O bus 130. Bus bridge 123 includes a translator for bridging signals between high speed I / O bus 120 and I / O bus 130. The I / O bus 130 is used to transfer information between peripheral devices operating at low throughput rates. The I / O bus 130 may be a single bus or a combination of multiple buses. As an example, the bus 130 may include an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, or a Micro Channel Architecture (MCA) bus. Bus 130 provides a communication link between components within computer system 100. Data storage device 131 may be a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device. Keyboard interface 132 enables coupling of the keyboard to the computer system and transmits signals from the keyboard to computer system 100. An optional audio controller 133 operable to coordinate sound recording and playback is also coupled to the I / O bus 130. A pointing device 135 such as a conventional mouse or trackball can also be coupled to the I / O bus 130.
[0018]
The invention is related to the use of computer system 100 for communicating ink data between multiple parties on a network. According to one embodiment, performing the ink data communication is performed by the computer system 100 in response to the processing unit 101 executing a sequence of instructions contained in the memory 113. Such instructions can be read into memory 113 from other computer-readable media, such as data storage device 131 or from a network. Executing the instruction sequences contained in memory 113 causes the processing unit to perform the functions described herein. In an alternative embodiment, hardwired circuitry may be used instead of or in combination with software instructions to implement the invention. Accordingly, the invention is not limited to any particular combination of hardware circuits and software. Those skilled in the art will appreciate that the present invention recognizes a wide variety of other computing devices and / or devices, such as chat devices, personal digital assistants (PDAs), palm-top computers, or similar devices. Or it will be appreciated that it can be used on a communication device.
[0019]
FIG. 2 is a diagram illustrating a data distribution system that implements one embodiment of the present invention using a unicast protocol. The data distribution system 200 is a network including a plurality of clients 100 and 201 to 204. Clients 100 and 201-204 can be implemented by a computer system as described in FIG. The data distribution system 200 also includes a server 210. Each client maintains a separate connection with the server 210. Lines 221-225 represent connections between clients 100 and 201-204 and server 210, respectively. Connections 221-225 may be, for example, Internet connections using Transfer Control Protocol (TCP) or User Datagram Protocol (UDP). According to one embodiment of the present invention, a graphic or ink data message generated from the client is transmitted to the server 210 over the connection in the form of a packet of data. The graphic or ink data message is sent to the server 210 message queue. The message queue contains other graphic or ink data messages from other clients. A copy of the graphic or ink data message received from each client is then transmitted from server 210 to clients 100 and 201-204 in the order received. Clients 100 and 201-204 store graphics or ink data messages received from server 210 in their own queue of received messages. Thus, each client receives a copy of the message sent to server 210 in the order in which the messages were received by server 210. Clients 100 and 201-204 display on their display devices packets of graphics or ink data messages stored in their received message queue.
[0020]
According to one embodiment of the present invention, server 210 may be connected to multiple similarly configured servers to form a network of servers that share graphics or ink data. According to other embodiments of the present invention, connections 221-225 may be non-Internet connections over a local area network (LAN) or a wide area network (WAN). Server 210 and clients 100 and 201-204 can be implemented by any known circuits. It should be appreciated that data distribution system 200 may be a single computer system having multiple clients connected to the computer system, and the multiple clients may use inter-process communication to communicate with one another.
[0021]
FIG. 3 is a diagram illustrating a data distribution system that implements one embodiment of the present invention using a multicast protocol, such as the Multicast Internet Protocol. The data distribution system 300 is a network including a plurality of clients 100 and 201 to 204. Clients 100 and 201-204 can be implemented by a computer system as described in FIG. Clients 100 and 201-204 are coupled to transmission medium 310. Transmission medium 310 may be, for example, an Internet network or a local area network (LAN) or a wide area network (WAN). According to one embodiment of the present invention, in the data distribution system 300, the first graphic or ink data message is transmitted from the first client 100 to the plurality of other clients 201-204 via the transmission medium 310. Transmitted as a packet of graphic or ink data. Clients 100 and 201-204 each have a queue of received messages in which messages sent from other clients are stored in the order received. The client 100, which is the message sending client, also stores a copy of the message sent to the other client in its own received message queue. Clients 201-204 also implement the same protocol when sending graphic or ink data messages. Clients 100 and 201-204 may display graphic or ink data messages stored in their received message queue on a display device.
[0022]
One embodiment of the present invention is directed to a method for converting the current chat and instant messaging infrastructure into a system in which a user obtains ink or graphics acquired by a graphic or handwriting input device such as a touch screen, digitizer, electronic tablet, mouse, light pen, and the like. It is designed to be extended to allow communication with data. The use of simple ink data is well known in the art. For example, it is well known to capture handwriting input in electronic form by capturing information associated with the movement of an electronic pen on an electronic tablet. This type of information is commonly called ink data. In the prior art, ink data is represented as simple pen x, y positions at specific time increments. In more advanced conventional ink data capture devices, other parameters, including hand position, time, pen pressure, style, color, and other attributes associated with pen movement, are captured by the capture device. To the processing device via the interface. Although these ink data capture devices exist in the prior art, no applications are known or suggested for their use in instant messaging or chat systems.
[0023]
Traditional chat or instant messaging systems have been used primarily to send text from one network user to another during a chat session. Although many conventional instant messaging players have added voice carrying capabilities, none currently provide or suggest the use of the ink data format in instant messaging or chat sessions. However, the ink data format is particularly suitable for integration with instant messaging or chat infrastructure for at least the following reasons: First, the ink data format is an ideal way to share creative activities such as drafting, architecture, and design. Second, the ink data format is ideal for supporting communication with visual content, such as directions on a map, changing the layout of a floor plan, and the like. Third, the ink data format is the most efficient way for people communicating in many countries outside the United States. Even in the United States, millions of people can't type fast enough, but handwriting is generally a very natural and convenient means of communication for them.
[0024]
Referring next to FIG. 4, a block diagram illustrating the basic architecture of one embodiment of the present invention is shown. Input device 410 is representative of any one of a variety of conventional electronic pen input devices, handwriting input devices, and other types of conventional ink data capture devices. Such an input device 410 produces a raw ink data input, typically including x-coordinates, y-coordinates, and time information from an ink data capture device. Ink data input device 410 operates with a hardware interface software layer or driver 420 to format the raw ink data into a form suitable for processing at a higher functional level. The x, y coordinate information is used to locate the electronic pen tip at a particular time instance. The timing information provided by the input device 410 can be used to replicate handwriting strokes of the electronic pen with a high degree of accuracy, thereby increasing the understandability and aesthetic appeal of the ink data display. This improved understandability applies to both human-based and machine-based handwriting recognition. If a more advanced ink data input device 410 is used, additional information such as pen pressure, drawing style or line type information, color information, and other types of ink data information may be provided by the input device 410 in hardware. -Can be provided to the interface layer 420. It will be apparent to one skilled in the art that providing additional ink data parameters enhances the rendering and presentation of the ink data information to remote users.
[0025]
Once the hardware interface layer 420 assembles the raw ink data into raw ink data packets, the packets are transferred to the ink management layer 430 as shown in FIG. Generally, the ink management layer 430 handles the initial processing of raw ink data received from the hardware interface layer 420. Depending on the type and amount of data received from the input device 410, it may be necessary to shorten or compress the raw ink data received via the hardware interface layer 420. For example, the ink management layer 430 may use conventional techniques to filter the duplicate raw ink data or to interpolate and smooth the raw ink data, which may reduce some of the data. You may lose. Further, the ink management layer 430 can use conventional data compression techniques to reduce the amount of data transferred to higher functional levels without reducing the information represented by the raw ink data. As an additional level of functionality, ink management layer 430 may also encrypt the raw ink data using well-known techniques. In addition, the ink management layer 430 can apply conventional handwriting recognition techniques to convert the ink data to text format. Any of these processing functions, such as shortening, compression, encryption, and handwriting recognition, can be selectively enabled or disabled using the configuration user interface provided by the present invention. The ink management layer 430 also converts the raw ink data into a network transportable format. Upon completion of this processing of the input raw ink data, the ink management layer 430 generates processed ink data. This is transferred to the network interface layer 440. The network interface layer 440 handles the conversion of the processed ink data into a format compatible with the local network communication protocol. In some conventional systems, the network interface layer 440 may not be necessary if the computing device on which the invention is implemented has already established an Internet connection. Such devices, such as desktop personal computers, may already have the ability to carry processed ink data across a network. However, if these features are not implemented or available for a particular system using the present invention, the network interface layer 440 can transport the processed ink data across the network to a remote user. Can be converted to a format. Once the networked transportable ink data is generated by the network interface layer 440, the networked transportable ink data is transferred to a messaging layer 450, such as an instant messaging (IM) interface layer 450. Interface layer 450 provides a conventional system infrastructure for transferring instant messages between two users on a computer network. The present invention enhances the conventional instant messaging interface layer 450 to include the transport of processed ink data using a conventional instant messaging infrastructure. In this manner, a user at one network location with an input device 410 that generates ink data can carry the ink data to a remote network user via the instant messaging infrastructure 450. . It will be apparent to one skilled in the art that other conventional messaging infrastructures can be used as well. For example, the interface layer 450 may be an interface layer that implements a chat function. In this embodiment, the network interface layer 440 provides the processed ink data to the chat interface layer, which transmits the processed ink data to other message receivers during a particular chat session. become. The present invention also enhances the conventional chat interface layer to provide the ability to transport processed ink data to other chat receivers. In another embodiment of the present invention, interface layer 450 may be a conventional email transport system. In this case, the network interface layer 440 provides the processed ink data to the email interface layer, which integrates the processed ink data into a conventional email message, where the ink data is embedded. The e-mail message to a remote network using a conventional e-mail transport mechanism. In addition, enhancements to conventional systems are needed to allow processed ink data to be embedded in conventional e-mail messages for transport by e-mail applications. Accordingly, the generation and transmission of ink data using a conventional chat or email infrastructure is described.
[0026]
Still referring to FIG. 4, the illustrated embodiment of the present invention includes functionality at the receiver side to receive a chat or instant message containing ink data. On the receiver side, interface layer 450 receives messages from remote users using conventional techniques. Further, interface layer 450 may be an instant messaging interface, a chat interface, or an email interface. Using any conventional means, a message containing the processed ink data is received by a receiver at interface layer 450. The incoming ink data message is then forwarded to the network interface layer 440. The network interface layer 440 unwraps the incoming ink data message and extracts the ink data. The ink data packets received via the incoming message are passed from the network interface layer 440 to the ink management layer 430. The ink management layer 430 is responsible for decoding incoming ink data messages. In response to the ink data processing performed by the message transmitter, the receiver-side ink management layer 430 may decrypt the incoming ink data if the ink data message was encrypted by the transmitter. become. Further, the ink management layer 430 at the receiver decompresses the received ink data using conventional techniques. The receiver-side ink management layer 430 can also apply conventional handwriting recognition techniques to the ink data. If necessary, the ink management layer 430 at the receiver will further process the received ink data to facilitate rendering the ink data on the display device 460 at the receiver. Once the incoming ink data is processed by the ink management layer 430 at the receiver, the decrypted ink data is transferred to the hardware interface layer 420 at the receiver. In addition, the hardware interface layer 420 at the receiver formats the ink data to be compatible with a particular display device 460 at the receiver. The hardware interface layer 420 then forwards the received ink data to the display device 460 for presentation to the network user at the receiver end. Accordingly, ink data captured or generated by a networked transmitter may be displayed at a receiver end via a conventional instant messaging, chat, and / or email infrastructure. Forwarded to network receiver.
[0027]
Thus, a method and apparatus for implementing ink data communication between multiple parties using a computing and / or communication device over a network has been disclosed. Although the invention has been described herein with reference to certain preferred embodiments, many modifications and variations will readily occur to those skilled in the art. Accordingly, all such variations and modifications are within the intended scope of the invention as defined by the appended claims.
[Brief description of the drawings]
FIG.
1 is a configuration diagram illustrating an example of a computer system that implements an embodiment of the present invention.
FIG. 2
FIG. 2 illustrates a data distribution system that implements one embodiment of the present invention using a unicast protocol.
FIG. 3
FIG. 1 illustrates a data distribution system that implements one embodiment of the present invention using a multicast protocol.
FIG. 4
FIG. 2 is a diagram showing a functional structure and a processing flow of an embodiment of the present invention.

Claims (21)

A hardware interface layer for receiving raw ink data from the input device;
An ink management layer for processing the raw ink data into a form usable by the chat interface layer;
A network interface layer for transferring processed ink data across a network to a chat interface layer of a remote computer. The electronic chat system of claim 1, wherein the chat interface layer further comprises an instant messaging layer for receiving processed ink data. The electronic chat system of claim 1, further comprising a chat subsystem for displaying processed ink data as part of a multi-user chat session. The electronic chat system of claim 1, further comprising an instant messaging subsystem for displaying processed ink data as part of an instant messaging session. The electronic chat system of claim 1, wherein the ink management layer further comprises a data compression component for compressing the raw ink data. The electronic chat system of claim 1, wherein the ink management layer further comprises a data encryption component for encrypting the raw ink data. The receiver of claim 1, further comprising a receiver for receiving the processed ink data transmitted by the remote computer system and converting the received ink data into a form that can be displayed as part of a chat session. Electronic chat system. The electronic chat system of claim 7, further comprising a data decompression component for decompressing ink data received from the network. The electronic chat system of claim 7, further comprising a data decoding component for decoding ink data received from the network. The electronic chat system of claim 1, wherein the ink management layer further comprises a handwriting recognition component. Receiving raw ink data from an input device;
A method comprising processing the raw ink data into a form usable by a chat interface layer, and transferring the processed ink data across a network to a chat interface layer of a remote computer. The method of claim 11, wherein the chat interface layer further comprises an instant messaging layer for receiving processed ink data. The method of claim 11, further comprising displaying the processed ink data as part of a multi-user chat session. The method of claim 11, further comprising displaying the processed ink data as part of an instant messaging session. The method of claim 11, further comprising compressing the raw ink data. The method of claim 11, further comprising encrypting the raw ink data. The method of claim 11, further comprising: receiving processed ink data transmitted by a remote computer system; and converting the received ink data into a form that can be displayed as part of a chat session. Method. The method of claim 17, further comprising decompressing the ink data received from the network. The method of claim 17, further comprising decrypting the ink data received from the network. The method of claim 11, further comprising performing handwriting recognition on the received ink data. A product comprising a computer-readable medium having the processing instructions encoded thereon, wherein the processing instructions comprise:
A hardware interface component for receiving raw ink data from the input device;
An ink management component for processing the raw ink data into a format usable by the chat interface component;
A network interface component for transferring processed ink data across a network to a chat interface component of a remote computer.

Images