JPH08204877A - Interactive communication system and control method therefor - Google Patents

Abstract

PURPOSE: To transmit all of image data, handwritten data and voice data without using a high speed transmission line and to inexpensively realize the interaction by handwriting and voice based on the images displayed on both a master machine and a slave machine. CONSTITUTION: A master machine 10 preliminarily transmits the image data to be used for an interaction to a slave machine 30, the slave machine 30 stores the received image data. Every time the voice data and handwritten data related to image data is inputted in the master machine 10, the data is transmitted to the slave machine 30, subsequently displaying the same image at both the master machine 10 and the slave machine 30. As a result, because received voice data and handwritten data is outputted and handwritten data is stored in the slave machine 30, and every time the voice data related to image data is inputted in the slave machine 30, the data is transmitted to the master machine 10, the received voice data is outputted in the master machine 10.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication dialogue system for multimedia and its control method, and more particularly to transmitting image data, handwriting data and voice data without using a high speed transmission path. The present invention relates to a communication dialogue system and a control method thereof that can realize a dialogue by handwriting and voice at low cost based on images displayed on both a parent device and a child device.

[0002]

2. Description of the Related Art FIG. 9 shows a conventional communication dialogue system.
There was one using telewriting as shown in. FIG. 9 is an explanatory diagram showing a data flow in a conventional dialogue by telewriting. Telewriting is a method of performing dialogue by exchanging voice data and handwritten data with each other (bidirectionally) using a communication line such as a telephone line. As shown in FIG. 9, in telewriting, handwriting data and voice data input on the transmitting side are multiplexed and transmitted via a communication path, and received data is demultiplexed on the receiving side to produce handwriting data and voice data. It is separated into data and output respectively.

Here, the handwritten data is data indicating a locus (trace) of a figure drawn on input coordinates using an input pen or the like, starting from pen down at an arbitrary point, and pen at an arbitrary point. It is transmitted in units of a series of trajectories that end with up.

Further, as a conversion method for converting a locus input by handwriting into digital data (handwriting data),
The zone coding method and the differential chain coding method are known. The zone coding method is a method of coding with a quadrant code and a differential coordinate, and is an effective method for real-time line drawing transmission. The handwritten data and the voice data encoded in this way are multiplexed and further converted into a signal suitable for transmission and transmitted.

Further, another conventional communication dialogue system using a fax will be described with reference to FIG. FIG. 10 is an explanatory diagram showing a data flow in a conventional dialog using a fax. The fax is for reading the grayscale of an input image, converting it into data, and transmitting it via a communication line. As shown in FIG. 10, in a dialogue using a fax, first, image data is transmitted in advance by fax from the transmitting side, and an image is reproduced on the receiving side. When the transmission is completed, the fax line is disconnected. After that, a telephone call is made to connect the telephone line, and while both the transmitting side and the receiving side look at the above image, voice data is exchanged between the telephones for dialogue.

Further, conventionally, in order to transmit all images, handwriting, and voice for dialogue, a communication system provided with a dedicated high-speed transmission line capable of transmitting moving images and voice was required. .

[0007]

However, in the above-mentioned communication dialogue system using telewriting of the conventional example, only handwriting and voice can be exchanged, and images cannot be transmitted. Therefore, complicated tables and figures are included. There is a problem in that it is not possible to conduct a dialogue using a document and the use is significantly limited.

Further, in the conventional communication dialogue system using the above-mentioned fax, although the image data can be transmitted, since only the image is sent in advance, there is a problem that the dialogue interlocked with the image cannot be performed. there were.

Further, conventionally, when trying to transmit all images, handwriting and voice, a dedicated high speed transmission line is required,
There was a problem that the cost of the system became high.

The present invention has been made in view of the above circumstances, and can transmit all image data, handwriting data, and voice data without using a high-speed transmission path, and further, can transmit the data to both the parent device and the child device. It is an object of the present invention to provide a communication dialogue system and a control method thereof that can realize a dialogue by handwriting and voice at low cost based on a displayed image.

[0011]

The invention according to claim 1 for solving the problems of the above-mentioned conventional example is a transmitting side device for transmitting image information, voice information and handwriting information and receiving voice information, In a communication dialogue system including a reception side device that receives image information, voice information, and handwriting information and transmits voice information, the transmission side device stores image information.
The image data is read out from the first storage unit in advance and transmitted, and thereafter, the voice information and the handwriting information related to the image information are transmitted every time they are input, and the receiving side device Is a device for outputting the voice information each time it is received, and the reception side device has a second storage section for storing the image information received in advance, and the voice information and the handwritten information related to the image information Is output each time it is received, and is transmitted to the transmission side device each time voice information is input.

According to a second aspect of the invention for solving the problems of the conventional example, in the communication dialogue system according to the first aspect, the transmitting side device and the receiving side device transmit the image information and the handwritten information in page units. And a first and a second display unit for displaying image information and handwritten information, wherein the transmission side device has an image of the page according to the input page information. This is a device for reading handwritten information related to information from the first storage unit, displaying it on the first display unit, and transmitting the page information to the receiving side device, and the receiving side device receiving the page. According to the information, the image information stored in the second storage unit and the related handwriting information are read out and displayed on the second display unit.

According to a third aspect of the invention for solving the problems of the conventional example, in the control method of the communication dialogue system according to the second aspect, the transmission side device stores the input image information in the first storage. When the page information is input, the receiving side device receives the transmitted image information and stores it in the second storage part, and the transmitting side device stores the image information of the page. Read from the first storage unit and display on the first display unit,
The page information is transmitted to the receiving side device, and the receiving side device reads out image information of the corresponding page from the second storage section according to the received page information and displays it on the second display section. The device displays the handwriting information on the first display unit when voice information and handwriting information are input, and transmits the voice information and the handwriting information to the receiving side device to display the handwriting information on the page. Stored in the first storage unit in association with each other, the receiving side device receives the transmitted voice information and handwriting information, outputs the voice information, and displays the handwriting information on the second display unit. , The handwriting information is stored in the second storage unit in association with the page, and when the page information is input again, the transmission side device outputs the handwriting information related to the image information of the page as the first information. Read from storage When the page information is displayed on the first display unit and the page information is transmitted to the receiving side apparatus, and the receiving side apparatus receives the page information, the handwritten information related to the image information of the page is stored in the second storage. And is displayed on the second display unit.

[0014]

According to the invention described in claim 1, the transmitting side device is
Image information is transmitted in advance, and is transmitted each time voice information and handwriting information is input, and the receiving side device receives and stores image information in advance, and outputs each time voice information and handwriting information is received. Since it is a communication dialogue system, when transmitting and receiving voice information related to image information and handwriting information in real time, since the image information is transmitted in advance, the data amount of the transmission data in real time can be reduced, and the high speed transmission line And an inexpensive system configuration can be achieved.

According to the second aspect of the present invention, the transmission side device and the reception side device store the image information and the handwriting information in the first and second storage sections in association with each other, and the transmission side device stores the page information. When input, the handwritten information related to the image information of the page is read from the first storage unit and displayed on the first display unit on the transmission side, and the receiving side device that received the page information determines that the image information of the page is received. Since the handwriting information associated with is read from the second storage unit and is displayed on the second display unit on the receiving side, the handwriting information related to the image information is held on both the transmitting and receiving sides. Therefore, it is possible to easily talk by voice based on the image and the handwritten information.

According to the third aspect of the invention, handwritten information related to the image information is held on both the transmitting and receiving sides, and when the page information is input to the transmitting side apparatus, the transmitting side apparatus causes the transmitting side apparatus to store the page information. If the image information of the page is already stored, the related handwritten information is read from the first storage unit, displayed on the first display unit, and the page information is transmitted to the receiving side device, and the receiving side device The control of the communication dialogue system according to claim 2, wherein the handwritten information related to the image information of the page of the transmitted page information is already stored and is displayed on the second display unit. Since the method is used, when transmitting and receiving voice information and handwriting information related to image information in real time, since the image information is transmitted in advance, the data amount of transmission data in real time can be reduced and a high-speed transmission path is required. Without There is an advantage of being able to valence system configuration can be easily interact by voice based on the image and the handwritten information.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings. A communication dialogue system according to an embodiment of the present invention first transmits image data used for dialogue from a master unit to a slave unit, and then specifies a page of image data to be displayed on both sides and synchronizes them. While displaying, the voice data and handwritten data are transmitted from the base unit to the handset in real time.
On the other hand, it is a system that transmits voice data in real time from the handset to the base unit. By reducing the amount of data to be transmitted in real time, it is possible to use a voice with handwritten data while displaying images without using a high-speed transmission path. Dialogue can be realized.

First, the configuration of a communication dialogue system according to an embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram of a master unit of a communication dialogue system according to an embodiment of the present invention, and FIG. 2 is a block diagram of a slave unit of a communication dialogue system according to an embodiment of the present invention.

Base unit 10 of the communication dialogue system of this embodiment
Is a CPU 11 and an image input device 1 as shown in FIG.
2, an image encoding device 13, a handwriting input device 14,
A handwritten data encoding device 15, a voice input device 16, and
A voice encoding / decoding device 17, an auxiliary storage device 18,
It comprises an MPX / DMPX (Multiplexer / Demultiplexer) 19, a communication control device 21, an image output device 22, and an audio output device 23.

As shown in FIG. 2, the slave unit 30 of the communication dialogue system of this embodiment has a CPU 31, a communication control unit 32, and an MPX / DMPX (Multiplexer / Demultipl).
exer: multiplexing / demultiplexing device) 33 and auxiliary storage device 34
An image decoding device 35 and a handwriting data decoding device 3
6, an audio encoding / decoding device 37, and an image output device 3
8, voice output device 39, and voice input device 40.

Next, each unit constituting the master unit 10 of the communication dialogue system of the present embodiment will be concretely described. The image input device 12 is an image input device such as a video camera or an image scanner for inputting an image (still image) to be transmitted. The image encoding device 13 is a device for source-encoding pixel information (image data) of an image input from the image input device 12 and stored in the auxiliary storage device 18. Here, the information source coding method is, for example, “JPEG (Joint Photographic Exp
"ert Group)" method is well known ("The latest MP
EG textbook "edited by Multimedia Communication Research Group, published by ASCII Co., Ltd., published in August 1994 (see p53 to p67) is generally an image compression coding (information source coding) method capable of compressing an image on a screen-by-screen basis. Any method may be used as long as it is not an essential part of the present invention, and a detailed description thereof will be omitted here.

The handwriting input device 14 is a handwriting data input device such as a mouse for inputting (handwriting input) handwritten data. Handwritten data is data that shows a curve of arbitrary shape that usually starts from an arbitrary point (pen down) and ends at an arbitrary point (pen up). For example, in the case of a mouse, the position of the mouse cursor when the button is pressed. Corresponds to pen down, the position of the mouse cursor when the button is released corresponds to pen up. Then, the locus of movement of the mouse cursor from pen down to pen up becomes handwritten data. Besides the mouse, a handwriting input device using a pen and a digitizer, or a handwriting input device of a touch panel can be considered. In addition to the handwriting input, the handwriting input device 14 uses a field on the screen displayed on the image output device 22, which will be described later, to change the display page, input a page change, and input a dialogue end to instruct the dialogue end. It is also supposed to be done.

The handwritten data encoding device 15 is a device for encoding the handwritten data input by the handwriting input device 14, and the encoding method is a general one such as zone encoding and differential chain encoding similar to the conventional one. Although a handwritten data encoding method is used, it is not an essential part of the present invention, and therefore a detailed description is omitted here.

The voice input device 16 is an MI for inputting voice.
It is a voice input device such as C (Microphone). The voice encoding / decoding device 17 is a device that encodes the voice data input from the voice input device 16 and, conversely, decodes the received voice encoded data and outputs the decoded voice data to the voice output device 23.

Here, the voice encoding / decoding method is, for example, a DPCM (Differential Pulse Code Mo) which takes a difference (increase / decrease) between a previous signal and an input signal and quantizes the difference.
(dulation: differential pulse code modulation) and adaptive differential pulse code modulation that quantizes the difference (increase / decrease) between the input signal and the prediction by utilizing the strong correlation between the sampling values of the audio signal (Adaptive Differential Puls
e Code Modulation (ADPCM) and CEL based on vector quantization to further improve voice quality
A compression encoding technique such as P (Code Excited Linear Prediction) is generally known.
("Latest MPEG textbook", Multimedia Communication Research Group, published by ASCII Corporation, published in August 1994, p1
95-p204)

The technique used in the voice encoding / decoding device 17 of this embodiment is a general technique, and the technique used in the voice encoding / decoding device 17 of the master device 10 and the technique of the slave device 30 are used. Any technique may be used as long as it corresponds to the technique used in the audio encoding / decoding device 37.

The auxiliary storage device 18 is a storage device for storing the image data input from the image input device 12 and the handwritten data input from the handwriting input device 14. still,
Image data and handwritten data are managed so as to correspond page by page. The MPX / DMPX 19 is image coded data coded by the image coding device 13 or handwritten coded data coded by the handwritten data coding device 15 and voice coded by the voice coding / decoding device 17. This is a multiplexing / separating device that multiplexes encoded data to create a transmission packet and, conversely, extracts speech encoded data from a reception packet.

The communication control device 21 uses the MPX / DMPX1.
9. The transmission packet of the image coded data or the handwritten coded data and the voice coded data created by multiplexing at 9 is transmitted to the communication path, or conversely from the communication path to the other party (slave unit 3
0) is a device for receiving the packet of the voice coded data transmitted.

The image output device 22 is an image display device such as a CRT which displays the image input from the image input device 12 and the handwritten data input from the handwriting input device 14 by combining them. Here, the image output device 22 will be specifically described with reference to FIG. FIG. 3 is an explanatory diagram showing a display example of the display screen of the image output device 22 of the parent device 10 of the communication dialogue system of the present embodiment.

As shown in FIG. 3, the display screen of the image output device 22 of the parent device 10 of the communication dialogue system of the present embodiment has the image input from the image input device 12 and the handwriting input device 14, as shown in FIG.
A display field 41 for combining and displaying handwritten data input from, an information field 42 for displaying the total number of pages of image data and a page number of the image being displayed, and an operation mode of the handwriting input device 14 are selected. A menu field 43 for changing the page and a page changing field 44 for changing the page displayed on the image output device 22.

When the page change field 44 is clicked with the handwriting input device 14 to switch pages (page change input), the image data of the designated page is displayed in the display field 4.
1, the total number of pages of image data and the page number of the image being displayed are displayed in the information field 42. Furthermore, when handwritten data is input (trace from pen down to pen up) in the display field 41 using the handwriting input device 14, the handwritten data is combined and displayed. In addition, when the page is changed, the handwritten data input by handwriting on the page being displayed is stored, and when the page is displayed again, the image data and the stored handwritten data are combined and displayed. It is supposed to do. Further, a dialogue end input for instructing the dialogue end using the menu field 43 is also made.

The voice output device 23 is a voice output device such as a speaker that reproduces and outputs the voice data transmitted from the child device 30. The CPU 11 controls the operation of each unit that configures the parent device 10. Specifically, the image data is input to the child device 30 prior to the image input processing for inputting and storing an image and the interaction with the child device 30. The image data transmission process for transmitting the image data to the mobile device 30 and the interaction process for interacting with the child device 30 are performed. The details of each process will be described later.

Next, each part of the slave unit 30 of the communication dialogue system of this embodiment will be described in detail. The communication control device 32 receives packets of image coded data or handwritten coded data and voice coded data transmitted from the other party (base device 10) from the communication path, or conversely MPX / D.
This is a device for transmitting a transmission packet of voice encoded data created by multiplexing with the MPX 33 to a communication path. MPX /
The DMPX 33 is a multiplexing / separating device that creates a transmission packet of voice encoded data and, conversely, separates and extracts image encoded data or handwritten encoded data and voice encoded data from a received packet.

The auxiliary storage device 34 is a storage device for storing the image data and the handwritten data transmitted from the parent device 10. As with the auxiliary storage device 18 of the parent device 10, image data and handwritten data are managed so as to correspond page by page. The image decoding device 35 receives the MPX / DMPX transmitted from the master device 10 via the communication control device 32.
It is a device for generating pixel information (image data) of an image by decoding the encoded image data taken out at 33 as an information source. Here, the information source decoding method is a decoding technique corresponding to the information source encoding method used in the image encoding device 13 of the master device 10.

The handwritten data decoding device 36 is the master device 10
Sent from the communication control device 32 and received by the MPX.
/ DMPX33 is a device for decoding handwritten coded data separated and taken out to create handwritten data.
Here, the handwriting data decoding method is a decoding technique corresponding to the coding method used in the handwriting data coding device 15 of the parent device 10.

The voice encoding / decoding device 37 is composed of the base unit 10.
Is a device that decodes the voice coded data transmitted from the mobile device 30 and is output to the voice output device 39, and conversely encodes the voice data input from the voice input device 40.

The image output device 38 receives the image data transmitted from the master device 10 and received by the slave device 30 and stored in the auxiliary storage device 34, and the handwritten data and auxiliary data transmitted from the master device 10 and decoded. It is an image display device such as a CRT that combines and displays handwritten data stored in the storage device 34. The specific display screen of the image output device 38 is composed of only the display field 41 and the information field 42 of the display screen of the image output device 22 of the parent device 10 shown in FIG.

The voice output device 39 is a voice output device such as a speaker which reproduces and outputs the voice data transmitted from the parent device 10. The voice input device 40 is an M for inputting voice.
It is a voice input device such as an IC (Microphone).

The CPU 31 controls the operation of each unit constituting the slave unit 30, and specifically, the image data receiving process for receiving the image data prior to the dialogue with the master unit 10, and the master unit 10. In the dialogue, the downlink processing for processing the received data and the uplink processing for transmitting the voice data are performed. The details of each process will be described later.

Next, the outline of the operation of the communication dialogue system of this embodiment will be described with reference to FIG. FIG. 4 is an explanatory diagram showing a transmission / reception sequence between the master device 10 and the slave device 30 in the communication dialogue system of the present embodiment. In the communication dialogue system of the present embodiment, first, one or a plurality of pages of images used in dialogue in the master device 10 are input and stored. Then, the parent device 10 encodes the image information (image data) stored prior to the conversation with the child device 30 into image encoded data, and further transmits it to the child device 30 as an image packet (see FIG. 4).
Process (a)), the slave unit 30 decodes the received image coded data and stores the image information.

After the image information has been transmitted, handwriting information is transmitted in real time from the parent device 10 to the child device 30,
Further, an interactive process for bidirectionally exchanging voice information is performed (process (b) in FIG. 4). Here, at the time of the dialogue processing, as shown in FIG. 4, a packet including handwritten information and voice information transmitted from the master device 10 to the slave device 30 and conversely the slave device 3 are transmitted.
The communication path is time-division multiplexed with the packet of voice information transmitted from 0 to the base unit 10 and used. At the end of the dialogue, a dialogue end packet is transmitted from the parent device 10 to the child device 30 so that the child device 30 recognizes the dialogue end.

Next, details of the operation of the communication dialogue system of the present embodiment will be described with reference to FIGS. 1, 2 and 3.
In the communication dialogue system of the present embodiment, first, in the parent device 10, the CPU 11 controls each unit to perform an image input process of inputting and storing one or a plurality of pages of images used in the dialogue. Here, the image input processing is performed by the image input device 12 to 1
Alternatively, an image of a plurality of pages is input, and the pixel value (image data) forming the input image is stored in the auxiliary storage device 18.
The auxiliary storage device 18 stores the image data of each page in association with the page number.

Next, the base unit 10 performs the image data transmission process prior to the conversation with the handset unit 30. Here, the image data transmission process means 1 stored in the auxiliary storage device 18.
Alternatively, the image encoding device 13 encodes the image data of a plurality of pages as the information source to generate the image encoded data, and the MPX / DMP
This is a process of packetizing the image coded data in X19 to create an image packet and transmitting it to the handset 30 via the communication control device 21.

When the image packet is transmitted from the master device 10, the slave device 30 receives the image packet via the communication control device 32 as an image data receiving process, and the MPX / D
The MPX 33 extracts the encoded image data, the image decoding device 35 performs information source decoding to create image data, and stores the image data in the auxiliary storage device 34. The auxiliary storage device 34 also stores the image data of each page in association with the page number, similar to the auxiliary storage device 18 of the master device 10.

After the image data transmission is completed, the main unit 1
Dialogue processing between 0 and the child device 30 is performed. Here, the interactive process includes a downlink process for transmitting handwritten information and voice information from the master device 10 toward the slave device 30 and an uplink process for transmitting voice information from the slave device 30 toward the master device 10 on the contrary. The communication paths are performed in parallel by time division multiplexing.

First, when the dialogue between the parent device 10 and the child device 30 is normally started, the first field is displayed in the display field 41 of the image output device 22 of the parent device 10 and the image output device 38 of the child device 30.
The page image is displayed. When handwriting data is input (handwriting input) in the master device 10, the CPU 11
Displays the handwritten data in the display field 41 of the image output device 22, and temporarily stores the handwritten data in the work area in the CPU 11. Furthermore, the input handwritten data is encoded by the handwritten data encoding device 15 to create handwritten encoded data, and "handwritten data"
Is added and a handwritten data packet is created by MPX / DMPX19. The handwriting input is an input made by the handwriting input device 14 in the display field 41 shown in FIG. 3 in an operation from pen down to pen up.

When a page switching input (page change input) is made in the master unit 10, the CPU 11
Handwritten data for the displayed page stored in the work area of 11 is saved in the auxiliary storage device 18, and the image data and handwritten data of the new page are read from the auxiliary storage device 18 and displayed on the image output device 22. From the above, a header of "page change" is added, and a page change packet is created by MPX / DMPX19. The page change input is input when the [end] button is clicked by the handwriting input device 14 in the menu field 43 shown in FIG.

When an input for instructing the end of the dialogue (interaction end input) is made in the master device 10, "end code"
The header of is added by the CPU 11, and MPX / DMPX19
Create a dialogue end packet with. Further, when voice input is performed from the voice input device 16 in the master device 10, the voice encoding / decoding device 17 encodes to generate voice encoded data, and the CPU 11 adds a header of "voice data". , MPX / DMPX 19 creates a voice packet.
When handwriting input, page change input, or dialogue end input is performed, the packet and the voice packet are M
The PX / DMPX 19 multiplexes to create a transmission packet and transmits it via the communication control device 21.

In the child device 30, which has received the packet transmitted from the parent device 10 via the communication control device 32, the MPX / DMPX 33 separates the packet as received data processing, and the packet is controlled under the control of the CPU 31. Header analysis is performed and processing is performed according to the type of packet. In the case of a handwritten data packet, it is decoded by the handwritten data decoding device 36 to create handwritten data, displayed in the display field 41 of the image output device 38, and temporarily stored in the work area in the CPU 31. C for page change packet
Handwriting data relating to the page being displayed accumulated in the work area in the PU 31 is saved in the auxiliary storage device 34, and the image data and handwriting data of the new page are stored in the auxiliary storage device 3
4 is read out and displayed on the image output device 38. In the case of the dialogue end packet, the dialogue processing is ended. The voice packet is decoded by the voice encoding / decoding device 37 to create voice data, and the voice output device 39 outputs the voice.

On the other hand, when the voice data is transmitted from the handset 30 to the base unit 10, voice is input from the voice input device 40 and the voice encoding / decoding device 37 is used as voice data transmission processing in the handset 30. The data is encoded to create voice encoded data, multiplexed by the MPX / DMPX 33 to create a transmission packet, and transmitted via the communication control device 32. Upon receiving the voice packet transmitted from the child device 30 via the communication control device 21, the parent device 10 receives MPX / DMPX.
The audio encoded data is taken out at 19, decoded by the audio encoding / decoding device 17 to create audio data, and the audio output device 23 outputs the audio.

5 to 7 for details of the downlink processing for transmitting data from the master unit 10 to the slave unit 30 among the dialogue processes in the master unit 10 of the communication dialogue system of the present embodiment. It will be explained concretely by using. FIG. 5 is a flowchart of the main processing of the downlink processing of the dialogue processing in the CPU 11 of the master device 10 of the communication dialogue system of the present embodiment, and FIG. 6 is a flowchart of the event monitoring processing of the downlink processing. FIG. 7 is a flowchart of the voice encoding process of the downlink processing.

Specifically, the downlink processing in the master device 10 is performed periodically by an event monitoring processing which is performed when any event of handwriting input, page change input or dialogue end input occurs and a timer interrupt. MPX / DMP packet generated by voice encoding processing
It is multiplexed with X19 and transmitted via the communication control device 21.

Specifically, as shown in FIG. 5, the CPU 1
1 first performs event monitoring processing (100), fla
It is determined whether g is END (102). Where f
“lag” is a flag indicating the end of the dialogue process, and when the dialogue end input is made in the voice encoding process described later, E
It is designed to be set to ND. When the flag is not END (not the end of the dialogue) in the process 102, the voice encoding / decoding device 17 is caused to perform the voice encoding process (110), and MPX / DMPX1.
In step 9, a packet is created, the communication control device 21 is caused to transmit the packet (112), and the process returns to step 100. On the other hand, in the process 102, if the flag is END (end of dialogue), the MPX / DMPX 19 creates a dialogue end packet (120), and the communication control device 21 transmits the packet (122).

Then, in the event monitoring process, as shown in FIG. 6, the CPU 11 determines whether a page change input has been made (200). Here, the page change input is determined by whether or not the page change field 44 on the display screen is clicked by the handwriting input device 14. When a page change input is made, the handwritten data of the current page stored in the work area is stored in the auxiliary storage device 18 (23
0), clear the work area (231), read the image data of the new page and the handwritten data from the auxiliary storage device 18 (232), synthesize the image and the handwritten data, and display them in the display field 41 of the image output device 22. (23
4) The MPX / DMPX 19 causes the page change packet to be created (236), and the event monitoring process ends.

On the other hand, when the page change input is not made in the process 200, it is judged whether or not the handwriting input is made (202). Here, the determination as to whether or not the handwriting input is made is made based on whether or not the pen-down is performed by the handwriting input device 14 in the display field 41 of the display screen. Then, when handwriting is input, it is determined whether the handwriting data is END (204). Here, the handwriting input device 1 determines whether the handwritten data is END.
Judgment is made based on whether or not the pen-up was carried out according to 4.
If the handwritten data is not END, the handwritten data is input from the handwriting input device 14 (210), the input handwritten data is displayed in the display field 41 of the image output device 22 (211), and the handwritten data is displayed in the work area. The data is accumulated (212), the handwritten data encoder 15 encodes the handwritten data (213), the handwritten encoded data is received from the handwritten data encoder 15,
The process returns to the process 204. On the other hand, when the handwritten data is END in the process 204, the CPU 11 adds a header and an end code of the handwritten data to the handwritten data, and MPX / DM
Create a handwritten data packet with PX19 (21
4) The event monitoring process ends.

On the other hand, in the process 202, if the handwriting input is not made, it is judged whether the dialogue end input is made (220). Here, the dialogue end input is determined by whether or not the [end] button in the menu field 43 on the display screen is clicked by the handwriting input device 14. When the dialogue end input is not made, the event monitoring processing is ended, and when the dialogue end input is made, the flag is set to END (222) and the event monitoring processing is ended.

In the voice encoding process, the CPU 11 determines whether the voice data is END (300). Here, whether or not the voice data is END is determined by, for example, when the voice input above a certain level cannot be detected for a certain period of time.
Judge as D. If the voice data is not END, the voice data is input from the voice output device 23 and is encoded by the voice encoding / decoding device 17 to generate voice encoded data (304), and the process 300 is returned to. On the other hand, when the voice data is END in the process 300, the CPU 1
1 adds a header to audio data, MPX / DMPX1
In step S9, a voice packet is created (310), and it is determined whether there is another packet, that is, a page change packet or a handwritten data packet (312). If there is another packet, the voice encoding process is terminated. If there is no other packet, a dummy packet is created (314), and the voice encoding process is ended.

Next, of the dialogue processing in the handset 30 of the communication dialogue system of this embodiment, the details of the downlink processing for transmitting data from the handset 10 to the handset 30 will be described in detail with reference to FIG. To explain. FIG. 8 shows the CPU 31 of the child device 30 of the communication dialogue system of the present embodiment.
It is a flowchart figure of the downstream process of a dialogue process.
Note that in FIG. 8, the CPU 31 of the child device 30 has the communication control device 3
2 shows the processing from the point of time when the respective data obtained by separating the packet received via 2 via the MPX / DMPX 33 is received.

The CPU 31 of the child device 30 uses MPX / DMPX.
When the data separated in 33 is received, the header is analyzed (400). The header indicates the type of data, ″
There are "Page change", "Handwritten data", "Voice data", and "End code", and otherwise processed as an error. Next, the type of data is judged (402), and the data is "Page changed". In the case of ", the handwritten data of the current page accumulated in the work area is saved in the auxiliary storage device 34 (410), and the work area is cleared (411).
Auxiliary storage device 3 for image data and handwriting data of new page
4 (412), the image and the handwritten data are combined and displayed on the display field 41 of the image output device 38 (414), and the process 400 is returned to.

On the other hand, if the data is "handwritten data" in the process 402, it is judged whether the handwritten data is END (420). Here, the judgment as to whether the handwritten data is END is made by the end code included in the handwritten data. If the handwritten data is not END,
The handwritten data is read from the received data (422) and the process 420 is returned to. If the handwritten data is END, it is decoded by the handwritten data decoding device 36 to create handwritten data (424), the handwritten data is displayed in the display field 41 of the image output device 38 (426), and the handwritten data is displayed. (428) and returns to process 400.

On the other hand, if the data is "voice data" in the process 402, it is judged whether the voice data is END (430). If it is not END, the voice data is read from the received data (432) and voice encoded. / The decoding device 37 decodes the voice coded data to generate voice data (434), and returns to the process 430. In the process 430, when the audio data is END, the audio output device 23 reproduces and outputs the audio of the audio data decoded so far (436), and the process returns to the process 400.

On the other hand, when the data is the "end code" in the process 402, the downstream process is terminated, and the process 4 is executed.
In 02, the data is "page change" and "handwritten data"
And other than "voice data" and "end code",
An error message is displayed (440), and the downlink processing ends.

According to the communication dialogue system and the control method therefor of the present embodiment, image data is transmitted from the parent device 10 to the child device 30 prior to the dialogue between the parent device 10 and the child device 30, and the child device is transmitted. 30 and then the master unit 10 to the slave unit 30
Since handwritten data is transmitted to and received from and voice data is exchanged bidirectionally, there is an effect that a dialogue based on an image can be easily performed.

Further, since image data having a large amount of data is transmitted prior to the dialogue and only voice data and handwriting data having a small amount of data are transmitted in real time, the image can be transmitted without using a large capacity high speed communication path. Based on this, there is an effect that it is possible to realize an inexpensive system configuration.

According to the communication dialogue system and the control method thereof of the present embodiment, when the page change input is made in the master unit 10, the page displayed in the master unit 10 is switched and the page change data is transmitted to the slave unit 30. However, since the pages to be displayed on the child device 30 are switched, page switching can be performed synchronously even for a plurality of pages, and there is an effect that it can also be used for a description based on an image (original).

According to the communication interactive system and the control method thereof of the present embodiment, when handwritten data is input in the master device 10, the image data and the handwritten data are combined and displayed on the image output device 22 of the master device 10. However, since the handwritten data is transmitted to the child device 30 and is also displayed on the image output device 38 in the child device 30 in a form of being combined with the image being displayed, it is possible to interact while writing the handwritten data on the image in real time. effective.

[0067]

According to the first aspect of the present invention, the transmission side device transmits the image information in advance and transmits it every time the voice information and the handwriting information are input, and the reception side device transmits the image information in advance. Since it is a communication dialogue system that receives and stores it, and outputs it each time the voice information and the handwriting information are received, when the voice information and the handwriting information related to the image information are transmitted and received in real time, the image information is previously stored. Since it is transmitted, the amount of data transmitted in real time can be reduced,
There is an effect that an inexpensive system configuration can be obtained without requiring a high-speed transmission line.

According to the second aspect of the invention, the transmission side device and the reception side device store the image information and the handwritten information in the first and second storage sections in association with each other, and the transmission side device stores the page information. When input, the handwritten information related to the image information of the page is read from the first storage unit and displayed on the first display unit on the transmission side, and the receiving side device that received the page information determines that the image information of the page is received. Since the handwriting information associated with is read from the second storage unit and is displayed on the second display unit on the receiving side, the handwriting information related to the image information is held on both the transmitting and receiving sides. Therefore, there is an effect that the user can easily interact with the voice based on the image and the handwritten information.

According to the third aspect of the invention, the handwriting information associated with the image information is held on both the transmitting and receiving sides, and when the page information is input to the transmitting side apparatus, the transmitting side apparatus displays the page information. If the image information of the page is already stored, the related handwritten information is read from the first storage unit, displayed on the first display unit, and the page information is transmitted to the receiving side device, and the receiving side device The control of the communication dialogue system according to claim 2, wherein the handwritten information related to the image information of the page of the transmitted page information is already stored and is displayed on the second display unit. Since the method is used, when transmitting and receiving voice information and handwriting information related to image information in real time, since the image information is transmitted in advance, the data amount of transmission data in real time can be reduced and a high-speed transmission path is required. Without There is an advantage of being able to valence system configuration, it is easily interact can effectively by voice based on the image and the handwritten information.

[Brief description of drawings]

FIG. 1 is a configuration block diagram of a master unit of a communication dialogue system according to an embodiment of the present invention.

FIG. 2 is a configuration block diagram of a child device of the communication dialogue system according to the embodiment of the present invention.

FIG. 3 is an explanatory diagram showing a display example of a display screen of the image output device 22 of the master device 10 of the communication dialogue system according to the present embodiment.

FIG. 4 is a master device 10 in the communication dialogue system of the present embodiment.
It is explanatory drawing which shows the sequence of transmission / reception between the child device 30, and the handset 30.

FIG. 5 is a CP of the master device 10 of the communication dialogue system of the present embodiment.
It is a flowchart figure of the main process of the downlink process of a dialogue process in U11.

FIG. 6 is a flow chart of an event monitoring process of the downstream processing.

[Fig. 7] Fig. 7 is a flowchart of a voice encoding process of a downlink process.

FIG. 8 is a CP of the child device 30 of the communication dialogue system according to the present embodiment.
It is a flowchart figure of the downlink process of a dialogue process in U31.

FIG. 9 is an explanatory diagram showing a data flow in a conventional telewriting dialogue.

FIG. 10 is an explanatory diagram showing a data flow in a conventional dialog using a fax.

[Explanation of symbols]

10 ... Parent machine, 11 ... CPU, 12 ... Image input device,
13 ... Image coding device, 14 ... Handwriting input device,
15 ... Handwritten data encoding device, 16 ... Voice input device, 17 ... Voice encoding / decoding device, 18 ... Auxiliary storage device, 19 ... MPX / DMPX, 21 ... Communication control device, 22 ... Image output device, 23 ... Audio output device,
30 ... Slave device, 31 ... CPU, 32 ... Communication control device, 33 ... MPX / DMPX, 34 ... Auxiliary storage device, 35 ... Image decoding device, 36 ... Handwriting data decoding device, 37 ... Voice coding / decoding Device, 38 ...
Image output device, 39 ... Audio output device, 40 ... Audio input device

Claims (3)

[Claims] 1. A transmission side device that transmits image information, voice information, and handwriting information and receives voice information, and a reception side device that receives image information, voice information, and handwriting information, and transmits voice information. In the communication dialogue system described above, the transmission side device has a first storage unit for storing image information, reads image information from the first storage unit in advance, and transmits the image information.
After that, it is a device that transmits every time the voice information and handwriting information related to the image information are input, and outputs each time the voice information from the receiving side device is received. A device having a second storage unit for storing image information, outputting voice information and handwriting information related to the image information each time the voice information is received, and transmitting the voice information to the transmission side device each time the voice information is input. A communication dialogue system characterized by: 2. The transmission side device and the reception side device store the first and second storage units that store image information and handwriting information in association with each other on a page basis, and the first and second storage units that display image information and handwriting information. A display unit, the transmission side device, according to the input page information, reads the handwritten information associated with the image information of the page from the first storage unit, and displays it on the first display unit, A device for transmitting page information to the reception side device, wherein the reception side device reads out the image information stored in the second storage unit and the related handwriting information according to the received page information and displays the second display. The communication dialogue system according to claim 1, wherein the communication dialogue system is a device displayed on a section. 3. The transmission side device stores the input image information in a first storage unit and transmits it, and the reception side device receives the transmitted image information and stores it in the second storage unit. When the page information is input, the transmission side device reads the image information of the page from the first storage unit and displays it on the first display unit, and transmits the page information to the reception side device, The receiving device is
According to the received page information, the image information of the corresponding page is read from the second storage unit and displayed on the second display unit, and when the voice information and the handwriting information are input, the transmission side device outputs the handwriting information. While displaying on the first display unit, the voice information and the handwriting information are transmitted to the reception side device, the handwriting information is stored in the first storage unit in association with the page, and the reception side device, The transmitted voice information and handwritten information are received, the voice information is output, the handwritten information is displayed on the second display unit, and the handwritten information is stored in the second storage unit in association with the page. Then, when the page information is input again, the transmission side device reads out handwritten information related to the image information of the page from the first storage unit, displays the handwritten information on the first display unit, and displays the page information. The said Send apparatus, the receiving-side apparatus,
3. The control of the communication dialogue system according to claim 2, wherein when the page information is received, the handwritten information related to the image information of the page is read from the second storage unit and displayed on the second display unit. Method.