JP2007116527A - Visual two-way real time communication system, program, and information storing medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the economical burden in a television conference system etc., compared with a case actually moving between remote places for the conference. <P>SOLUTION: A terminal capable of transferring/receiving data with other terminals for visual two-way real time communication comprises a memory means for storing at least one of image information, character information and voice information, a means for inputting at least one of image information, character information and voice information, a displaying/reproducing means for at least one of displaying the image information and the character information and reproducing the voice information, an additional information storing means for storing additional information independently inputted by the inputting means or in association with at least one of the image information, the character information and the voice information, and an information exchanging means for displaying/reproducing at least one of image information, character information, voice information and additional information sent by other terminals on the displaying/reproducing means of the terminal. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a visual interactive real-time communication system for sharing information among a plurality of information terminals, for example.

<About network meetings>
Conventionally, when a videophone conference is performed using a computer terminal, a server is prepared for each conference place, and a “shared whiteboard device” and a “shared material display device” as separate processes are included in the server. A known system is known (Patent Document 1).
According to the technique of this patent document 1, drawing data written in a “shared whiteboard device” or “shared” by screen data exchange between independent “shared whiteboard device” and “shared material display device”. It was possible to display, store, exchange, etc. the conference materials stored in the “material display device”.

<Preliminary verification of network meetings at your station>
Conventionally, there has been a technique such as Patent Document 1 as described above, but regarding the display mode to the other party of the video conference when drawing information or the like is actually written in the “shared whiteboard device” However, there are few techniques for verifying with a self-terminal before it is actually displayed on the partner terminal.

<Transmission rate monitor>
Conventionally, in a videophone system, by setting a compression rate that allows transmission of compressed image data to be transmitted in a fixed time, a portion that is desired to be viewed on the image receiving side while ensuring a fixed transmission time of the compressed image data. A technique for displaying the image with high image quality by freely setting the value is disclosed (Patent Document 2).

<Encryption confirmation>
Furthermore, in order to solve the problem in distinguishing between devices that require encryption communication and devices that do not use encryption communication due to cost requirements, etc. Therefore, a technique for transmitting an identifier indicating whether or not it has an encryption function is disclosed (Patent Document 3).

JP-A-10-51581 Japanese Patent Laid-Open No. 10-257461 Japanese Patent Laid-Open No. 2005-236841

<Overall issues>
In each of the above-mentioned patent documents, an actual meeting can be simulated with good operability (Patent Document 1). With a smaller hardware scale, a portion desired to be viewed on the receiving side can be freely selected with respect to an image captured on the transmitting side. The purpose is to provide an image transmission device, an image reception device, an image communication system, and an image communication method that can be set and displayed (Patent Document 2). AV data reception can be performed without eavesdropping or random transmission / reception. An object of the present invention is to provide an AV data wireless communication system capable of freely constructing a system capable of wireless AV data transmission only between an authorized AV data receiving device and an AV data transmitting device (Patent Document 3). However, in video conferences, etc., it is close to the state in which interlocutors can exchange their opinions visually in real time, and actually conduct face-to-face meetings. , It has yet to to make a user-friendly interaction.
Therefore, in the present invention, for example, in a video conference system or the like, a user-friendly dialogue is possible, and the economic burden is reduced as compared with a case where a conference is performed by actually moving between remote locations. One of the problems to be solved.

<Issues about network meetings>
In the above-mentioned Patent Document 1, since the “shared whiteboard device” and the “shared material display device” are clearly separated, for example, the images included in both are displayed in a superimposed manner and displayed on each participant's terminal. It was impossible to share on the screen.
In one aspect of the present invention, for example, in a video conference system or the like, information desired to be communicated to the other party among the interlocutors is shared in real time, and the shared information is stored as necessary for future use. One of the problems to be solved is to provide it.

<Issues about pre-verification of network meetings at your station>
In video conferencing systems, etc., when drawing information etc. scheduled to be presented to the other party of the conference is written, the display mode etc. for the other party of the video conference is displayed in advance on the self terminal before it is actually displayed on the other party terminal. Verification is one of the problems to be solved.

<Problems with transmission rate monitoring>
In Patent Document 2, the maximum compression rate of a screen that can be transmitted is calculated from the image size required by the receiving terminal, the transmission capacity of the line, the transmission time limit, etc., and image transmission is performed using the compression rate. However, there is no consideration as to how much the image quality is degraded when the transmission terminal transmits at the compression rate.
Therefore, in one aspect of the present invention, for example, in a video conference system, one of the problems to be solved is to monitor the quality of data transmitted to the receiving terminal at the transmitting terminal.

<Issues with encryption verification>
In the above-mentioned Patent Document 3, a technique for transmitting an identifier indicating whether or not each device has an encryption function is disclosed. However, in a system that requires confidentiality, such as a video conference system, the data that is actually transmitted is actually disturbed and whether or not each terminal has a cryptographic function, rather than whether each terminal has an encryption function. There is a need to confirm on the transmitting terminal side whether or not the disturbance state has been released with a valid key.
Therefore, in an aspect of the present invention, for example, in a video conference system, it is possible to confirm whether the data transmitted to the receiving terminal is really disturbed by encryption or the like, and to perform data transmission with peace of mind. This is one of the problems to be solved.

  In order to solve the above problems, the present invention provides the following inventions.

<Means for solving problems related to network meetings>
(Invention 1)
A first terminal capable of sending and receiving data to and from other terminals;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
Both the first terminal and the second terminal are
Storage means capable of storing at least one of image information, character information, and audio information;
At least one input means of image information, character information, and audio information;
Display / reproduction means for at least one of the display of image information, character information, and reproduction of audio information;
Additional information storage means for storing the additional information input from the input means alone or in association with at least one of the image information, character information, and audio information in the storage means;
At least one of the image information, the character information, the voice information, and the additional information stored in the storage unit of any terminal from either the first terminal or the second terminal. Exchanging information on the display / playback means of its own terminal,
Comprising
Visual interactive real-time communication system.

According to the first aspect of the present invention, in a scene of information exchange such as a TV conference, it is possible to present the material prepared in advance in the storage means to the reproduction / display means of the participant's terminal. Then, each participant inputs additional information such as a comment on the presented material from the input means such as a light pen for the presented material, and the image information to which the additional information is added is also stored in the storage means. Can be saved. Also, the audio information of the discussion made on the presented material can be stored in the storage means in association with the image information to which the additional information is added. Such image information with audio information and additional information added can be distributed to each participant as an electronic minutes.
Furthermore, after the meeting is over, various materials stored in the storage means, image information with additional information added, audio information, etc. can be freely edited to prepare for the next and subsequent meetings.
The above “storage means capable of storing at least one of image information, character information, and sound information”, “at least one input means of image information, character information, and sound information”, “the image information, The relationship with “display / reproduction means for at least one of display of character information and reproduction of audio information” will be slightly added here.
If the information that can be stored in the “storage unit” is only “image information”, for example, the “input unit” must be capable of inputting at least “image information”. That is, the type of information stored in the “storage unit” needs to correspond to the type of information input in the “input unit”.
On the other hand, the type of information that can be displayed / reproduced in the “display / reproduction unit” and the type of information stored in the “storage unit” and / or input to the “input unit” do not necessarily correspond to each other. There is no need to be. That is, the first terminal has only “input means” and “storage means” for “image information”, and the “display / reproduction means” reproduces only “character information” sent from the second terminal. It may be possible. Even in such a case, only “image information” is sent from the first terminal to the second terminal, and only “text information” is sent from the second terminal to the first terminal. This is because bidirectional communication is possible in such a usage pattern.
This point is also valid in the following (Invention 2) and subsequent inventions.
In other words, in the first aspect, the storage unit, the input unit, and the display / playback unit do not necessarily have the same configuration in each of the first and second terminals.
In one terminal, at least
Input means and display / playback means corresponding to information input by the self-terminal,
Storage means for storing information sent from other terminals and information provided by the terminal, and display / playback means corresponding to the stored information;
Input means for additional information, storage means, display / playback means,
As long as it has.
Each of these input means, each storage means, and each display means may be physically separate or may share the same thing.
In the first aspect of the present invention, the component “from the first terminal and the second terminal is stored in the storage means of any terminal because of its action / effect. An information exchanging means that enables at least one of image information, character information, audio information, and additional information to be displayed / reproduced on the display / reproduction means of its own terminal ”is essential. It is considered to be a part or an important element.

(Invention 2)
A first terminal capable of sending and receiving data to and from other terminals;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
Both the first terminal and the second terminal are
Storage means capable of storing at least one of image information, character information, and voice information;
At least one input means of image information, character information, and voice information;
At least one display / playback means for displaying the image information, the character information, and the voice information;
By transmitting at least one of image information, text information, audio information, and additional information displayed / reproduced on its own display / reproduction means to the other terminal, display / reproduction on the display / reproduction means of the other terminal Mirroring means,
Visual interactive real-time communication system with

Here, “mirroring” means storing a copy of the data in another place in real time and copying the data to another computer on the network. “Mirroring” as a computer term often means a unidirectional copy, but in the present invention, it also means a bidirectional copy that enables additional information transmission.
By the way, Web servers, etc. can reduce the time required for data transfer by releasing mirrored servers to distribute the load, or by collecting data from computers that are close to the network.
Here, in the invention 2, the storage unit, the input unit, and the display / playback unit do not necessarily have the same configuration in the first and second terminals.
In one terminal, at least
Input means and display / playback means corresponding to information input by the self-terminal,
Storage means for storing information sent from other terminals and information provided by the terminal, and display / playback means corresponding to the stored information;
Input means for additional information, storage means, display / playback means,
As long as it has.
Each of these input means, each storage means, and each display means may be physically separate or may share the same thing.

(Invention 3)
At least one of the image information, the character information, the voice information, and the additional information displayed and reproduced on one terminal of each terminal is a signal generated by the timing signal generating unit of the one terminal. The visual interactive real-time communication system according to (Invention 1) or (Invention 2), which is stored in the storage means of the one terminal based on the invention.

(Invention 3-1)
The timing signal of the one terminal is
(1) An input signal from the input means of the one terminal,
(2) a predetermined time set in advance,
(3) The visual interactive real-time communication system according to invention 3, which is generated based on at least one of predetermined time intervals.

(Invention 3-2)
The input signal from the input means of the one terminal is
The visual interactive real-time communication system according to invention 3-1, which is given by clicking a predetermined icon on the display screen of the one terminal or by an input operation from a keyboard.

(Invention 3-3)
The visual interactive real-time communication system according to invention 3-1, wherein the occurrence at a predetermined time or a predetermined time interval includes occurrence at the end of use of the terminal. .

(Invention 4)
At least one of the time at which the timing signal is generated at one terminal of each terminal, the ID information of the other terminal at that time, and the attribute of the display image of the one terminal, the image information, the character information, The visual interactive real-time communication system according to claim 3, further comprising identification information providing means for storing in the storage means of said one terminal as identification information associated with at least one of audio information and said additional information.

According to Invention 4, (1) meeting process confirmation during dialogue, (2) additional writing input,
Will be very easy.
Here, the meeting process confirmation refers to the past during the meeting or after the meeting, and confirms the progress of the meeting according to the elapsed time of the image / character information presented by each interlocutor. It is to be. With the above configuration, it becomes easy to sort the presented image / character information and the like using the ID information of the operator of the other terminal, the timing signal generation time, the attribute of the display image, and the like as keys. The ID information of the operator of the other terminal includes the IP address of the other terminal, the login name of the operator of the other terminal, and the like.
In addition, with additional writing input, a memo can be added with a mouse, keyboard, microphone, etc. to the image / character information presented during the meeting, and another memo can be added later. It means that. By tracking the time addition process of the added memo using the timing signal generation time as a key, it becomes easy to grasp and organize the added memo in time series.

(Invention 4-1)
The display image attribute of the one terminal includes at least one of distinction between a moving image / still image, whether a text document is included, whether a graph / table is included, or additional information is included. The visual interactive real-time communication system according to claim 4,

<Means for solving problems related to pre-verification of network meeting and transmission rate monitoring in own station>

(Invention 5)
A first terminal capable of sending and receiving data to and from other terminals;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
Both the first terminal and the second terminal are
At least one input means for image information, text information, and voice information;
Display / reproduction means for at least one of image information, display of character information, and reproduction of audio information;
With
At least one of the terminals is
Sending data preparation means capable of compressing sending data to the other terminal of each terminal;
Other terminal display state monitoring means for monitoring the state of data to be displayed on the other terminal of each terminal,
Comprising
Visual interactive real-time communication system.

Here, in the invention 5, the storage means, the input means, and the display / playback means are not necessarily required to have the same configuration in each of the first and second terminals.
In one terminal, at least
Input means and display / playback means corresponding to information input by the self-terminal,
Storage means for storing information sent from other terminals and information provided by the terminal, and display / playback means corresponding to the stored information;
Input means for additional information, storage means, display / playback means,
As long as it has.
Each of these input means, each storage means, and each display means may be physically separate or may share the same thing.
According to the fifth aspect of the present invention, when data is transmitted from the self terminal to another terminal, it is possible to check on the self terminal how the data received by the other terminal is displayed.
In the fifth aspect of the present invention, the “other terminal display state monitoring means” for “monitoring” the component “the state of data to be displayed on the other of the terminals” because of its action / effect. It is considered to be an essential part or an important element.

(Invention 6/7)
(A) The other terminal display state monitoring means includes sending data transfer means for passing the sending data to the other terminal prepared by the sending data preparation means to the display / playback means of the one terminal.
Or
(B) The other terminal display state monitoring means includes means for receiving again the data sent from the one terminal and sending it to the display / playback means of the one terminal.
Or
(C) The other terminal includes a reception state return means for transmitting the reception state of the data transmitted from the one terminal at the other terminal to the one terminal from the other terminal,
The other terminal display state monitor means passes the reception state data of the other terminal returned from the other terminal to the one terminal by the reception state return means to the display / playback means of the one terminal, reception state transfer means, Comprising
Or
(D) display / playback status returning means for sending display / playback status data to the one terminal when the data received from the one terminal is displayed / played by the display / playback means of the other terminal; Prepared,
The other terminal display state monitoring means comprises means for sending the display / playback state data of the other terminal returned to the one terminal by the display / playback state return means to the display means of the one terminal.
The visual interactive real-time communication system according to claim 5.

According to the present invention 6/7, when data is compressed and transmitted, the display / playback state in a compressed state in the self terminal is grasped and transmitted from the self terminal according to the degree of the compression rate of the compression. It is possible to check the display / playback state in a compressed state after being recorded and the display / playback state in the receiving terminal.
More detailed, step-by-step effects include changing additional components from (A) to (D) above,
(A): It is possible to confirm the deterioration by the sending data preparation means. The data to be displayed / reproduced on its own terminal is the “pre-compression”, “decompressed”, “encrypted”, “encrypted”, self-prepared for transmission to other terminals. Any of those released on the terminal may be used.
(B): In addition to (A), it can be confirmed by taking into account the deterioration due to the network status.
(C above): In addition to (B), it is possible to confirm the deterioration up to the point of arrival at the second terminal. In this mode, “return the data that has arrived at the other terminal to the transmitting terminal as it is for display / playback”, “after receiving the data that has arrived at the other terminal, display the result compared with the original data・ "Play".
(D): In addition to (C), the display state on the second terminal can be taken into account and can be confirmed directly.
The advantage is obtained.
Of course, instead of “or” as in Invention 6/7, the above configurations (A) to (D) may be provided in a superimposed manner.
On the other hand, there are undeniable disadvantages. In other words, the delay time increases from (A) to (D).
Therefore, (A) to (D) may be selected according to the purpose to be monitored.

<Means for solving the problem of encryption confirmation>
(Invention 8) a first terminal capable of transmitting and receiving data to and from another terminal;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
The first terminal and the second terminal both have display / reproduction means for at least one of the image information, the display of the character information, and the reproduction of audio information;
Key information input means;
Means for disturbing data to be sent to another terminal based on the key information input from the input means;
Means for decrypting the disturbance data received by the self-terminal based on the inputted key information and giving it to the display / playback means of the self-terminal;
Comprising
Visual interactive real-time communication system.

According to the eighth aspect of the present invention, when data is disturbed and transmitted from the own terminal to another terminal, it is possible to first confirm that the other terminal receives data in a state where the data remains disturbed. Then, by inputting valid key information at the other terminal, the disturbance state of the disturbance data transmitted from the other terminal and received by the own terminal is released and displayed / reproduced. Can be confirmed to have been decrypted.
By practicing such a procedure, it is possible to confirm that transmission data is reliably disturbed in a modern visual interactive real-time communication system that strongly requires confidentiality, and then data can be transmitted safely. .
In the present invention 8, due to its action and effect, the components “data disturbing means for disturbing data transmitted from the own terminal” and “disturbed data received by the own terminal are decoded and the display / "Means to give to reproduction means" is considered to be an essential part or an important element.

(Invention 9)
The visual interactive real-time communication system according to claim 8,
Display that gives information indicating whether or not the disturbance state of the disturbance data received by the own terminal has been released to the display / reproduction means of the own terminal when it is determined that the input key information is not correct key information A visual interactive real-time communication system further including a reproduction state confirmation unit.
With the configuration as in the invention 8, the disturbed data is transmitted from the own terminal, the disturbed data is first received at the receiving terminal, and then disturbed by the correct key information at the receiving terminal. It can be confirmed that it has been released, and it can be confirmed at the transmitting terminal that the partner of the receiving terminal is a legitimate partner.
“Displaying / reproducing information indicating whether or not the disturbance state of disturbance data has been released” is not only “displaying / reproducing data that has not been subjected to disturbance cancellation” but also “ Audio information that has not been reproduced is left in a silent state, and an announcement that the disturbance has not been released is played instead, and “only a screen display that the disturbance has not been released” is output.

<Variations of the above inventions>
(Invention 10)
A configuration including three or more terminal groups, any one terminal in the terminal group being a first terminal, and the remaining terminal group having the second terminal according to any one of the above inventions A visual interactive real-time communication system.

  In this way, a one-to-many visual interactive real-time communication system can be constructed in a lecture format. However, the lecture format is not a one-sided lecture, but transmission of additional information, transmission rate monitoring, and encryption confirmation from the one-to-many “many” terminal side to the “1” terminal side. Etc. are also possible.

(Invention 11)
The visual according to any one of the above inventions, wherein at least a part of data exchange between the first terminal and the second terminal is performed via an intermediary terminal or under the control of the intermediary terminal Two-way real-time communication system.

By doing so, there are advantages that the intermediary terminal adjusts the traffic of the entire system, or in some cases, restricts transmission from a specific terminal with high transmission frequency, and enables various controls. .
Here, the intermediary terminal includes a so-called server, but is not limited thereto. All terminals that mediate exchange of data among a plurality of terminals are included.

(Invention 12)
The visual interactive real-time according to claim 11, wherein the intermediary terminal comprises at least a part of the means of the first terminal or the second terminal according to any of the above inventions. Communications system.

  Here, “substitute” means, for example, that the server replaces functions such as the storage means of the first terminal, the additional information storage means, etc. as a client, and only the processing result of the function replaced by the server is obtained. Including sending to the first terminal through the communication line, but is not limited to this.

<Network meeting>
(Invention 13)
It is a terminal that can send and receive data with other terminals for visual two-way real-time communication,
Storage means capable of storing at least one of image information, character information, and audio information;
At least one input means of image information, character information, and audio information;
Display / reproduction means for at least one of the display of image information, character information, and reproduction of audio information;
Additional information storage means for storing the additional information input from the input means alone or in association with at least one of the image information, the character information, and the audio information in the storage means;
Information exchange means for enabling at least one of image information, character information, audio information, and additional information transmitted from another terminal to be displayed / reproduced on the display / reproduction means of the terminal. Terminal.

Here, in the thirteenth aspect, the storage means, the input means, and the display / playback means do not necessarily have the same configuration in the first and second terminals.
In one terminal, at least
Input means and display / playback means corresponding to information input by the self-terminal,
Storage means for storing information sent from other terminals and information provided by the terminal, and display / playback means corresponding to the stored information;
Input means for additional information, storage means, display / playback means,
As long as it has.
Each of these input means, each storage means, and each display means may be physically separate or may share the same thing.

By using the terminal of the present invention, it is possible to present the material prepared in advance in the storage means to the playback / display means of the participant's terminal in the scene of information exchange such as a TV conference. Then, each participant inputs additional information such as a comment on the presented material from the input means such as a light pen for the presented material, and the image information to which the additional information is added is also stored in the storage means. Can be saved. Also, the audio information of the discussion made on the presented material can be stored in the storage means in association with the image information to which the additional information is added. Such image information with audio information and additional information added can be distributed to each participant as an electronic minutes.
Furthermore, after the meeting is over, various materials stored in the storage means, image information with additional information added, audio information, etc. can be freely edited to prepare for the next and subsequent meetings.
Note that, in the present invention, because of its action / effect, the component “the image stored in the storage means of any terminal from either the first terminal or the second terminal” An information exchanging means that enables at least one of information, character information, voice information, and additional information to be displayed / reproduced on the display / reproduction means of its own terminal ” Or it is considered an important factor.

<Preliminary verification of network meeting and transmission rate monitor at own station>
(Invention 14)
It is a terminal that can send and receive data with other terminals for visual two-way real-time communication,
At least one of image information, character information, and voice information input means; and display / reproduction means for at least one of image information, character information display, and voice information reproduction;
Sending data preparation means capable of compressing sending data to the other terminal;
The other terminal display state monitoring means for monitoring the state of data to be displayed on the other terminal;
A terminal equipped with.

According to the terminal of the present invention, when data is compressed and transmitted from the self terminal to another terminal, the display / reproduction state in the compressed state in the self terminal is determined according to the degree of the compression rate of the compression. It is possible to check grasping, grasping of the display / playback state in a compressed state after being transmitted from the own terminal, and grasping of the display / playback state at the receiving terminal.
In the present invention, the “other terminal display state monitoring function” for “monitoring” the component “the state of data to be displayed on the other of the terminals” is essential because of its function and effect. It is considered to be an important part or an important element.

<Encryption confirmation>
(Invention 15)
It is a terminal that can send and receive data with other terminals for visual two-way real-time communication,
Display / reproduction means for at least one of image information, display of character information, and reproduction of audio information;
Key information input means;
Means for disturbing data to be sent to another terminal based on the key information input from the input means;
Means for decrypting the disturbance data received by the self-terminal based on the inputted key information and giving it to the display / playback means of the self-terminal;
A terminal equipped with.

According to the terminal of the present invention, when data is disturbed and transmitted from its own terminal to another terminal, the reception in the state where the data remains disturbed can be first confirmed at the other terminal. Then, by inputting valid key information at the other terminal, the disturbance state of the disturbance data received from the other terminal is released and displayed and reproduced, and thereby the disturbance state of the disturbed data is released. I can confirm that.
By practicing such a procedure, it is possible to confirm that transmission data is reliably disturbed in a modern visual interactive real-time communication system that strongly requires confidentiality, and then data can be transmitted safely. .
According to the fifteenth aspect of the present invention, the components “data disturbing means for disturbing data transmitted from the own terminal” and “disturbance data received by the own terminal are converted into the input key because of its action / effect. The means for decoding based on the information and giving it to the display / reproduction means of the terminal itself is considered to be an essential part or an important element.

<Network meeting>
(Invention 16)
At least on the computer,
As a terminal that can send and receive data with other terminals for visual two-way real-time communication,
A storage function capable of storing at least one of image information, text information, and audio information;
An input function of at least one of image information, text information, and audio information;
A display / playback function for at least one of the image information, the display of character information, and the playback of audio information;
An additional information storage function for storing additional information input using the input function alone or in association with at least one of the image information, character information, and voice information;
It is possible to display / reproduce at least one of the image information, the character information, the voice information, and the additional information stored in the storage unit of another terminal on the display / reproduction unit of the terminal. Information exchange function,
Visual interactive real-time communication program that realizes

Here, in the invention 16,
at least,
Input function and display / playback function corresponding to the information entered on the terminal itself,
Storage function for storing information sent from other terminals and information provided by itself, and display / playback function corresponding to the stored information,
Input function for additional information, memory function, display / playback function,
As long as it has.

According to the terminal configured by executing the program of the present invention, the material prepared in advance in the storage means is presented to the playback / display means of the participant's terminal in the scene of information exchange such as a TV conference. Is possible. Then, each participant inputs additional information such as a comment on the presented material from the input means such as a light pen for the presented material, and the image information to which the additional information is added is also stored in the storage means. Can be saved. Also, the audio information of the discussion made on the presented material can be stored in the storage means in association with the image information to which the additional information is added. Such image information with audio information and additional information added can be distributed to each participant as an electronic minutes.
Furthermore, after the meeting is over, various materials stored in the storage means, image information with additional information added, audio information, etc. can be freely edited to prepare for the next and subsequent meetings.
It should be noted that in the present invention 16, due to its action / effect, at least one of the constituent elements “the image information, the character information, the voice information, and the additional information stored in the storage means of another terminal” The “information exchange function that enables one to be displayed / reproduced on the display / reproduction means of the own terminal” is considered to be an essential part or an important element.

<Preliminary verification of network meeting and transmission rate monitor at own station>
(Invention 17)
At least on the computer,
As a terminal that can send and receive data with other terminals for visual two-way real-time communication,
A display / reproduction function for at least one of image information, character information, and sound information input function and at least one of image information, character information display, and sound information reproduction;
A send data preparation function capable of compressing send data to the other terminal;
The other terminal display state monitor function for monitoring the state of data to be displayed on the other terminal;
Visual interactive real-time communication program that realizes

According to a terminal configured by executing this program, when data is compressed and transmitted from the self terminal to another terminal, the compressed state in the self terminal is determined according to the degree of compression rate of the compression. It is possible to check the display / playback state at the mobile terminal, the display / playback state in the compressed state after being sent from the self terminal, and the display / playback state at the receiving terminal.
In the present invention, the “other terminal display state monitoring function” for “monitoring” the component “the state of data to be displayed on the other terminal” is essential because of its function and effect. It is considered to be a part or an important element.

<Encryption confirmation>
(Invention 18)
At least on the computer,
As a terminal that can send and receive data with other terminals for visual two-way real-time communication,
A display / reproduction function for at least one of display of image information, display of character information, and reproduction of audio information;
Key information input function,
A function of disturbing data to be sent to another terminal based on the key information input using the input function;
A function of decrypting the disturbance data received by the own terminal based on the inputted key information and giving it to the display / playback means of the own terminal;
Visual interactive real-time communication program that realizes

According to a terminal configured by executing this program, when data is disturbed and transmitted from its own terminal to another terminal, the other terminal receives data in a state where the data remains disturbed first. I can confirm. Then, by inputting valid key information at the other terminal, the disturbance state of the disturbance data received from the other terminal is released and displayed and reproduced, and thereby the disturbance state of the disturbed data is released. I can confirm that.
By practicing such a procedure, it is possible to confirm that transmission data is reliably disturbed in a modern visual interactive real-time communication system that strongly requires confidentiality, and then data can be transmitted safely. .
In the present invention 18, due to its action and effect, the constituent elements “data disturbing means for disturbing data transmitted from the self terminal” and “disturbance data received by the self terminal are decoded and the self terminal is The function given to the display / reproduction means ”is considered to be an essential part or an important element.

(Invention 19)
An information storage medium storing the program according to any one of (Invention 16) to (Invention 18).
The present invention describes the claims in the categories of the system, the program for executing the function of the system, and the medium on which the program is recorded as described above. However, the present invention is not limited to these forms. There is a need for all forms of program products for executing functions, signals used to execute the functions, and other equipment other than computer terminals used to execute programs used to implement the system. In the claims.
Also, for each invention of the above system category, “˜means” is replaced with “˜function”, and the invention of the category of the program for causing the computer to realize such each function, or the information storage storing the program Of course, the invention of the media category can be easily derived.

As mentioned above, although the representative example of each (invention) which concerns on this application was described as mentioned above, this invention is not limited to these, All the said each (invention) or some structural requirements are mentioned. Of course, it is possible to construct another invention in an appropriate combination for any purpose. By doing so, in the “visual two-way real-time communication system” directed by the present invention, the two-way communication is more realistic. Of course, a rich system can be obtained.
For example, it is possible to combine the inventions included in <Network Meeting>, <Transmission Rate Monitor>, and <Encryption Confirmation> so as to superimpose the effects of all these inventions. Is possible.
Accordingly, it is obvious to those skilled in the art that such various variations may be added to the scope of claims by amendment of procedures, so-called domestic priority claim, divisional application, etc. in the examination process of the present application. Is predictable.

  In the device, the system invention, the program invention, and the medium invention, the description order of each component does not stipulate the chronological order in which each invention is implemented. Conversely, within the scope of achieving the object and effect of each invention, the operation sequence of the functions realized by each component can be freely changed when implementing each invention, and such technology is also within the scope of each invention. Belongs.

The following is a list of terms used in this specification.

Table 1 Significance of terms used in this specification

  According to the embodiment of the present invention, for example, in a video conference system or the like, a user-friendly dialogue is possible, and an economic burden can be reduced as compared with a case where a conference is performed by actually moving between remote locations. It becomes possible.

Various embodiments of the present invention will be described below with reference to the drawings. However, the following description is merely an example of the present invention, and the technical scope of the present invention is not limited by the following description.
It should also be understood that the techniques of the present invention are suitable for any known or future developed visual interactive real-time communication system.

<System configuration>
FIG. 1 is a diagram showing a schematic configuration of a network system to which an information sharing technique can be applied according to an embodiment of the present invention. In FIG. 1, the terminal 1A and the terminal 1B are connected via the network 2, but the terminals 1A and 1B may be directly connected by a communication cable such as a LAN cable. A plurality of types of networks such as the Internet and existing telephone lines may be used in combination. As the connection form of these terminals, in addition to the one-to-one connection not via the server as shown in FIG. 1, one-to-one connection and one-to-one connection via the server are used for authentication. There are one through a server only, one-to-many connection in which one terminal and a plurality of terminals are connected through a server, and a group conversation without using a server. The following explanation will be made mainly by taking a one-to-one information sharing aspect as an example, but such a one-to-many aspect of the lecture format and each participant arbitrarily to one or more of different participants Of course, many-to-many aspects such as transmitting information are also within the scope of the present invention.
In addition, the following description will be made taking an example of a form of peer-to-peer connection (peer to peer) directly connected via a communication cable, but one-to-one, one-to-many, and many-to-many performed via a server. Of course, the embodiments are also within the scope of the present invention. The advantages of performing communication in the form of peer-to-peer are that the server is not burdened and that high-speed communication can be realized by direct connection without using a relay server. Applications suitable for peer-to-peer connections include video applications (such as video chat) that handle large amounts of data and applications that require real-time processing (such as whiteboards). Is mentioned.

In FIG. 1, terminals 1A and 1B are connected to a network 2 as transmission means via nodes 2a and 2b, respectively, and a bidirectional dialogue between these terminals 1A and 1B is possible. For example, the terminal 1A generally includes a PC main body 100A that executes main information processing functions such as a communication control function, and a display device 200A. Similarly, the terminal 1B includes a PC main body 100B and a display device 200B.
The information sharing technique according to the embodiment of the present invention is connected directly or indirectly via the transmission means as described above (for example, in the case of FIG. This includes selectively transmitting a part of the display screen of the terminal 1A from the terminal 1A to the terminal 1B from the set terminal 1A to the second terminal 1B.

  FIG. 2 is a block diagram of a configuration of a terminal that can implement the information sharing technique according to the embodiment of the present invention. The terminal 1A (1B) includes a display device 200A (200B) such as a monitor, a PC main body 100A (100B) that performs video processing, an external recording device 700A (700B), and a pointing device 400A (400B) such as a mouse. And a keyboard 500A (500B), a microphone 600A (600B), and a speaker 300A (300B). Since the terminal 1B has the same configuration, only the terminal 1A will be described below.

  Note that the specific examples of the present invention described in the embodiments or examples of the present invention are realized by the CPU controlling each hardware described above based on software stored in the PC.

The PC main body 100A includes a control unit 120 that implements information sharing technology related to the invention of voice data and image data from the other party through the network 2, and storage means for storing image data to be transmitted and received 130 (D / B in the figure), a drawing unit 140 having a VRAM 141 for displaying a predetermined pattern on the display device 200A according to an instruction from the control unit 120, and position information from a pointing device 400A such as a mouse. A data input / output unit 150 (I / O in the figure) that enables capturing and data exchange with the external device 700A, a sound source controller 160 for controlling the speaker 300A from which the other party's voice is output, a camera 800A.
The external recording device 700A includes, for example, a magnetic tape, a CD, an optical disk, a hard disk, a DVD, a flash memory, and the like, and includes a computer program that implements the information sharing technique according to the present invention, transmission data from the other party, and the like. Stored.

  The information sharing technique according to the embodiment of the present invention is particularly effective for interactive dialogue executed between the point A where the terminal 1A having the above-described structure is installed and the point B where the terminal 1B is installed. It is.

  FIG. 3 shows a configuration diagram that is easy to understand visually when the information sharing technique according to the embodiment of the present invention is applied to the interactive dialogue. As shown in the figure, in the embodiment of the present invention, a conversation person 3A at the point A on the left side of the drawing and a conversation person 3B at the point B on the right side of the drawing, which is away from the point A, A device (see FIG. 2) (hereinafter referred to as “terminal device” 900A (900B) including both of them) is used for mutual dialogue and information exchange.

  The dialogue includes images of the face of each conversation person 3A (3B) captured by the cameras 800A (800B) at both points A and B, remarks from each conversation person 3A (3B) collected by the microphone 600A (600B), The presentation of various data stored in the external storage device 700A (700B) on the screen and the writing of the various data presented on the screen by the mouse 400A (400B) or the keyboard 500A (500B) to each other Is transmitted to the terminal device 900B (900A) in real time, and is mainly performed by image display / audio reproduction.

  FIG. 3 shows various features of each embodiment of the present invention on the screen of the display 200A (200B) of each terminal device 900A (900B). FIG. 4 shows an enlarged view of this feature. .

  In FIG. 4, numbers (1) to (5) are numbered. Hereinafter, screen display by these numbers and each embodiment of the present invention realized through the screen display will be briefly described below. An outline will be described, and then each embodiment will be described in detail. The screen display in this figure will be described on the assumption that it is a display at point A in FIG.

  (1) is a control screen. By checking and moving the operation icon (volume adjustment knob) on this screen by operating the mouse, etc., control of camera imaging, image transmission, audio transmission, volume adjustment, transmission data encryption mode switching, etc. Do. In this case, “transmission data encryption mode switching” means that “the encryption key is switched to a common key that is automatically set in advance and uses a pre-designated mutual key. This means that a state such as “not performed” is selected.

(2) is a phase manual image reception screen. The face image of the conversation partner at the point B captured by the camera 800B in FIG. 3 is displayed on the display 200A at the point A. The face image of the other party may be a completely real-time moving image, a moving image compressed to reduce the transmission capacity, or an image that transmits a still image intermittently.
The frame image received by the terminal 1B is enlarged or received after being received so that it can be displayed in the display window of the other party image reception screen (2) or the other party image reception screen (4) set in advance by the other party conversation person 3B. It is displayed on the display screen after processing such as reduction.

  Further, as shown in FIGS. 5A to 5C, at each of the points A and B, each conversation person causes the other party's face image to be displayed on his / her display 200A (200B). The position of the image capturing unit (the image capturing unit is disposed in the display 200A (200B)) of the camera 800A (800B) of the apparatus is above the position of the eyes of the partner face image and as close to the face as possible. As described above, moving the position of the camera 800A (800B) of the self-terminal device or moving the window ((2) in FIG. 4) displaying the face image in the display 200A (200B) Preferred from a contact point of view. At this time, an image captured by the camera of the self terminal is monitored to check whether the self image is captured in a well-balanced manner. When the balance is poor, it is better to adjust the imaging state by changing the angle of the camera.

  (3) is an image capture frame / transmission frame. The image capture frame moves image information, character information, etc., which is displayed on its own display 200A to the other party (3B) of the dialogue, into the image capture frame by operating the keyboard 500A or the mouse 400A. Or, conversely, by moving the image capture frame / transmission frame over the image, the image information displayed in the image capture frame is temporarily stored in the image organizer frame (5) described later. Save or transmit the image information displayed in the image capture frame to the other party (3B) manually or automatically in real time, or generate a whiteboard with the selected image Bears the function of

  For image information and text information that can be superimposed on the image capture frame, PowerPoint (registered trademark of Microsoft Corporation), spreadsheet software (eg “Excel” etc.), word processing software (eg “word” etc.), browser, etc. Material data (text, photograph, video) generated by the application software is included, but is not limited to this. It is possible to select all the images displayed on the display or any part thereof and send them to the other party, and further generate a whiteboard with an image on the other party's screen.

  (4) is a partner image reception screen. This is also called a reception frame. This is a window that displays information such as images and characters sent from the terminal device of the other party (3B). It is also possible to move information such as images and characters displayed here to an image organizer frame (5) described later.

(5) is an image organizer frame.
The above image capture frame / transmission frame or image / character information captured on the other party's image reception / capture screen is captured on this screen, and a whiteboard with an image is generated for the captured image. Enter a memo with a keyboard or the like, store image / sound information with the memo in the external storage device 700A (700B), and input a voice comment on the captured image / character information. It has a function of storing in the external storage device 700A (700B) in association.

Hereinafter, (1) control screen, (2) phase manual image reception screen, (3) image capture frame / transmission frame, (4) partner image reception screen (reception frame), (5) image organizer frame, Each embodiment of the present invention using the above functions will be described.
Example 1 describes <Network Meeting>, Example 2 <Preliminary Verification of Network Meeting at Local Station>, Example 3 <Transmission Rate Monitor>, and Example 4 <Encryption Confirmation>.

  Here, <network meeting> means a mode in which a plurality of interlocutors exchange information via PC terminal devices coupled by communication means such as a network in a state where they are isolated from each other.

  <Preliminary verification of network meeting at own station> means that in <network meeting>, other information may be displayed in a pseudo or simulated manner before being transmitted to other terminal devices. Simulates the state in which information is transmitted to the terminal in a closed system inside the own terminal, the display mode of the image / character information displayed on the display 200B of the partner terminal device, the quality of the own voice reproduced by the partner terminal And checking the volume. As a result, it is possible to prevent a situation in which the camera of the self-terminal device does not face its own face or the microphone of the self-terminal device is turned off.

  <Transmission rate monitor> refers to compressed data that has been compressed in the own terminal, compressed data that has been sent from the own terminal in <network meeting> etc. (including communication between other terminal devices) This is a mode in which the compressed data and the compressed data sent from the partner terminal are decompressed by the own terminal, and the extent to which the data quality deteriorates in the data compression / transmission process is monitored. By performing such monitoring, it is possible to confirm whether the image / character data used in <Network Meeting> or the like is reliably transmitted with the quality intended by the sender. If you feel that the data quality is lower than the required level, prioritize multiple data to be transmitted to the other party, lower the compression ratio of the higher priority data, and conversely the compression ratio of the lower priority data. Can be taken. For example, in the case of X-ray photography, it is necessary to send a high-detail image with a reduced compression rate, but it is flexible for situations where there is not much problem with sending your own face etc. with high compression. It is possible.

  <Encryption confirmation> indicates that when the data transmitted from the self-terminal device is encrypted or the like, when it is received at the other-party terminal, first, the other-party terminal device displays the encrypted disturbance data. Next, the main sequence mainly includes a mode in which a series of sequences in which the disturbance state is canceled by inputting correct key information can be confirmed on the self-terminal side.

<Network meeting>
Reference is made to FIGS. 6 to 8A and 8B.
Please refer to FIG. Here, the image information stored in the external storage device 700A of the self-terminal device, for example, displayed on the screen, the screen obtained via the Internet, etc. are displayed in the image capture / transmission frame (3) of FIG. And is captured (FIG. 6A). Here, “capture” means that an image capture / transmission frame (3) (square frame) surrounds a part of the display screen (a) in FIG. This means that only that part is designated for transmission to the counterpart terminal device.

The captured image or the like is displayed after being transmitted to the counterpart terminal device by an operation on the control screen (1) in FIG. 4 (FIG. 6B).
In this way, by sending only a portion of an image that requires special attention to the counterpart terminal device, communication focusing on more important information can be achieved as compared with the case of transmitting the entire screen. It becomes possible. Also, the agenda can be focused.

  Further, without waiting for the operation on the control screen (1) in FIG. 4, the image information captured on the self-screen may be automatically transmitted (for example, every 2 seconds) to the counterpart terminal device. Also, so-called moving picture transmission is possible if 30 frames or more are sent per second.

  In this way, by sharing and browsing the materials, it becomes easy for all the interlocutors to have a common recognition.

Reference is now made to FIG.
FIG. 7A shows the function of the virtual whiteboard. The virtual whiteboard includes, for example, various images and character information pasted in the image organizer of FIG. 4, and includes a virtual white area similar to, for example, a white white board, and the like. When the character information or the like can be written not only on the white board-like area but also on the various images, it means that the various images are also included in the virtual whiteboard.

The whiteboard with an image can be generated by a transmission frame and an image organizer.
In FIG. 7, an area similar to a white white board is arranged in the lower right part adjacent to the screen downloaded from the Internet pasted in the image organizer. An extending arrow is filled with a light pen on the screen downloaded from the Internet.

The characters and drawing information entered in the virtual whiteboard in this way are displayed on the display 200B of the counterpart terminal device in the same manner as the screen FIG. 7 pasted in the image organizer (FIG. 4). (B)).
In a more specific aspect, when the virtual whiteboard function with image on the transmission side is turned ON, the image capture screen (transmission frame) is automatically switched to the virtual whiteboard mode with image on the transmission side, and image display is performed on the reception side. The screen automatically switches to virtual whiteboard mode with images. In addition, it is possible to perform line drawing and text drawing on the displayed image on both the sending and receiving sides in the same way as with the paint software, and the results can be saved in the image organizer. Is possible. Here, the “virtual whiteboard mode” means a state in which the virtual whiteboard is displayed.

  Also, as shown in FIG. 8A, the control screen (see (1) in FIG. 4) is simultaneously displayed on the display screen on the display of the self-terminal device while drawing with a light pen or the like, and comment entry with the keyboard 500A. The voice comment by the microphone 600A can also be added by the operation via, and the whole can be stored in the image organizer (see (5) in FIG. 4). It can also be attached to an email. The recipient who has received the e-mail can decompress the information attached to the e-mail using the receiving image organizer.

  The mail data stored in the image organizer is transmitted to the counterpart terminal device by sending an electronic mail, and displayed on the image organizer of the counterpart terminal. On the control screen of the counterpart terminal (see (1) in FIG. 4), all of the transmitted images, drawing comments, and voice comments are reproduced in the same sequence as the sequence performed on the transmitting terminal side.

FIG. 8B shows a more detailed usage mode. In this figure, the message “Please write the explanatory text of the“ Manual ”in the attachment” in the body of the email sent from the sending terminal to the receiving terminal is written. The image information of the “manual” part and the voice message “It is better to write the explanation of this part a little easier to understand” are included. Received, displayed and played.
According to such an aspect, it is possible to send an emotional mail to the other party through voice. In addition, phrases that are difficult to write with characters can be easily added.

The effects of the first embodiment as described above will be described in more detail including additional configurations. The following are the effects of the present embodiment, and are not necessarily included in the effects of each “(invention)” described in the above “Means for Solving the Problems”.
On the other hand, the correction which adds the additional structure in the following description to said each (invention) can be naturally predicted by those skilled in the art.

(1) Everyone can see the same material focused on a specific image area:
The display is limited in size, and it is important to extract the topic area and display it with image focus. This is a function of the transmission frame.

(2) Anyone can write at and around the image focused:
Then, a whiteboard area is generated in a form where the image is focused or a target image is captured so that text input or pen input can be performed on the whiteboard area including the captured image. .
By doing so, it is possible to make a meeting while looking at the image, and the captured image and the information added on the whiteboard can be integrated and stored or redisplayed. This function is one of the features that can be realized in the dialog through the monitor. This is an image capture type whiteboard function of a transmission frame.

(3) Save can be done with one click or automatically and paged:
Thereby, it is possible to confirm through a stored image of the meeting process. This is an advantageous function of the image organizer.

(4) The meeting is identified as to which meeting session and which material it is:
This is one function of an image organizer that can automatically generate a database. In particular, in a group dialogue, it is very easy to check a document when the document is identified along with the session name.

(5) You can redisplay the saved items and rewrite them as common materials:
This is an integrated function of image organizer redisplay and transmission frame.

(6) Furthermore, the above-described image transmission dialogue integration technology can be applied not only to one-on-one meetings but also to group meetings. As an application example, as a group meeting, about 4 to 5 people use the software of the present invention over the network, without using a camera (without a face image), voice and this transmission / reception frame, white board with image, By using the image organizer in an integrated manner, it is possible to make a document-sharing type meeting. Until now, all the members have gathered in the conference room, but it is possible to have a meeting at their own seat. In the case of a one-on-one meeting, it is also effective to perform voice using a telephone and use the software of the present invention to share material information.

(Supplementary explanation on (2) and (5) above)
Conventionally, the document and the whiteboard were integrated, and generation, writing, voice input, etc. were not possible. However, with the technology of this embodiment, the necessary images are captured and the whiteboard can be freely sized around it. Can be created, written, and voiced.
Furthermore, the document and the whiteboard are integrated, and the whiteboard area including the information can be freely sized for a part or the whole of the additional information body generated, written, or input by voice. Can be created, written, and voiced. This can be done during interaction over the network. For this purpose, integrated utilization of a transmission frame (reception frame) and an image organizer is effective.
By using the transmission frame, the material information and the information added on the whiteboard are integrated, generated, written, and voice input, etc., then saved in the image organizer and redisplayed there On the other hand, in the transmission frame, it is integrated with the information added on the whiteboard, and repeated information addition such as generation, writing, and voice input can be performed. Of course, the repeated information addition can be performed on the entire preceding information or on the selected portion until the monitor size allows.
Previously, there was no software adapted to such a visual dialogue, so smooth thinking and cognitive dialogue could not be realized.
In this embodiment, the focus image / material is taken out, and the whiteboard area including the information is generated in a free size, stored, and then re-selected to create a new whiteboard area in the free size. But it can be generated. Can facilitate the development of creative dialogue. In addition, the history also remains, so you can reconsider. Such easy execution of visual analog processing based on digital processing will serve as a breakthrough tool for creative dialogue.

<Application of this embodiment>
This embodiment provides a creative dialogue using images and materials in a visual dialogue via a network. By using this embodiment, it is possible to quickly communicate images and materials overcoming distances, and as a result, it is possible to reduce travel expenses and divert travel time to others. It also contributes to the conservation of the environment by reducing people's mobile energy. In addition, the creation of functions that were difficult to achieve face-to-face, such as the formation of a focused common vision space, bidirectional writing, storage, redisplay, and additional writing, can provide multi-functional creative dialogue equivalent to or better than face-to-face .

The visual dialogue in the present embodiment means at least a dialogue performed by displaying images / materials on a display, and a facial image has never been done but is not an essential element. Also, voice is not necessarily an essential element of the present embodiment, such as using a telephone.
Further, this embodiment can be used not only for one-on-one but also for creative dialogue between groups. It can be a server or client type via a server, or can be used for peer-to-peer such as one-to-one without a server at the time of dialogue or group dialogue. In the description of the present embodiment, one-to-one peer-to-peer and group conversations of about 4 to 5 people (without a server) are mainly described as examples.

  For the realization of creative dialogue, participants' emotional exchange and cognitive (thinking) exchange are important. Emotional exchange requires facial expression observation, line-of-sight matching, and line-of-sight recognition. In cognitive (thinking) exchange, real-time, analog-type handling of all images and materials is important. This embodiment proposes a method for handling images and materials mainly focusing on cognitive (thinking) exchange in order to realize creative dialogue.

<Explanation of image organizer and functions of its transmission frame>
Image Organizer Features Image Organizer includes, but is not limited to:

(A) Basic function (A-1) Saving of the image area selected in the transmission frame (reception frame) (A-2) Image area selected by the image organizer itself (the image organizer itself has an image capturing function) Have)
(A-3) Display of stored image (A-4) Generation of image capture type whiteboard (A-5) Storage of processed image in image organizer (Generate image capture type whiteboard, save it, image Import)

(B) Extended function (B-1) Full-screen whiteboard generation without image capture (B-2) Search sort based on stored image identification information (B-3) Print function (B-4) E-mail attachment function (B -5) Image generation function with sound (B-6) Image storage, playback and editing functions with sound

(C) Application development (C-1) Recording during dialogue (selected audio, face image, document image, etc.)
(C-2) Playback and editing of what was recorded during the dialogue (C-3) Image capture type whiteboard and series of processing for still images recorded during the dialogue

(D) Function of transmission frame (D-1) Saving of image area selected in transmission frame (save destination is image organizer)
(D-2) Transmission of the image area selected in the transmission frame to the dialog destination (D-3) Capture the image selected in the transmission frame and generate a whiteboard. Images and whiteboards can be written in both directions, sharing information. Images and whiteboard content can be stored in the image organizer.

(E) Function of reception frame (E-1) Display and storage of received image (F) Explanation of integration of image organizer and transmission frame

As an example,
(F-1) Prepare PowerPoint as meeting materials in advance. (F-2) Overlay the transmission frame on the PowerPoint display area at the time of the visual dialogue. Transmission display (F-3) If necessary, use the transmission frame. Extract the focal point and generate a whiteboard (F-4). Write to the whiteboard in both directions (representative entry type may be used)
(F-5) Save contents as needed (save destination is image organizer)
(F-6) When writing on the whiteboard and a place to be focused particularly occurs, the image can be captured by the image organizer's image capture function.
(When divergent thinking causes a separate meeting)
(F-7) Repeat these steps. Save page is filled.
(F-8) Review the page with the image organizer and deepen the meeting.
At this time, the information stored in the image organizer can be shared by overlapping the transmission frame on the display area of the image organizer.
As a simple case of this function, an image organizer storage information transmission display sharing function and a whiteboard processing transmission display sharing function may be added to the image organizer so that transmission frames are not overlapped. (Corresponding to meetings with convergent thinking)

  As described above, the combination of the whiteboard generated in the transmission frame and the image organizer, and the combination of the image organizer and the transmission frame, etc., display the image freely, enter additional information, and use the history of the meeting process It can be used as a powerful support tool for creative dialogue.

  This embodiment is a powerful support tool for creative dialogue, but it can also be applied to a system closed with its own terminal and used for internal communication tools with itself and creative dialogue with self. It can be used as a special case.

<Preliminary verification of network meetings at your station>
In the present embodiment, a case will be described in which each usage mode described with reference to FIGS. 6 to 9 in the above embodiment is simulated only on its own terminal.
Reference is again made to FIGS. 6 to 8A and 8B.

In the present embodiment, the “preliminary verification mode” in the control screen of FIG. 4 is selected, and the vicinity of the portion surrounded by a circle in the display screen as shown in FIG. When only the capture portion is enclosed in the transmission frame (3) and simulated, the capture portion displayed on its own screen and on the counterpart terminal device in normal transmission is displayed on its own display.
Furthermore, as shown in FIG. 7, how the entry on the transmission-side virtual whiteboard is reflected on the reception-side whiteboard can be displayed on the terminal device in a pseudo manner.
Similarly, as shown in FIGS. 8A and 8B, a mail attached with a material including image information and audio information can be transmitted in a pseudo manner and displayed / reproduced on the own terminal.
By having such a function, it is possible to verify in advance what kind of display will be performed on the counterpart terminal before actually transmitting image information etc. to the counterpart terminal. It becomes.

  In the following description, the meaning of the term “compression” is different from the so-called “compression” in the present invention, such as “reducing the amount of data”, such as reducing the frame rate or reducing the image size. Widely regarded as “compression”.

<Transmission rate monitor>
Please refer to FIG.
In (a) of this figure, (1) (control screen) of FIG. 4 is shown. In this control screen, by adjusting the level of the control bar, the frame rate (proportional to the compression rate) of the image data sent from the own terminal device (PC-A) to the counterpart terminal device (PC-B) It is possible to change.
In this embodiment, as shown in FIG. 10, the compressed data (1) compressed by the compression means in the own terminal device, and the compressed data sent from the own terminal device (PC-A) ( 2) By expanding at least one of the compressed data (3) sent from the counterpart terminal device (PC-B) and giving it to the display / playback means 200A, the self-terminal device (PC-A) The image quality degradation occurs in the transmission process from the self-terminal device (PC-A) to the counterpart terminal device (PC-B), depending on the degree of compression rate according to the frame rate selected in step 3, and the counterpart terminal It is possible to monitor the quality of the image displayed on the apparatus (PC-B).
If the quality of the image displayed on the monitored counterpart terminal device (PC-B) is lower than the desired level, the compression rate of the image on the own terminal device (PC-A) is reduced, or the counterparty It is possible to take measures such as improving the image quality by reducing the size of the image displayed on the display screen of the terminal device (PC-B).

<Encryption confirmation>
A first aspect of the present embodiment will be described.
Please refer to FIG.
This is an encryption + decryption scheme close to that known in the prior art.
The image (1101) of the terminal at point A is encrypted (1102) with the key K1 to disturb the image (1105). If this disturbed image is transmitted to the point B (1106) and decrypted with the disturbing key K1 at the point A at the point B (1110), a screen (1113) from which the disturbance is released is obtained.
On the other hand, if the decryption is performed at a point B other than the disturbance key K1 at the point A (1114), only the screen (1117) where the disturbance is not canceled is obtained, and the decryption fails.
Next, the image (1121) of the terminal at point B is encrypted (1122) with the key K2 to disturb the image (1125). If this disturbed image is transmitted to the point A (1126) and decrypted with the disturbing key K2 at the point B at the point A (1130), a screen (1133) with the disturbance removed is obtained.
On the other hand, if decryption is performed at a point A other than the disturbance key K2 at the point B (1134), only the screen (1137) where the disturbance is not canceled is obtained, and the decryption fails.
Next, an aspect in the present embodiment in which some additional conditions are added to such a conventional encryption scheme will be described.

In this embodiment aspect,
(1) Data transmission is always encrypted using a predetermined encryption key.
(2) Use the same encryption key and decryption key ("K1" in the following description).
It has a precondition.

Please refer to FIG.
The image (1201) of the terminal at point A is encrypted with the common key K1 (1202), and the image is disturbed (1205). If this disturbed image is transmitted to the point B (1206) and decrypted with the common key K1 at the point B (1210), a screen (1213) from which the disturbance is canceled is obtained.
On the other hand, if the decryption is performed with the key other than the common key K1 at the point B (1214), only the screen (1217) that does not release the disturbance is obtained, and the decryption fails.
Next, the image (1221) of the terminal at the point B is encrypted (1222) with the common key K1 to disturb the image (1225). If this disturbed image is transmitted to the point A (1226) and decrypted with the common key K1 at the point A (1230), a screen (1233) from which the disturbance is canceled is obtained.
On the other hand, the image of the terminal at point B (1237) is encrypted with a key other than the common key K1 (1238) to disturb the image (1241), and this disturbed image is transmitted to point A (1242). In this case, even if decryption is performed with the common key K1 (1246), only a screen (1249) in which the disturbance is not canceled can be obtained.

By adopting such a configuration, when the screen transmitted from the point B can be decrypted with the common key K1 at the point A,
(1) It is encrypted using the common key K1 at the point B as agreed.
as well as,
(2) If the fact of (1) is assumed, the disturbance sent from the point A and encrypted with the common key K1 is “should” that the image is also used for the decryption at the point B. It must be "decrypted" correctly with a certain common key K1.
Can be confirmed.

  In this embodiment, since “bidirectional real-time communication” is performed, the other party's screen state, the other party's screen operation, etc. are reflected in real time on his / her own terminal. The disturbance that is encrypted with the common key K1 is "should" be correctly decrypted by the common key K1, which is "should" also be used for decryption at point B " If it can be confirmed, it can be understood that “should” = “the possibility is very high”, so there is almost no problem in practical use.

Furthermore, FIG. 12 is also referred to as another embodiment of the fourth embodiment.
The image (1201) of the terminal at point A is encrypted (1202) with an initial setting common key K1 automatically set in advance, and the image is disturbed (1205). This disturbed image is transmitted to the point B (1206), and at the point B, the same initial setting common key K1 is decrypted (1210), and the screen (1213) from which the disturbance is released is obtained.

On the other hand, at the point B, the image (1221) of the terminal is encrypted (1222) with the same initial setting common key K1, and the image is disturbed (1225). When this disturbed image is transmitted to the point A (1226) and decrypted with the same initial setting common key K1 at the point A (1230), a screen (1233) from which the disturbance is canceled is obtained. As a result, at each of the points A and B, the information sent to each other can be confirmed without being particularly conscious of being encrypted.
With such a configuration, the encrypted screen information is transmitted from the self-terminal device, the encrypted image information is first displayed in the receiving terminal device, and then the correct information is displayed in the receiving terminal device. It can be confirmed by the key information that the encryption has been released. Then, it can be confirmed that the transmission data is really encrypted in the transmission terminal and that the other party of the reception terminal is a legitimate party.

The effects of the fourth embodiment as described above will be described in more detail including additional configurations. The following are the effects of the present embodiment, and are not necessarily included in the effects of the above (invention).
On the other hand, the correction which adds the additional structure in the following description to said each (invention) can be naturally predicted by those skilled in the art.

Conventionally, when sending ciphertext from A to B, for example, using the encryption key 123, conversely, when sending ciphertext from B to A, for example, using the encryption key 456 So, I couldn't confirm whether the other party was able to decipher or whether the encryption was applied.
On the other hand, when the ciphertext is sent from A to B, for example, the encryption key 123 is used, and when the ciphertext is sent from B to A, the decryption key is also set to the same encryption key 123. Thus, when the ciphertext arrives from B to A, it can be confirmed that the other party has been able to decipher, and that if the ciphertext other than the cipher key 123 is inserted, it cannot be deciphered. The confirmation here corresponds to the presence / absence of image scramble and the presence / absence of audio scramble, and means that it can be confirmed visually.

This embodiment is effective not only in one-to-one but also in group dialogue.
In this embodiment, the present invention can be applied by interactively exchanging any one of voice, face image, and material. The encrypted material is sent from A to B, and the confirmation can be made not by the material but by checking the scramble status of the other party's voice.
In addition, since the audio scramble becomes noise, it may be silenced and the audio scramble status may be displayed visually and with a character or graphic design. For example, when Mr. C inputs an incorrect encryption key in the group dialogue, the face image of only Mr. C is displayed scrambled, and when there is no face image transmission, the volume display of Mr. C is displayed individually. It is preferable to display the scrambled state of the voice in a character or graphic design.

  Generally, another means such as a telephone is desired for the delivery of the encryption key. However, as a countermeasure for cases where the encryption key is unknown, a common encryption key that is automatically initialized in advance is used. If an “encryption mode switching function” is included so that the transmission state can be selected without performing encryption at all, it is good for emergency response.

In addition, the above-mentioned Example of this invention was demonstrated as an example, Comprising: This invention is not limited to this.
Various alternatives, modifications, and changes can be made without departing from the spirit of the invention as set forth in the appended claims.

  Each embodiment of the present invention can be used in fields such as a video conference system and interactive education via a network, but is not limited thereto.

1 is a schematic configuration of a network system to which an information sharing technique can be applied according to an embodiment of the present invention. The block diagram of the structure of the terminal which can implement the information sharing technique which concerns on the Example of this invention. The block diagram which is easy to understand visually when the information sharing technique which concerns on the Example of this invention is applied to a two-way dialogue. The figure which expanded the various characteristics of each terminal device represented on the screen of the display 200A (200B) of each terminal device 900A (900B) of FIG. The example of the movement method of a window screen for making the person image on a screen transfer to an eye contact state. For example, the image information stored in the external storage device 700A of the self-terminal device, for example, displayed on the screen or the screen obtained via the Internet is surrounded by the image capture / transmission frame (3) in FIG. Outline diagram of how to do. The figure which shows the function of a virtual whiteboard. Functional explanatory diagram of the image organizer. Detailed explanation of how to use the image organizer. Outline diagram of transmission rate monitor. System block diagram of transmission rate monitor. An encryption + decryption scheme close to that known in the prior art. Explanatory drawing about the aspect in one Example of this invention which added some additional conditions to the conventional encryption scheme like FIG.

Explanation of symbols

1A terminal
1B terminal
2 network
2a node
2b node
100A PC body
100B PC body
110 I / O interface
120 Control unit
130 Storage means
140 Drawing part
141 VRAM
150 Data input / output section
160 Sound controller
200A display device
200B display device
300A speaker
300B speaker
400A pointing device
400B pointing device
500A keyboard
500B keyboard
600A microphone
600B microphone
700A external recording device
700B External recording device
800A camera
800B camera
1101 Image of the terminal at point A
1105 Disturbing images
1113 Disturbance canceled screen
1117 Screen that does not cancel disturbance
1121 Image of terminal at point B
1125 Disturbing the image
1133 Disturbance canceled screen
1137 Disturbance is not canceled
1201 Image of the terminal at point A
1205 Disturb image
1213 Disturbance canceled screen
1217 Screen that does not release disturbance
1221 Image of the terminal at point B
1225 Disturbing image
1233 Disturbance canceled screen
1237 Image of the terminal at point B
1241 Disturb image
1249 Disturbance is not released

Claims (19)

A first terminal capable of sending and receiving data to and from other terminals;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
Both the first terminal and the second terminal are
Storage means capable of storing at least one of image information, character information, and voice information;
At least one input means of image information, character information, and voice information;
Display / reproduction means for at least one of the display of the image information, the display of the character information, and the reproduction of the audio information;
Additional information storage means for storing the additional information input from the input means alone or in association with at least one of the image information, the character information, and the audio information in the storage means;
At least one of the image information, the character information, the voice information, and the additional information stored in the storage unit of any terminal from either the first terminal or the second terminal. An information exchanging means that can display and reproduce the information on the display / reproduction means of its own terminal,
Visual interactive real-time communication system. A first terminal capable of sending and receiving data to and from other terminals;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
Both the first terminal and the second terminal are
Storage means capable of storing at least one of image information, character information, and voice information;
At least one input means of image information, character information, and voice information;
Display / reproduction means for at least one of the display of the image information, the display of the character information, and the reproduction of the audio information;
By transmitting at least one of image information, character information, audio information, and additional information input from the input means to the other terminal, the display on the other terminal is displayed. -Mirroring means for displaying / reproducing on the reproducing means,
Visual interactive real-time communication system with   At least one of the image information, the character information, the voice information, and the additional information displayed and reproduced on one terminal of each terminal is a signal generated by the timing signal generating unit of the one terminal. 3. The visual interactive real-time communication system according to claim 1, wherein the visual interactive real-time communication system is stored in the storage means of the one terminal.   At least one of the time at which the timing signal is generated at one terminal of each terminal, the ID information of the other terminal at that time, and the attribute of the display image of the one terminal, the image information, the character information, The visual interactive real-time communication system according to claim 3, further comprising identification information adding means for storing in the storage means of said one terminal as identification information associated with at least one of audio information and said additional information. A first terminal capable of sending and receiving data to and from other terminals;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
Both the first terminal and the second terminal are
At least one input means for image information, text information, and voice information;
Display / reproduction means for at least one of display of image information, display of character information, reproduction of audio information;
With
At least one of the terminals is
Sending data preparation means capable of compressing sending data to the other terminal of each terminal;
Other terminal display state monitoring means for monitoring the state of data to be displayed on the other terminal of each terminal,
Comprising
Visual interactive real-time communication system. The other terminal display state monitoring means is
Sending data transfer means for passing the sending data to the other terminal prepared by the sending data preparation means to the display / playback means of the one terminal,
The visual interactive real-time communication system according to claim 5, comprising: The other terminal comprises a reception state return means for returning the reception state of the data sent from the one terminal to the one terminal from the other terminal,
The other terminal display state monitoring means is
A reception state data transfer unit that passes the reception state data of the other terminal returned from the other terminal to the one terminal by the reception state return unit to the display / playback unit of the one terminal;
The visual interactive real-time communication system according to claim 5, comprising: A first terminal capable of sending and receiving data to and from other terminals;
A second terminal capable of transmitting and receiving data to and from the first terminal;
A visual interactive real-time communication system comprising:
The first terminal and the second terminal both have display / reproduction means for displaying at least one of display of image information, display of character information, and reproduction of audio information;
Key information input means;
Means for disturbing data to be sent to another terminal based on the key information input from the input means;
Means for decrypting the disturbance data received by the self-terminal based on the inputted key information and giving it to the display / playback means of the self-terminal;
Comprising
Visual interactive real-time communication system. The visual interactive real-time communication system according to claim 8,
Display that gives information indicating whether or not the disturbance state of the disturbance data received by the own terminal has been released to the display / reproduction means of the own terminal when it is determined that the input key information is not correct key information A visual interactive real-time communication system further including a reproduction state confirmation unit.   It comprises three or more terminal groups, an arbitrary one terminal in the terminal group is a first terminal, and the remaining terminal group has the second terminal according to any of the above claims A visual interactive real-time communication system comprising a configuration.   The at least part of data exchange between the first terminal and the second terminal is performed via an intermediary terminal or under the control of the intermediary terminal. Visual interactive real-time communication system.   12. Both visuals according to claim 11, wherein the intermediary terminal comprises at least part of the means of the first terminal or the second terminal according to any of the above claims. Real-time communication system. It is a terminal that can send and receive data with other terminals for visual two-way real-time communication,
Storage means capable of storing at least one of image information, character information, and audio information;
At least one input means of image information, character information, and audio information;
Display / reproduction means for at least one of the display of image information, character information, and reproduction of audio information;
Additional information storage means for storing the additional information input from the input means alone or in association with at least one of the image information, the character information, and the audio information in the storage means;
Information exchange means for enabling at least one of image information, character information, audio information, and additional information transmitted from another terminal to be displayed / reproduced on the display / reproduction means of the terminal. Terminal. It is a terminal that can send and receive data with other terminals for visual two-way real-time communication,
At least one of image information, character information, and voice information input means; and display / reproduction means for at least one of image information, character information display, and voice information reproduction;
Sending data preparation means capable of compressing sending data to the other terminal;
The other terminal display state monitoring means for monitoring the state of data to be displayed on the other terminal;
A terminal equipped with. It is a terminal that can send and receive data with other terminals for visual two-way real-time communication,
Display / reproduction means for at least one of image information, display of character information, and reproduction of audio information;
Key information input means;
Means for disturbing data to be sent to another terminal based on the key information input from the input means;
Means for decrypting the disturbance data received by the self-terminal based on the inputted key information and giving it to the display / playback means of the self-terminal;
A terminal equipped with. At least on the computer,
As a terminal that can send and receive data with other terminals for visual two-way real-time communication,
A storage function capable of storing at least one of image information, text information, and audio information;
An input function of at least one of image information, text information, and audio information;
A display / playback function for at least one of the image information, the display of character information, and the playback of audio information;
An additional information storage function for storing additional information input using the input function alone or in association with at least one of the image information, character information, and voice information;
It is possible to display / reproduce at least one of the image information, the character information, the voice information, and the additional information stored in the storage unit of another terminal on the display / reproduction unit of the terminal. Information exchange function,
Visual interactive real-time communication program that realizes At least on the computer,
As a terminal that can send and receive data with other terminals for visual two-way real-time communication,
A display / reproduction function for at least one of image information, character information, and sound information input function and at least one of image information, character information display, and sound information reproduction;
A send data preparation function capable of compressing send data to the other terminal;
The other terminal display state monitor function for monitoring the state of data to be displayed on the other terminal;
Visual interactive real-time communication program that realizes At least on the computer,
As a terminal that can send and receive data with other terminals for visual two-way real-time communication,
A display / reproduction function for at least one of display of image information, display of character information, and reproduction of audio information;
Key information input function,
A function of disturbing data to be sent to another terminal based on the key information input using the input function;
A function of decrypting the disturbance data received by the own terminal based on the inputted key information and giving it to the display / playback means of the own terminal;
Visual interactive real-time communication program that realizes   An information storage medium storing the program according to claim 16.

Images