JPH1040068A - Method for recording conference information and device therefore and device for reproducing convention information - Google Patents

Abstract

PROBLEM TO BE SOLVED: To record information by which the content of a conference can be exactly reproduced by a small information amounts by recording picture information including at least an object to be indicated by relating it with the spoken part of an indicating word of voice information when an indicating operation corresponding to the indicating word is present. SOLUTION: An indicating word detection reporting part 62 monitors the output of an indicating word detecting part 61, and when the detected output of an indicating word is present, the pertinent indicating word detection reporting part 62 reports the detection of the indicating word to an object to be detected detecting part 63, picture fetching device 64, and timer 65. Then, a recording controller part 60 detects the indicating word from the voice input information of a participant, and when the indicating word is detected, the pertinent recording controller part 60 specifies the indicating operation of the participant corresponding to the indicating word and the object to be indicated by the indicating operation, and when the indicating operation corresponding to the indicating word is detected, the pertinent recording controller part 60 stores the voice information and picture information when the indicating word is spoken with the indicating operation of the participant in a file by making the spoken time of the indicating word correspond to them as link information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for recording and reproducing voice information and image information in a conference and instruction operation information on a screen of a conference participant. The present invention relates to a method and an apparatus capable of accurately reproducing the contents of a main part.

[0002]

2. Description of the Related Art In a conference, much information is exchanged by conversation. Few of the meeting information is documented as text, such as whiteboards and minutes.
There is a problem that important remarks are not recorded and cannot be remembered accurately.

[0003] From this point, the most primitive method of recording meeting information is a method of recording all information. One example of this kind of meeting recording method is disclosed in Japanese Patent Laid-Open No. 6-343146.
No. in the official gazette.

In the invention disclosed in Japanese Patent Application Laid-Open No. Hei 6-343146, a conference is recorded by multimedia information including audio information from a microphone, image information from a video camera, characters of a conference participant, graphic input information, and the like. Thus, the conference information is recorded without fail. Then, the input times of various input information of the conference participants are stored as an index in association with the multimedia information, and the multimedia information can be easily accessed by the index during reproduction.

According to the technique disclosed in Japanese Patent Application Laid-Open No. Hei 6-343146, it is possible to solve the problems that the remarks of the conference are not recorded and cannot be easily accessed. However, multimedia recording as disclosed in this publication requires an extremely large storage capacity for recording and preservation. This enormous storage capacity is problematic, especially when trying to preserve that record over a long period of time. For image compression,
Although there are general-purpose compression techniques such as PEG and MPEG, the amount of information cannot be sufficiently reduced. That is, it is an important issue for a conference record to accurately reproduce the information of the conference with a minimum amount of information.

[0006] Instead of recording all the information of the conference,
From the viewpoint of recording only the information necessary for accurately reproducing the contents, fairly accurate information can be obtained only from the audio information. Conventionally, a tape recorder is generally used for recording the audio-only conference information.

[0007] However, as a feature of the speech of the meeting, there are many cases where the speech is made while referring to visual information such as meeting material data. For this reason, when only the voice information is recorded, there is a problem that it is not possible to know what the user is talking about, and it is not possible to accurately understand the utterance content. Even if the OHP and the handouts used in the conference can be referred to when the conference information is reproduced, there is no means for knowing the corresponding relationship with which part.

Therefore, it has been conventionally considered to take in image information for complementing audio information for more accurate reproduction.

In the audio image recording and displaying apparatus disclosed in Japanese Patent Laid-Open No. 6-276478, a still image is recorded by a camera as supplementary information while recording audio. In this publication, recorded voice and image information are held in association with each other according to the image recording time (time code).

When fetching image information for complementing audio information, it is better to fetch only necessary images instead of fetching all images. It is important to select the target to be used.

Japanese Patent Application Laid-Open No. HEI 6-165509 discloses a method for automatically judging at which point in time an image is significant with respect to the timing of capturing an image. There has been proposed a method of giving priority to an image by using the attribute of (1).

It is conceivable to use voice recognition as a timing for capturing an image. That is, for example, the invention of Japanese Patent Application Laid-Open No. 6-68168 is not a technique relating to recording and reproduction of conference information, but an audio keyword is registered in advance in a video information database, and the audio keyword is detected by voice recognition. A device for searching / reproducing corresponding video information is described. In view of this, it is conceivable to apply the technology of this publication to a conference record and determine the timing of capturing an image that complements the conference voice by using a voice keyword based on voice recognition.

As a technique for selecting an object to be photographed, a technique for detecting what a participant of a conference pays attention to becomes important. Japanese Patent Laying-Open No. 7-200632 discloses a technique for identifying information of interest of a participant by using information of a line of sight. Also, as described in Japanese Patent Laid-Open No.
In JP-343146-A, stroke information from an input device such as a pen is extracted and instruction information of a conference participant is recorded. In this case, according to the invention of this publication, whether or not the movement of the pen is meaningful instruction information is determined by the user intentionally instructing the pen or by making the pen itself have the ability to recognize the mode. I assume.

When a recording target is selected, since there are a plurality of participants in a conference, it is important to select who should pay attention information. JP-A-6-13904
Japanese Patent Laid-Open Publication No. 2 (1999) -1995 discloses a technique for selecting an appropriate pointing device from competition between a plurality of pointing devices. That is, in the invention of this publication, the movement of the speaker's pointing device is regarded as significant, assuming an environment in which a plurality of participants can operate the pointing device in the face-to-face conference. In this case, the speaker
A participant whose sound pressure is detected by the sound pressure sensing device at a certain level or higher is identified, and by highlighting a cursor corresponding to the pointing device of the participant paired with the sound pressure sensing device. In this case, the selected pointing device indicates something.

[0015]

As described above, it is preferable that the contents of a conference can be accurately reproduced with a minimum amount of information in a conference record.
It is desirable to record all the audio during the conference and also record image information that complements the audio. In this case, it is important to capture only necessary information as image information for complementing audio information. Therefore, it is particularly important to capture the complementary image information and specify a capture target. is there.

As described above, there have been various proposals for techniques for this purpose. However, the techniques disclosed in the above publications have respective problems as described below.

The technique disclosed in Japanese Patent Application Laid-Open No. 6-276478 is
This solves the problem that audio information and image information cannot be correlated, but it is left to the user to determine which video is important for complementing audio information. further,
There is a problem in that the user has to input where the point is in the video by using a text or a handwritten image, which is troublesome.

In the technique disclosed in Japanese Patent Application Laid-Open No. 6-165509, the degree of importance is automatically determined from a button operation at the time of shooting in order to solve the problem that the photographer must determine which image is important. By solving the problem. However, there remains the problem of ambiguity in remarks involving references to visual information. In particular, "this" or "the part"
In the case of remarks with directives such as, etc., it is not possible to know only by the voice information what the directive refers to,
The problem becomes prominent.

To solve this problem, if an attempt is made to supplement only with image information, zooming,
It is difficult to determine an appropriate degree of importance only with button attributes such as continuous shooting. Therefore, in the case of Japanese Patent Application Laid-Open No. 6-165509, similarly to the technology of the above-mentioned Japanese Patent Application Laid-Open No. 6-276478, the determination of the importance of an image is left to the photographer.

In the case where the technique using a speech keyword based on speech recognition disclosed in Japanese Patent Application Laid-Open No. 6-68168 is used as a means for taking in important image information for complementing a conference audio, the technique disclosed in Japanese Patent Application Laid-Open No. The problem that the photographer has to specify the importance in the techniques of Japanese Patent Application Laid-Open No. 1650009 and Japanese Patent Application Laid-Open No. 6-276478 can be improved. However, in this case, there are the following problems.

One is that it is difficult to determine in advance keywords that are considered important. Further, even if it is determined, there is a problem that the utterance of the keyword does not always match the importance of the video, and unnecessary image information is unnecessarily taken in.

Furthermore, when an acoustic keyword is used as a trigger to capture image information, as compared to attaching an acoustic keyword to an already recorded database, it is recognized that an important keyword has been uttered after the utterance. Therefore, there is a problem that a time error occurs due to a delay due to the recognition process, and it is difficult to accurately associate the image with an appropriate image.

With respect to the problem of what should be recorded as image information, the problem is to specify what the participant is paying attention to. However, Japanese Patent Application Laid-Open No. 7-200632 discloses a gaze detection device. Automatically extracts what the user is paying attention to. However, the line-of-sight detection device is too expensive as a means for capturing complementary information of audio information.

On the other hand, Japanese Patent Application Laid-Open No. 6-343146 uses a relatively inexpensive pen-shaped pointing device. However, according to the technology disclosed in this publication, the user needs to intentionally indicate the instruction mode, which imposes a heavy burden on the user. In addition, stroke information (a unit of continuous movement at a designated position having a predetermined speed is referred to as a stroke) information is extracted as a unit of recognition. Stroke is not appropriate. Rather, a technique for extracting a meaningful instruction operation unit from a continuous series of strokes is an issue.

Further, as described above, in a conference in which a plurality of participants exist, it is specified which one of the plurality of conference participants should record the attention information as meaningful information. It is also important to solve this problem, as described above, as described in JP-A-6-13904.
There is a technique disclosed in Japanese Patent Application Laid-Open No. 2 (1994) -209, but in this case, whether the movement of the pointing device is significant is determined only by the sound pressure of the speaker.

However, considering the purpose of complementing the voice information, there is a problem that the continuous recording of the movement of the pointing device of the speaker during the speech leads to an increase in useless information amount. In an actual face-to-face meeting, whether the movement of the cursor is meaningful can be inferred from information such as the facial expression of the speaker,
It is difficult to judge from the reproduced image information alone, and halfway excess information may confuse the reference person.

Further, in an actual conference, not all participants necessarily speak with a pointing device, but the moderator or a person near the blackboard points the attention information to the statement. In some cases, the technique disclosed in Japanese Patent Application Laid-Open No. 6-139042 cannot cope with such a situation.

In view of the above problems, according to the present invention, in a conference information recording method and a recording / reproducing apparatus capable of accurately reproducing conference information with a minimum amount of information, audio information is recorded together with audio information. It is an object of the present invention to provide a method and apparatus capable of automatically recording attention information of an appropriate participant at an appropriate timing as much as possible with minimum human effort in order to complement the information.

Another object of the present invention is to provide a conference information reproducing apparatus capable of recording as described above and effectively reproducing the recorded information.

[0030]

In order to solve the above-mentioned problems, a meeting information recording method according to the present invention as defined in claim 1 is provided.
While recording audio information when a plurality of conference participants hold a conference while watching the common display screen content, and image information including the display screen content at an appropriate timing to supplement this audio information, In a method of recording together with information of an instruction operation of an arbitrary conference participant on the display screen as necessary, when a conference participant speaks an instruction word, any conference participation corresponding to the instruction word is performed. Detecting whether there is an instruction operation on the display screen of the user, and when there is an instruction operation corresponding to the instruction word, at least, image information including the instruction target, the instruction information of the audio information It is characterized in that it is recorded in association with the utterance part.

According to a second aspect of the present invention, in the conference information recording method according to the first aspect, when an instruction operation corresponding to the instruction word is performed, in addition to the image information including the instruction target, The information of the operation trajectory of the instruction operation corresponding to the instruction word is recorded in association with the voice information of the utterance part of the instruction word.

According to a third aspect of the present invention, there is provided a conference information recording device, wherein a display device for displaying image information contributing to the conference, and a conference participant designates an arbitrary position on a screen of the display device. Instruction operation means for performing the operation, position recognition means for recognizing a position on the screen designated by the instruction operation means, audio information input means for receiving audio information input during the conference, and the audio information input means From the voice information from, a command word detecting means for detecting voice information of the command word, a command operation detecting means for detecting a command operation by the command operation means corresponding to the command word detected by the command word detecting means, A voice during the meeting from the voice information input means is recorded, and when the pointing operation detecting means detects the pointing operation by the pointing operating means corresponding to the instruction word, at least the The image information including the instruction target of the detected instruction operation, characterized by comprising a recording means for recording in association with the voice information of the detected utterance portion of the instruction word in the instruction word detecting means.

According to a fourth aspect of the present invention, there is provided the conference information recording apparatus according to the third aspect, wherein the recording means includes, in addition to the image information including the pointing target detected by the pointing target detecting means, Recording information of an instruction operation trajectory of the instruction operation means using position information from the position recognition means in association with voice information of an utterance portion of the instruction word detected by the instruction word detection means. And

According to a fifth aspect of the present invention, in the conference information recording apparatus according to the first aspect, a plurality of instruction operation means are provided corresponding to a plurality of conference participants. The detecting means includes: a participant attribute determining means for determining the priority of each conference participant based on the attribute of the conference participant; An instruction operation significance determination unit that determines whether the instruction operation is significant based on whether a moving amount condition is satisfied is compared with a variation amount of an indicated coordinate position on a display screen by the plurality of instruction operation units. And an instruction operation means superiority determination means for judging superiority of the instruction operation means, wherein the determination result of the participant attribute determination means, the instruction operation means superiority determination means, and the instruction operation Based on the determination result that the uniqueness determination unit, and detects the instruction operation by the instruction operation means corresponding to the instruction word.

According to a sixth aspect of the present invention, in the conference information recording apparatus according to the fourth aspect, the instruction operation detecting means is configured to determine, for each instruction operation means, coordinate data of an instruction operation position for a predetermined period of time. Instruction history recording means for recording the
The instruction operation detecting means detects a start time of the instruction operation by correcting a time delay of a time point at which the instruction word is detected with respect to an utterance time of the instruction word using the coordinate data of the instruction history recording means. It is characterized by.

In the conference information recording apparatus according to a seventh aspect of the present invention, the instruction operation detecting means determines the end of the instruction operation by the instruction operation means based on the amount of change in the indicated coordinate position by the instruction operation means. End position determining means, end time determining means for determining the end of the instruction operation by the instruction operation means by monitoring the elapsed time from the start time of the instruction operation, and detecting the next instruction word And a notifying means for notifying, and detecting an end point of the instruction operation by the instruction operating means based on the end position determining means, the end time determining means, and the notifying means.

In the conference information recording / reproducing apparatus according to the present invention, a display device for displaying image information contributing to the conference and a conference participant point to an arbitrary position on a screen of the display device. Instruction operation means, a position recognizing means for recognizing a position on the screen instructed by the instruction operation means, a voice information input means for receiving voice information input of a speech of the conference participant, and the voice information Instruction word detection means for detecting audio information of an instruction word from audio information from an input means, and instruction operation detection means for detecting an instruction operation by the instruction operation means corresponding to the instruction word detected by the instruction word detection means And, while recording the audio during the meeting from the audio information input means, when the instruction operation detecting means detects an instruction operation by the instruction operation means corresponding to the instruction word, at least, Recording means for recording image information including the instruction target in association with audio information of an utterance portion of the instruction word detected by the instruction word detection means; and audio reproduction means for reproducing the recorded audio information And image information display means for displaying image information corresponding to the instruction word at the time of utterance of the instruction word when the audio information is reproduced.

According to a ninth aspect of the present invention, there is provided a conference information recording and reproducing apparatus according to the eighth aspect, wherein:
The recording means stores, in addition to the image information including the pointing target detected by the pointing target detecting means, information of a pointing operation trajectory of the pointing operating means using position information from the position recognition means, the pointing word. It is recorded in association with the voice information of the utterance portion of the instruction word detected by the detection means,
According to the information on the instruction operation trajectory, during the sequential reproduction of the voice information, the instruction operation trajectory of the instruction operation means is displayed by superimposing on the image information corresponding to the instruction word displayed at the time of utterance of the instruction word. And specifying an instruction target corresponding to the instruction word.

A conference information reproducing apparatus according to a tenth aspect of the present invention
When the conference participant uttered the instruction word, the instruction operation on the display screen of any conference participant corresponding to the instruction word was performed as the audio information during the conference and the information supplementing the audio information. In this case, the image information including the instruction target and the information of the trajectory of the instruction operation corresponding to the instruction word are recorded in association with the voice information, using the time at which the instruction word was uttered as link information. A conference information reproducing device, wherein a display device for displaying the image information and the operation trajectory of the instruction operation, and a change in a reproduction position of the audio information on a screen of the display device, the change corresponding to a current reproduction position. First display control means for displaying a screen display to be displayed, and referring to the link information, the presence of an utterance of an instruction word at a playback position future than the current playback position and image information related thereto A display to be foreseen, characterized in that it comprises a second display control means to perform in connection with the change display of the playback position.

The conference information reproducing apparatus according to the eleventh aspect of the present invention
When the conference participant uttered the instruction word, the instruction operation on the display screen of any conference participant corresponding to the instruction word was performed as the audio information during the conference and the information supplementing the audio information. In this case, the image information including the instruction target and the information of the trajectory of the instruction operation corresponding to the instruction word are recorded in association with the voice information, using the time at which the instruction word was uttered as link information. A conference information playback device, comprising: a display device for displaying the image information and the operation trajectory of the instruction operation; and a start time of an instruction operation corresponding to the instruction word determined based on the link information, Means for calculating a point in time before a predetermined period of time, and from the point in time before the predetermined period of time, displaying corresponding image information on the screen of the display device, and displaying the display for the predetermined period of time. In the stage screen, and means for displaying visual attention information on the starting point of the instruction operation, and characterized in that a foreseeable for instructions object.

According to a twelfth aspect of the present invention, there is provided a conference information reproducing apparatus,
When the conference participant uttered the instruction word, the instruction operation on the display screen of any conference participant corresponding to the instruction word was performed as the audio information during the conference and the information supplementing the audio information. In this case, the image information including the instruction target and the information of the trajectory of the instruction operation corresponding to the instruction word are recorded in association with the voice information, using the time at which the instruction word was uttered as link information. A conference information reproducing device, a display device for displaying the image information and the operation trajectory of the instruction operation, an audio reproducing unit that reproduces the recorded audio information, and, when reproducing the audio information, Image display control means for displaying image information corresponding to the instruction word on the screen of the display device at the time of utterance of the instruction word, and the image information is displayed by the image display control means at the time of utterance of the instruction word Instruction operation reproducing means for visually reproducing the instruction operation corresponding to the information of the trajectory of the instruction operation, superimposed on the image information corresponding to the instruction word, and determining from the end time of the reproduced instruction operation. Means for specifying a time after a lapse of a predetermined time, means for displaying after-image information of a trajectory of an instruction operation for the predetermined time, and displaying visual attention information at an end point of the instruction operation. And a means for erasing the display of the attention information after the lapse of the predetermined time.

[0042]

The conference information recording apparatus according to the first aspect of the present invention recognizes and detects utterances of instruction words such as "this" and "that" from the voice input information of the utterances of the conference participants. When detecting the utterance of the instruction word, it detects whether or not there is an instruction operation on the display screen of any conference participant corresponding to the instruction word. The reason why the instruction participant is not a speaker only but an arbitrary participant in the conference is that it is necessary to consider a case where the person performing the instruction operation is not a speaker but a moderator.

When there is an instruction operation corresponding to the instruction word, image information including the instruction target of the instruction operation is recorded in association with the utterance part of the instruction word. For example, if the image information is image information obtained by capturing the contents of a meeting with a video camera, an instruction operation may be included in the image information.

Thus, according to the first aspect of the present invention, when the descriptive word is uttered by a conference participant, the image information including the target of the descriptive word is automatically converted to image information that complements the voice information. Will be recorded. In this case, when a descriptive word indicating something that is not displayed on the display screen is uttered by the meeting participant, there is no instruction operation corresponding to the descriptive word, so that unnecessary image information is not recorded. . Therefore, it is possible to record information that can reproduce the contents of the meeting almost accurately with a minimum amount of information.

In the case where the image information to be recorded is the display content displayed on the display screen, the instruction operation may not be included in the image information. Information of the operation trajectory of the operation is recorded together with the image information in association with the utterance part of the instruction word. Therefore, by reproducing the instruction operation by using the trajectory information of the instruction operation at the time of reproduction, it becomes possible to specify the target of the instruction word in the image information at the time of reproduction.

When the information of the operation trajectory of the instruction operation corresponding to the instruction word is recorded as in claim 2, the image information includes a minimum image including the instruction target, For example, information of one still image may be recorded. This is because the instruction operation can be reproduced on the still image display image using the information of the operation trajectory of the instruction operation, and the instruction target is specified.

According to the conference information recording apparatus of the third aspect, the same operation and effect as those of the conference information recording method of the first aspect can be obtained. Similarly, in the conference information recording apparatus according to the fourth aspect of the invention, the same operation and effect as those of the conference information recording method according to the second aspect of the invention can be obtained.

In the conference information recording apparatus according to the fifth aspect of the present invention, each of a plurality of conference participants has an instruction operation means, and when an instruction word is uttered, who corresponds to the instruction word. It is possible to detect a conflict as to whether an instruction operation was performed.

In the conference information recording apparatus according to the sixth aspect of the present invention, the coordinate data of the designated operation position of the designated operation means up to a certain time before the current time is always recorded.
Even if the start point of the instruction operation is passed due to the delay until the detection of the instruction word, the coordinate data of the instruction operation position is obtained from the start point of the instruction operation as far back as the delay. Can be. Therefore, it is possible to accurately reproduce the instruction operation.

In the meeting information recording apparatus according to the seventh aspect of the present invention, the end of the pointing operation is determined mainly based on the magnitude of the variation of the position coordinates of the pointing device. I have.

That is, the elapsed time from the start of the instruction operation is monitored, and the end of the instruction operation is determined based on the elapsed time. For example, for the elapsed time,
The minimum time and the maximum time for recording the instruction operation are determined, and it is guaranteed that the instruction operation is recorded for the minimum time or more.
When the maximum time has elapsed, it is determined that the instruction operation has been completed. Also, when the notifying means for notifying the detection of the next directive is detected to have detected the next directive,
It is determined that the instructing operation being processed is ended. Based on the combination of the end position determination, the end time determination, and the notification from the notification unit, appropriate end conditions of the instruction operation are determined while ensuring the reproduction of the recognizable instruction operation trajectory.

In the conference information recording / reproducing apparatus according to the eighth aspect of the present invention, the audio information during the conference is sequentially recorded, and the instruction information is recorded in the same manner as in the case of recording by the conference information recording apparatus according to the second aspect. When the presence of the instruction operation corresponding to the instruction word is detected when the utterance of the word is detected, the image information including the instruction target of the instruction operation is associated with the utterance part of the instruction word in the voice information. Recorded.

In the reproduction, the audio information during the conference is sequentially reproduced by the audio reproducing means, and when the instruction word is uttered and the corresponding image information is recorded, the image information is recorded. Is automatically displayed by the image information display control means. If this image information is imaging information from a camera, the instruction operation at that time is included in the image information, and the instruction target of the instruction word on the display screen is specified, which is useful for accurately grasping the contents of the conference. .

According to a ninth aspect of the present invention, in the conference information recording / reproducing apparatus according to the eighth aspect, in addition to the image information including the pointing target detected by the pointing target detecting means, the position information from the position recognizing means is provided. The information of the used instruction operation trajectory of the instruction operation means is recorded in association with the voice information of the utterance part of the instruction word detected by the instruction word detection means. Therefore, at the time of reproduction, when the utterance portion of the instruction word comes, the related image information is displayed, and the instruction operation trajectory of the instruction operation means is automatically displayed on the image information, and the trajectory corresponding to the instruction word is displayed. The instruction target is specified.

Since the information of the instruction operation trajectory is recorded, even if the image information is, for example, one still image, the instruction operation based on the operation trajectory information is displayed on the still image. . Therefore, it is not necessary to capture and record a plurality of frames including the start and end of the instruction operation with a camera or the like as image information.

In the conference information reproducing apparatus according to the tenth aspect, when audio information is sequentially reproduced, a change in the reproduction position is displayed on the screen with reference to, for example, the current reproduction position. . At this time, a reproduction position future than the current reproduction position is also displayed, and the utterance of the instruction word existing at the time of the future reproduction position, the corresponding image information, and the preview information including the instruction operation are displayed as images. Therefore, at the time of reproduction, it is possible to foresee the point at which the instruction word accompanied by the instruction operation is uttered on the display screen while reproducing the audio information.

In the conference information reproducing apparatus according to the eleventh aspect of the present invention, when the voice information is sequentially reproduced, the instruction target corresponding to the instruction operation corresponding to the instruction word is displayed on the screen. Can be recognized to some extent by image display before is reproduced.

That is, for example, a time that is a predetermined time before the instruction operation start time of the instruction operation means is calculated,
When the relevant time is detected, the image information corresponding to the instruction word is displayed at the relevant time, and the visual attention information is displayed at the start point of the instruction operation for a predetermined fixed time. Thus, the user can foresee the presence of the instruction word, and the user can foresee which part of the display screen the indication target of the predicted instruction word is, for example. .

In the conference information recording / reproducing apparatus according to the twelfth aspect, when a time after a lapse of a predetermined time from the instruction operation end time of the instruction operation means is detected, the conference information recording / reproducing apparatus performs the operation for a predetermined time. Displaying the afterimage information of the trajectory of the instruction operation, performing visual attention information display at the end point of the instruction operation,
After a certain time has elapsed, the display of the attention information is deleted.

Therefore, even if the pointing operation is completed in the actual time, the display of the afterimage information of the trajectory of the pointing operation and the visual attention information display of the ending point of the pointing operation are performed for a predetermined time. It is not lost which position has been instructed.

[0061]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a conference information recording or reproducing apparatus according to the present invention will be described below with reference to the drawings.

FIG. 2 is a conceptual diagram of a face-to-face meeting in an electronic meeting room to which an embodiment of a meeting information recording / reproducing apparatus according to the present invention is applied.

In FIG. 2, 1 is a personal computer, and 2 is a large screen display device. The personal computer 1 reads out image information of conference materials such as data and documents used in the conference from a storage unit storing the conference materials, and provides the large-screen display device 2 with the information. The image of the conference material provided from the personal computer 1 is projected and displayed on the screen 2D of the large screen display device 2.

Conference participants 10, 11, 12, 13, 14
Performs a presentation and a discussion through image information projected on the screen 2D of the large screen display device 2. In the example of FIG. 2, the conference participant 10 is a moderator. Other conference participants 11 to 14 are general participants. In this embodiment, the number of conference participants is five, but the number of participants is arbitrary. The presence of a moderator is also optional.

Reference numerals 20, 21, 22, 23, and 24 denote pointing devices as examples of the instruction operation means, which are assigned to the participants 10, 11, 12, 13, and 14, respectively. The participants 10 to 14 display the graphs and documents displayed on the screen 2D of the large screen display device 2 with the pointing devices 20, 21, 22, 23,
The discussion will be conducted while pointing at 24.

Pointing devices 20, 21, 22
2, 23 and 24 are used to indicate arbitrary positions on the screen 2D of the large screen display device 2.
The pointing device 20 is installed near the large-screen display device 2 and is used as a moderator's pointing device. This is mainly used by the moderator 10 or may be used by other participants coming near the large screen display device 2. The pointing devices 21 to 24 are for general participants. Each of the conference participants 21 to 24 can point to an arbitrary position on the display by remote control from the seat of the conference participant.

That is, the pointing devices 20 to 24 of this example are pen-shaped devices that can be remotely operated.
An infrared light emitting unit is provided, and an arbitrary position on the screen 2D of the large screen display device 2 can be indicated by the infrared beam. The pointing devices 20 to 24 are configured to be able to detect the amount of movement of the pointing device itself, and input the data of the amount of movement to the personal computer 1 together with the respective ID data (identification data). .

When the participant picks up and operates each pointing device, the personal computer 1 detects a significant amount of exercise of the pointing device that has been picked up and moved, and detects that the pointing device has been picked up. . Since the identification data ID is given to the data from the pointing device, the personal computer 1 detects which pointing device has been taken up by the participant.

As described above, one pointing device is assigned to each participant in advance, so that the personal computer 1 detects who is picking up and operating the pointing device. Will be.

It should be noted that the pointing device 20 is placed in the vicinity of the screen 2D of the large screen display device 2 in advance of the personal computer 1.
The operator of the pointing device 20 is usually recognized as a moderator.

The screen 2 of the large screen display device 2
On D, the operation trajectory of the pointing device is drawn as an infrared spot trajectory. The large-screen display device 2 includes a sensor and a camera for detecting the position of an infrared spot on the screen 2D as position coordinates on the screen 2D, and transmits position coordinate data on the screen 2D to the personal computer 1. It is configured to supply. Personal computer 1
Converts the position coordinates on the screen 2D into position coordinates on its own display screen, and uses it as pointing operation position data of the pointing device.

In this embodiment, a description will be given of an instruction operation detection for recognizing a position on the screen 2D of the two-dimensional large-screen display device 2. However, the present invention is not limited to this. May be a combination of a pointing device that recognizes the image and a directional camera linked to the direction of the pointing device.

Reference numerals 30, 31, 32, 33, and 34 denote microphones for collecting speeches of participants in the conference and other sounds. This microphone 30-34
Are associated with each participant and are assigned one by one. The microphone 30 is a large screen display 2
Is a voice input device for the moderator. The microphones 31 to 34 are voice input devices for general conference participants.

The correspondence between the microphones 30 to 34 and the conference participants 10 to 14 is registered and recorded in the personal computer 1 in advance. Thereby, the correspondence between the pointing device and each conference participant is also defined.

FIG. 1 shows a system configuration diagram of a recording system of the conference information recording / reproducing apparatus according to this embodiment. This is mainly a configuration of a functional block diagram in which processing functions of respective units are blocked.

In FIG. 1, reference numeral 51 denotes a display image providing unit.
This is a function of the personal computer 1 in this embodiment, as described above. From the display image providing unit 51, image data Vd projected and displayed on the screen 2D of the large screen display device 2 is output, for example, in frame units.

Reference numeral 52 denotes a pointing device group.
As described above, a plurality of pointing devices 20 to 24 are respectively assigned to a plurality of participants. When each of the pointing devices 20 to 24 is operated by an operator, data Pm0 to Pm4 of the relative movement amount is output and supplied to the personal computer 1.

The pointing devices 20 to 24
Generates an infrared beam, scans the screen 2D with the beam spot as described above, and performs an instruction operation on a display character, a display figure, a symbol, and the like on the screen 2D.

As described above, the designated coordinate generating unit 53
The function of the large screen display device 2 is to detect the position of an infrared beam spot on the screen 2D with a camera or a sensor, and to supply the designated coordinate output data Co to the personal computer 1. In this case, when a plurality of beam spots exist on the screen, coordinate output data for each of the plurality of beam spots is supplied to the personal computer 1.

Reference numeral 54 denotes a voice input device which comprises a plurality of microphones 30 to 34 assigned to a plurality of participants, respectively, and supplies the output voice signals to the personal computer 1 as described above. Things.

The recording control unit 60 detects the instruction word from the voice input information of the participant, and when the instruction word is detected,
The instruction operation of the participant corresponding thereto and the instruction target by the instruction operation are specified, and when the instruction operation corresponding to the instruction word is detected, the voice information and the image information at the time of uttering the instruction word are transmitted to the participant. Along with the instruction operation, in this example, the apparatus portion performs processing for storing the utterance time of the instruction word as link information and storing it in a file.

In this embodiment, the recording control unit 60
It is a functional part realized by the personal computer 1. Each unit of the recording control unit 60 in FIG. 1 shows its processing control function as a block, and each block is actually realized by software.

In this embodiment, the recording control unit 6
0 is a descriptive word detection unit 61, a descriptive word detection notification unit 62,
Pointing object detecting section 63, image capturing section 64, timer 65
, A file storage unit 66, a pointing device instruction history recording unit 67, an A / D conversion unit 68, and an audio signal mixing unit 69.

The A / D conversion section 68 includes the voice input device section 54
The analog audio signals from the respective microphones 30 to 34 are converted into digital audio signals, and the converted digital audio signals are supplied to the instruction word detection unit 61 and to the audio signal mixing unit 69. The audio signal mixing section 69 mixes digital audio data obtained by converting audio signals from the microphones 30 to 34 into digital signals, and supplies the digital audio data to the file storage section 66.

The instruction word detector 61 analyzes the voice data of each participant input from the A / D converter 68 and recognizes the utterance of the instruction word for each participant. A detailed example of the instruction word detection processing operation will be described later. The instruction word detection unit 61
When any participant utters the instruction word, the detection output is output to the instruction word detection notification unit 62.

The descriptive word detection notifying unit 62 is
1 is monitored, and when the detection output of the instruction word appears, the detection of the instruction word is notified to the instruction target detection unit 63, the image capturing device 64, and the timer 65.

The pointing target detection unit 63 includes the pointing coordinate generation unit 5
Pointing coordinate data of the pointing device from 3;
It receives data from an instruction history recording unit 67, which will be described later, as input data, and detects an instruction target of the instruction word when the instruction word is issued. Specifically, the pointing target detection unit 63 detects a pointing device that is being operated at the time of detection of a descriptive word, and detects a trajectory of the pointing operation on the image information of the screen displayed at the time of detection of the descriptive word. Thus, the pointing target is detected.

In this example, the pointing target detecting section 63 includes a pointing device selecting section 63a, a pointing operation start correcting section 63b, a pointing operation recording section 63c, and a pointing operation end determining section 63d.

The detailed processing operation of the instruction target detection unit 63 will be described later in detail, but the outline is that the instruction target detection unit 63 detects the instruction word in response to the notification from the instruction word detection notification unit 62. Upon detection, the pointing device selecting unit 63a selects an appropriate pointing device as the instruction operation of the instruction word from the plurality of pointing devices 20 to 24 of each of the plurality of participants as described later. I do.

When there is an instruction operation on the display screen for instructing the instruction target of the instruction word, a data storage timing signal TM indicating a data storage request is sent to the file storage unit 66.
Output to On the other hand, when the instruction target of the instruction word is not on the display screen, and thus there is no instruction operation corresponding to the instruction word, the data storage timing signal TM to the file storage unit 66 is not output.

When it is detected that there is an instruction operation corresponding to the instruction word, the actual instruction operation is performed due to a time delay from the time the instruction word is uttered until the instruction word is detected. In order to prevent the start point from being unable to be recorded, the instruction operation start correction unit 63b obtains the start point of the instruction operation using the data of the instruction history recording unit 67, as described later. Then, the trajectory of the instruction operation is recorded as series data of a series of position coordinates by the instruction operation recording unit 63c. Instruction operation end determination unit 63
In d, it is determined whether the instruction operation is completed.

[0092] The image capturing section 64 includes a designated word detection notifying section 62.
In this example, a snapshot (still image information) of the screen information currently displayed on the large screen display device 2 at that time is captured as image data, triggered by the detection notification of the instruction word from. That is, since the personal computer 1 manages the image data to be projected and displayed on the screen 2D of the large screen display device 2, the still image data at that time is taken in from the management data.

Note that, as described later, the image capturing unit 6
4 is not stored as conference information, but the captured still image information is sent to the file storage unit 66 only when an instruction operation corresponding to the instruction word is performed. This is to be stored as information.

In the case of this example, the image capturing unit 64
When there is a notification of the detection of the instruction word from the instruction word detection notification unit 62, the previously captured still image is compared with the still image to be captured this time. At 66, instead of image data,
Send a notification that the image content has not changed.

The timer 65 is for recording a time stamp (time) at the time of detecting the instruction word. The time stamp is information for association (link) with the utterance time of the instruction word of the audio information in which everything is stored from the start to the end of the conference as the conference information. When the recording start time and the recording end time of the audio information are recorded in absolute time, the absolute time information is used as the time stamp. When the recording start time and the end time of the audio information are not recorded, the information of the relative time starting from the conference start time, which is the recording start time of the audio information, is used.

The file storage section 66 stores the voice input device group 5
4, trajectory data of an instruction operation detected from the instruction target detection unit 63, image data from the image capture unit 64, and a time stamp as described above at the time of detection of the instruction word from the timer 65. Is stored in the storage area 70 provided in the file storage device 66.

In the case of this example, audio information from the audio input device group 54 stored in the file storage unit 66 is obtained by converting analog audio signals from each of the microphones 30 to 34 into
The digital signal is converted into a digital signal by the A / D converter 68, and the digital signal is mixed by the audio signal mixer 69. The image data is still image data.

In this case, the file storage unit 6
6 does not always store the above input data,
The data is stored only at the time point determined by the storage timing signal TM from the pointing target detection unit 63. This storage timing signal TM is, as described above,
, Is output when it is determined that an instruction operation has been performed to indicate the target of the instruction word when the instruction word is uttered.

In the file storage section 66, as will be described later, the image data is stored as a pair of image data of a still image and an image pointer for specifying the storage position of the still image data at the time of storage. However, as described above, when a notification indicating that there is no change in image data is received from the image capturing unit 64 during storage, the image data of the still image is not stored, and the corresponding still image data is Data indicating the same as the still image data, for example, nil, is recorded as an image pointer.

Note that the image capturing section 64 always performs the image capturing operation when the instruction word detection notification is received from the instruction word detection notification section 62, and stores the image data in the file storage section 66. It is determined whether the previously stored image and the image to be stored this time are the same, and when they are the same, the same still image data is not stored and the same as the previous still image data is not stored. Data, such as nil
May be recorded as an image pointer.

In the storage area 70 of the file storage 66,
In this example, as shown in FIG. 3, a file is divided for each data and stored, and link information for linking the files is also stored as a file.

That is, in the storage area 70, a link information file 71, an audio data file 72, an image data file 73, and an instruction target file 74 are recorded.

[0103] The link information file 71 is a file that holds the correspondence between the audio data, the image data, and the pointing target through the time stamp.

As described above, the audio data file 72 digitizes the audio information from the plurality of microphones 30 to 34 input from the audio input device group 54 by the A / D converter and holds the mixed data. .

The image data file 73 is stored in the image
4 and retains the still image data at the time when the instruction operation corresponding to the instruction word is detected.

The instruction target file 74 holds data relating to the instruction target of the instruction word detected by the instruction target detection unit 63 as series data of the position coordinates of the corresponding instruction operation.

FIG. 4 is a diagram showing the data structure of the information files 71 to 74 stored in the storage area 70.

The link information file 71 includes a designated word detection time 101, a designated pointer 102, and an image pointer 103. The instruction word detection time is a time stamp indicating the time at which the instruction word was detected, and is a value generated by the timer 65. Based on the time stamp, a link relationship is generated with the position where the audio information corresponding to the instruction word of the audio data file 72 is stored. Reference numeral 104 denotes a voice data portion, and the utterance portion of the instruction word is shown by being surrounded by a square. .. Are recorded in the order of utterance time, as instructive word 1, instructive word 2, instructive word 3, and so on.

In FIG. 4, the time stamp linked to the directive 1 is TimeStamp1, the time stamp linked to the directive 2 is TimeStamp2, the time stamp linked to the directive 3 is TimeStamp3.

The pointing target pointer 102 of the link information file 74 holds a pointer to series data of position coordinates indicating a series of pointing operations of the pointing target file 74. In the case of the example of FIG. 4, the timestamp TimeStamp1 is used as a link, and the pointing target pointer of the data of the pointing operation corresponding to the pointing word 1 is Obj-Pointer1. Further, with the time stamp TimeStamp2 as a link, the pointing target pointer of the data of the pointing operation corresponding to the pointing word 2 is Obj-Pointer2. The same applies hereinafter.

The image pointer 103 holds a pointer to the image data file 73 of the image data when the instruction word is detected and the corresponding instruction operation is detected.

The image data file 73 is sequentially stored in a file with snapshot (still image) data for one screen as one unit. As described above, the image capturing unit 64 compares the data with the previous screen at the time of the capturing processing, and if the data is the same, does not store the data in a file, and selectively selects only the changed screen data. I try to take in. As described above, when there is no change in the screen data between the time when the previous directive is generated and the time when the next directive is generated,
The image pointer value is nil.

In the case of FIG. 4, the time stamp Time
An image pointer corresponding to the instruction word 1 is a link, an image pointer corresponding to the instruction word 1 is Image-Pointer1, a time stamp TimeStamp2 is a link, an image pointer corresponding to the instruction word 2 is nil, and an image pointer corresponding to the time stamp TimeStamp3 is a link. The pointer is Image-Pointer2. FIG. 4 shows that a link is provided between the image data 2 and the image pointer Image-Pointer2 of the third record of the link information file.

The pointing target file 74 includes the selected pointing device number PDNo, and a series of position coordinate series data (x0, y0) to (x0) to (x0).
n, yn) as one unit. In the example of FIG. 4, a link relationship between the pointer to be pointed and the second record Obj-Pointer2 is shown.

The instruction history recording section 67 receives the movement amount data Pm0 to Pm4 from each of the plurality of pointing devices 20 to 24, and the instruction coordinate generation section 53.
, And stores and stores the coordinate data of each pointing device up to a predetermined time before the current time.

In the case of this example, the instruction history recording section 67 includes a data recording section 67a, and a buffer memory 67b for recording and storing coordinate data for a predetermined time for each pointing device by the data recording section 67a. Each pointing device 20 is stored in the buffer memory 67b.
To 24 dedicated memory areas are set.
These dedicated memory areas each have a capacity capable of recording the coordinate data for the fixed time, and for each pointing device, always, from the latest coordinate data, the coordinate data up to the fixed time before that, It is configured to be written to each memory area.

For example, the memory area for each pointing device in the buffer memory 67b is configured as a ring buffer, and when the coordinate data is recorded in the entire memory area, the next latest coordinate data becomes the oldest coordinate data. Overwritten on the data.

As described above, the data Pm0 to Pm4 of the movement amounts of the pointing devices 20 to 24, respectively.
Contains the identification ID (for example, pointing device number) of each pointing device. The data recording unit 67a of the instruction history recording unit 67 monitors the movement amount data Pm0 to Pm4, and when the pointing device is picked up by a conference participant, the pointing device whose movement amount is recognized to be significant is recognized. Is detected. When the data recording unit 67a recognizes that the movement amount of a certain pointing device is significant, the data recording unit 67a uses the coordinate data obtained from the pointing coordinate generation unit 53 at that time as the pointing position data of the pointing device in the buffer memory 67b. Is written to the memory area for the pointing device.

When a plurality of pointing devices are used at the same time, which coordinate data corresponds to which pointing device is selected based on the correlation between the movement amount data Pm and the change in the coordinate data. Upon recognition, coordinate data corresponding to the designated position on the screen 2D of the corresponding pointing device is recorded in each memory area of the buffer memory 67b. In addition,
As described above, the coordinate data Co is data converted into a coordinate system on the screen of the personal computer 1.

In this way, a log of the coordinate data of the pointing devices 20 to 24 up to a certain time before the current time is recorded in the buffer memory 67b. When the pointing device is not pointing on the screen 2D, specific data indicating that fact is recorded as the coordinate data.

The determination as to whether or not each pointing device has been operated is made not by detecting the amount of movement of each pointing device itself as in the above-described example, but by using a power switch for each pointing device. It may be so arranged that when the power switch is turned on, it is determined that the operation state has been reached. Instead of the power switch, a trigger switch such as an operation start switch may be provided, and the operation may be determined by turning on the trigger switch.

Next, a detailed example of the processing operation of each main part in FIG. 1 will be described.

First, a description will be given of a descriptive word detection process in the descriptive word detection unit 61. In the case of this example, the descriptive word detection unit 61
, Which detects a directive by voice recognition and is not shown in FIG. 1, but has a directive table TBL. This descriptive word table TBL is like a dictionary in which features relating to the voice information of the descriptive word are registered.

FIG. 5 is a view for explaining an example of the data structure of the descriptive word table 120. In the case of this example, the descriptive word table 120 is composed of a descriptive word field 121 and a waveform spectrum field. The instruction word field 121 is a text expressing an instruction word such as “this”, “that”, “these”. In the waveform spectrum field 122, a waveform spectrum indicating a feature parameter of the voice of the corresponding descriptive word is registered.

The instruction word detection unit 61 detects an instruction word using the instruction word table TBL. As previously mentioned,
The descriptive word detection notifying unit 62 sends a descriptive word detection notification based on the detection of the descriptive word to the instructed target detecting unit 63 and the image capturing unit 6
4. Send to timer 65.

FIG. 6 shows the instruction word detection unit 6 of the recording control unit 60.
6 is a flowchart for explaining the processing of a designated word detection notification unit 62.

First, in step 131, the A / D converter 68 receives the audio data converted from the audio data input from the audio input device 54 into a digital signal. In step 132, the section of the audio information to be recognized is detected. In this example, when a sound pressure equal to or higher than a certain threshold is continuously detected for a certain time or more, the section is extracted. Substitute the voice data of the extracted section into the variable Input.

Next, in step 133, feature analysis is performed on the extracted voice data Input to extract feature parameters, in this example, a waveform spectrum, and substitute them for a variable Parameter. In the next step 134, the variable Parameter is collated with the waveform spectrum field 122 of the instruction word table TBL. Then, in the next step 135, it is determined whether or not, as a result of the collation, an instruction word that is similar to the extent that it can be determined to be the same (this is called matching) is detected.

If the result of determination in step 133 is that there is no matched instruction word, the process returns to step 131 and the above-described instruction word recognition processing is continued. On the other hand, as a result of the determination in step 133, when a matched instruction word is detected, the process proceeds to step 136, where true is assigned to an instruction word detection flag Iflag. Next, in step 137, a designated word detection notification process is performed. Here, the pointing target detection unit 63, the image capturing device 64, and the timer 65 include:
Notify the event that the descriptive word was detected. As a result, the activation of the processing of each device is triggered.

Next, the processing of the pointing target detection unit 63 will be described. As described above, the pointing target detection unit 63 detects the pointing device operated in accordance with the spoken instruction word from the plurality of pointing devices, and indicates the pointing target by the detected pointing device. This is for detecting an operation trajectory.

FIG. 7 is a diagram for explaining an instruction operation using the pointing device. The instruction operation is, for example,
"There are numbers .....", "This measure is ....."
Thus, it is a situation where the user is having a conversation while pointing to a part of a document, a graph, or the like, which is displayed on the display, using the instruction word. Normally, it is considered that one conference participant performs the instruction operation while uttering the instruction word. However, the speaker of the instruction word and the person who performed the instruction operation may be different. There is a case where a moderator or another participant near the display performs an instruction operation in response to a participant's utterance of an instruction word.

There are various ways of the instruction operation.
As in the instruction operation trajectory 111 of FIG.
As in the case of the instruction operation trajectory 112 in FIG. 7B, the user may point out by moving in the horizontal direction. The instruction operation may be performed in one stroke (a series of operations continuously moving at a predetermined speed is referred to as a stroke) or may be performed in several strokes.

The target pointed by the pointing operation is the movement locus of the pointing operation. Therefore, detecting and specifying the pointing target records the locus of movement of the pointing device from the start point to the end point of the pointing operation of the pointing target.

FIG. 8 is a flowchart for explaining the outline of the pointing target detection processing in the pointing target detection unit 63.

First, in step 150, it is determined whether or not a notice of a designated word has been received. As described above, when a descriptive word is detected, a value of true is set to Iflag, which is a descriptive word detection flag, and a notification is received by the descriptive word detection notifying unit 62, so that the notification is detected.

When the instruction word detection notice is detected, the process proceeds to step 151, where the instruction word detection flag Iflag is returned to false. In the next step 152, the initial value of the list pdData for holding a series of series data of the pointing device position coordinates of the pointing operation is set to an empty list. Then, the process proceeds to a step 153, wherein a processing routine for selecting an appropriate pointing device used to indicate the object referred to by the descriptive word from the plurality of pointing devices is executed. The pointing device selection processing routine will be described later. A pointing device number, which is identification information of the selected pointing device, is returned as a result of the processing in step 153.

Next, proceeding to step 154, a processing routine for correcting the delay in detecting the start point of the instruction operation by the time required for recognition and detection of the instruction word and obtaining the start point of the correct instruction operation is described. Run. This instruction operation start point correction processing routine will be described later. As a result of this correction processing routine, a sequence of position data corresponding to the time from the instruction word utterance time to the instruction word detection time is held and returned in the list pdData.

Next, the routine proceeds to step 155. In this step 155, the unit time Δt is added to the time variable T, and the next position data after the detection time of the instruction word to be added to the list pdData of the position data sequence resulting from the processing routine of the step 154 is detected. Set the time to perform. That is,
The variable T is a variable for managing the timing of recording the position coordinates of the pointing device.

Steps 156 to 159 are processing for extracting a series of position data of the pointing device of the pointing operation for indicating the pointing target. Position data is extracted for each predetermined unit time Δt. For example, by setting Δt = 0.1 second, 10 seconds per second
Pieces of position data can be detected, and a data amount sufficient to accurately recognize the pointing target can be obtained.

In step 156, the time generated by the timer 65 is monitored to determine whether the current time Tc has passed the time T. If the current time Tc has not passed the time T, monitoring is continued again. If the current time Tc has passed the time T, step 15
Go to 7.

In step 157, the position coordinates of the selected pointing device PD at the time T are detected, and the detected position data is added to a list pdData holding position series data. Then, step 15
At 8, a process is performed to determine whether the instruction operation has been completed.
The details of the process of determining the end of the instruction operation in step 158 will be described later. As a result of the end processing, an end determination flag indicating whether or not the end of the instruction operation has been determined is returned.
The end determination flag indicates that the end of the instruction operation has been determined.
true, false if not finished.

Then, the process proceeds to a step 159, wherein a judgment is made as to the end judgment flag. If the end judgment flag is false, the process returns to the step 155 to continue the processing for recording the instruction operation. If the end determination flag is true, the instructed object detection processing routine of FIG. 8 ends.

Next, an example of the pointing device selection processing in step 153 will be described. In this example, priorities for instruction operations are set in advance as attributes of the conference participants. Then, the detection of the instruction operation corresponding to the instruction word is performed according to the priority order. That is,
The process of selecting the pointing device on which the instruction operation has been performed is performed in descending order of the probability of performing the instruction operation.

FIG. 9 shows a participant attribute table 180 used to select the pointing device and holding data on the priority between participants. This participant attribute table 180 has priority 181,
Participant attribute name 182, evaluation function 183, comment 184
Field. In this embodiment, each participant is assigned a pointing device, so each participant is identified by a pointing device number.

The priority 181 is a numerical value indicating the priority of the conference participant. "1" has the highest priority, and the higher the numerical value, the lower the priority. In FIG. 9 of this embodiment, the priority is registered up to “3”. The participant attribute name 182 is a label indicating the attribute of the participant. The evaluation function 183 is a function that defines a condition for satisfying the participant attribute, determines whether a pointing device possessed by each participant satisfies the condition, and returns a list of pointing device numbers that satisfy the condition. is there. Comment 1
Reference numeral 84 denotes a text representation of the contents of the participant attribute. Note that the comment 184 field may not be provided.

For example, the priority “1” has the highest priority. In this embodiment, an attribute of “designated word speaker” is assigned to the priority “1”. The evaluation function for determining the "instructor speaker" is Findicator. This evaluation function Findinator is a function that detects a pointing device corresponding to the voice input device in which the instruction word has been detected, and returns the pointing device number.

Similarly, the attribute “moderator” is assigned to the priority “2”. The evaluation function for determining “moderator” is Fmoderator. This evaluation function Fmoder
ator is the moderator's pointing device,
In the case of this example, the function returns a pointing device number corresponding to the pointing device 20 in FIG.

The attribute “other participant” is assigned to the priority “3”. The evaluation function for determining “other participants” is Fparticipant.
This evaluation function Fparticipant is a function that returns the pointing device number of a person who is not the demonstrator and is not the presenter.

Next, FIG. 10 and FIG.
FIG. 9 is a diagram showing a flow of a pointing device selection process for selecting a pointing device indicating a pointing target referred to by a detected descriptive word from a plurality of competing pointing devices.

First, a series of variable initial value setting processing is performed. That is, in step 200, the numerical value with the lowest priority is substituted as the initial value of the variable P. In this embodiment, “3” is substituted. In the next step 201, "1" is set as an initial value in a priority index variable prior. In the next step 202, a pointing device that does not move at all as an instruction operation is virtually defined as an initial value of the pointing device number PD as the final output, and the number PD0 is set. Here, the pointing device without any movement is
This is a virtual pointing device in which the variation amount of the series data of the position coordinates in the buffer of the instruction history recording unit 67 is all zero.

Next, a process of selecting a pointing device indicating the target of the instruction word is executed after step 203. First, in step 203, it is determined whether the processing has been completed for all priorities registered in the participant attribute table 180. If not, the process proceeds to step 204.

In step 204, the variable prio at that time is
Fetch the function that evaluates the participant attribute with the priority corresponding to r and assigns it to the variable function. In step 205, the evaluation function is executed, and the returned value is set in the variable pdLi.
Assign to st. In this process, the participant who satisfies the participant attribute corresponding to the variable prior, that is, the evaluation function functi
A list of pointing device numbers satisfying the condition defined by on is returned as a variable pdList.

At step 206, it is determined whether or not the variable pdList is nil. Step 205 to Step 2
If the variable pdList is nil, the judgment at step 206 is whether or not there is a participant attribute of the priority indicated by the index variable prior at that time. That is, none of the attributes satisfied the participant attribute corresponding to the priority index variable prior at that time. Therefore, in this case, the process proceeds to step 211 for the next priority process, and after adding 1 to the value of the variable prior, the process returns to step 203.

In the determination at step 206, the variable pd
If there is any device number in the list, there is a device that satisfies the participant attribute corresponding to the index variable prior of the priority at that time, and the process proceeds to step 207 in FIG. In this step 207, one pointing device number is extracted from the variable pdList, and
Substitute for pd and go to step 208.

In step 208, it is determined whether or not the pointing device of the number indicated by the variable pd indicates a significant movement as an operation for designating the target of the instruction word. The result is true if significant, otherwise
false is returned. The process of determining the significance of the pointing device will be described later in detail.

Then, in the next step 209, the result of the determination of the significance is determined. If the movement of the pointing operation of the pointing device of the number indicated by the variable pd is not determined to be significant, the process proceeds to step 210, where the variable For all pointing device numbers in the pdList, it is determined whether or not the significance of step 208 has been determined.
If there is still another pointing device number in st, the process returns to step 207 to execute significance determination processing. If it is determined in step 210 that the significance determination has been completed for all the pointing device numbers in the variable pdList, the process proceeds to step 211 for processing the next priority, and 1 is added to the value of the variable prior. , And return to step 203.

If it is determined in step 209 that the pointing operation of the pointing device with the number indicated by the variable pd is significant, the flow advances to step 221 to hold the pointing device number pd. And
In the next step 222, an evaluation function Dmax is executed to determine which of the pointing device candidates PD to be selected and the held variable pd is a pointing device that is superior as an instruction operation.

The evaluation function Dmax determines the magnitude of the moving amount of the pointing operation of the two pointing devices PD and pd, and returns the device number of the larger moving amount. As means for comparing the magnitude of the movement amount, for example, in the pointing history recording unit 67, in the area of the buffer corresponding to each pointing device, the variation Δ of the pointing position of the pointing device obtained from the respective coordinate data Δ
From x and Δy, the sum of the movement amount Δd = √ (Δx ** 2 + Δy ** 2) of the designated position during the predetermined unit time Δt is compared, and the larger one is selected. here,**
2 indicates a square operation. Also,） () is a parenthesis ()
Means the square root of Hereinafter, the same applies.

The means for comparing the magnitude of the movement amount of the instruction operation is, of course, not limited to this method.
Individually compared for each movement amount Δd during a predetermined unit time Δt,
A method of selecting a higher rate of change in the movement amount Δd may be used.

Then, in the next step 223, the variable pd
It is determined whether or not the significance of step 208 has been determined for all the pointing device numbers in the list. If there is still another pointing device number in the pdList, the process returns to step 207, and the other priority device of the same priority is returned. The above-described determination processing is executed for the pointing device.

If it is determined in step 223 that the determination of the significance has been completed for all the pointing device numbers in the variable pdList, the process proceeds to step 224, where the finally selected pointing device number p
d is assigned to the output pointing device number PD and output.

At step 203, participant attribute table 1
When it is determined that the processing has been completed for all the priorities registered in 80, as a result of the determination processing of the pointing operation for all the pointing devices, no pointing device determined to be significant is found. Means that At this time, step 212
To ni for the pointing device number PD to be output.
Substitute l and terminate the process.

FIG. 12 is a flowchart showing an outline of a processing example of the significance judgment processing step 208 for judging a pointing device whose instruction operation is significant in the above-mentioned pointing device selection processing routine.

First, in step 231, it is determined whether or not the current position coordinates of the corresponding pointing device fall within the coordinates indicating the size of the screen 2D of the large screen display device 2. If it does not fit on the screen screen, the process returns to step 234 with the return value set to false. That is, it is determined that the movement of the corresponding pointing device is not a significant instruction operation.

On the other hand, if the current position coordinates of the corresponding pointing device fall within the screen screen, it is determined in step 232 whether a significant movement amount of the pointing operation exists. Details of the determination of the significance of the movement amount of the instruction operation will be described later with reference to FIGS.

If it is determined in step 232 that the value is significant, the return value is set to true in step 233, and the process ends. If it is determined that the movement amount of the instruction operation is not significant, the process proceeds to step 234, and the return value is set to fa.
The processing ends as lse.

FIG. 13 is a diagram for explaining a significant movement amount of the pointing position of the pointing device in this example. In FIG. 13, the vertical axis represents the movement amount Δd of the pointing position of the pointing device for a predetermined unit time Δt, and Δd = √ (Δx ** 2 + Δy ** 2) as described above. The horizontal axis represents time t.

In FIG. 13, ΔD represents a predetermined moving distance of the designated position per unit time Δt that is recognized as significant. If Δd ≧ ΔD, the moving amount of the pointing position of the corresponding pointing device is a certain section Δ
It is considered significant at t. Here, as an example, ΔD = 5 is set.

For example, during Δt, Δx = 3 dots, Δy
= Suppose that it has moved by 4 dots. At this time, Δd = √ (9 + 16) = 5 ≧ ΔD, which is determined to be significant.

FIG. 14 is a flowchart showing a process for determining whether the movement amount of the pointing position of the pointing device is significant. This corresponds to the process of step 232 in FIG.

At step 251, the operation history recording section 67
Pointing device pd of buffer memory 67b
The position coordinate series data is read from the buffer area holding the movement trajectory, and is substituted for the variable Bxy. In the next step 252, a difference in coordinate data between Δt is obtained from the position coordinate series data of the variable Bxy, a moving distance Δd is calculated from the difference, and the moving distance Δd is substituted into the moving distance list ΔdList.

At step 253, the moving distance list Δd
In the List, it is determined whether there is a significant movement amount in a certain section. That is, Δd ≧ ΔD is determined. All the sections are determined, and if the unit section Δt determined to be a significant movement amount in the section is equal to or greater than a predetermined ratio R0, the corresponding pointing device pd is determined to be significant. Here, for example, if R0 = 70%, for example,
When the ratio of Δd determined to be significant is 75%, that is, when Rate (Δd ≧ ΔD) = 75%, the corresponding pointing device pd is determined to be significant.

If it is determined in step 253 that it is significant,
In step 254, a value of true is returned as a return value. On the other hand, if it is determined that it is not significant, the return value is set to false in step 255, and the process ends.

Next, the processing operation of the pointing operation start correcting section 63b of the pointing target detecting section 63 will be described in detail.

FIG. 15 is a diagram for explaining the concept of the time difference between the starting point of the pointing operation of the pointing target and the point of time when the pointing word is detected.

It takes a certain time ΔT to detect a command word by the voice recognition process in the command word detection unit 61. Therefore, assuming that the descriptive word is detected by the descriptive word detection unit 61 and the time of the time stamp at the time when the descriptive word obtained from the timer 65 is detected by the detection notification is the descriptive word detection time Tr, the descriptive word is actually spoken. A delay ΔT corresponding to the recognition time occurs between the instruction word utterance start time Ts and the instruction word detection time Tr.

Assuming that the instruction operation of the instruction target is started from the instruction word utterance start time Ts, the start point of the trajectory of the pointing device of the instruction operation is set to be from the instruction word utterance start time Ts. It is necessary to perform a correction process that returns the time point so that the instruction operation is started as accurately as possible.

For this correction process, in this embodiment, a predetermined time ΔT is determined in advance. Actually, the time difference ΔT between the utterance start time Ts of the instruction word and the detection time Tr is shown.
Varies depending on the length of the instruction word and the complexity of the process of detecting it, and there is some error between ΔT and the actual Tr-Ts instead of a fixed time. But,
In this example, assuming that the error is a value that can be ignored, for example, assuming that the delay in recognition in the case of an instruction word with an average number of phonemes is 2 seconds, ΔT is set to 2 seconds. . It is to be noted that ΔT may not be set to the fixed time but may be determined in proportion to the number of phonemes of the detected instruction word. Further, in consideration of the fact that the instruction word utterance start time Ts does not always coincide with the instruction operation start time and the instruction operation start time is slightly earlier, ΔT may be set to be larger than Tr-Ts. Good.

Next, FIG. 16 shows a buffer memory 67 for storing series data of the position coordinates of the pointing device.
FIG. 9 is a diagram illustrating a data structure of each memory area of FIG. Each pointing device records and holds a log of position coordinates for a certain time ΔT in a corresponding memory area of the buffer memory 67b. As described above and as shown in FIG. 16, the memory area of the buffer memory 67b is set one by one for each pointing device, and all the pointing devices used in the conference are stored in such a buffer memory. It has an area and records sequence data of position coordinates.

As the data structure, the first record includes:
Corresponding pointing device number PD1, PD
2,..., PDn (n is the number of provided pointing devices) are recorded. In this example, the position coordinates of the pointing device for each unit time Δt are recorded after the second record. Thus, a log of coordinate data of each pointing device from the current time Tc to a time before ΔT can be held. For example, ΔT = 2 seconds, Δt =
If it is set to 0.1 second, a record of n = 20 position coordinates is held.

FIG. 17 is a flowchart showing the flow of the pointing operation start point correction process using the data recorded in the buffer memory 67b.

First, in step 271, it is determined whether or not the first record of each memory area of the buffer memory 67b matches the currently selected pointing device PD, and the matched memory area is selected. The content is assigned to the variable pdBuff.

Then, in the next step 272, the variable pd
The series data of the position coordinates after the second record of Buff is extracted, and it is substituted for the variable pdData, and the resulting pdData
Is returned as the return value. By this processing, the sequence data of the position coordinates of the pointing device PD from the start time of the instruction operation at the instruction word utterance start time to the current time is held in pdData.

As described above, the position coordinate data of the pointing device PD after the current time is sequentially recorded in the instruction operation recording unit 63c for each Δt.

Next, the processing of the instruction operation end determination section 63d will be described.

FIGS. 18 and 19 are diagrams for explaining the outline of the process of determining the end of the instruction operation. In the case of this example, the termination determination conditions of the instruction operation are three conditions of time, the movement amount of the pointing device, and detection of the utterance of the next instruction word.

FIG. 18 is a diagram for explaining an end determination condition relating to the amount of movement of the pointing device. As in the case of FIG. 13, the vertical axis represents the movement amount Δd of the pointing position of the pointing device for a predetermined unit time Δt, and the horizontal axis represents time t.

In this example, it is determined that the pointing operation has been completed when the moving amount of the pointing position of the pointing device becomes equal to or less than the moving distance ΔD per unit time Δt which is regarded as significant as described above. However, the end determination time ΔT2
Is set, and in the section of the end determination time ΔT2, when the moving amount Δd is equal to or less than ΔD at a predetermined ratio or more and is equal to or less than ΔD, the head time of the section of the end determination time ΔT2 is determined as the instruction operation end time.

In this example, the end determination time ΔT2
Is set to the same value as the instruction operation start correction time ΔT described above. Thus, the end determination can be performed by the same processing as the determination of the significant movement amount of the designated position described with reference to FIGS.

In this end judgment, the ratio judgment condition in the section of the end judgment time ΔT2 is, for example, Rate (Δd ≦ ΔD)
≧ R1. In this example, R1 is set to 70%. For example, when the moving amount per unit time Δt is less than ΔD is 8
If it is 5%, it is determined that the movement of the pointing operation of the pointing device has been completed.

FIG. 19 is a diagram showing an end determination condition relating to time. In FIG. 19 also, the vertical axis indicates the movement amount Δd of the pointing position of the pointing device for a predetermined unit time Δt.
And the horizontal axis represents time t.

ΔT3 is the shortest recording time of the designated operation trajectory, and ΔT4 is the longest recording time of the designated operation trajectory. The shortest recording time ΔT3 is to record the pointing coordinate position of the pointing device as a pointing operation for at least ΔT3 hours even if the next instruction word is detected. On the other hand, the longest recording time ΔT4 means that even if the movement of the pointing position of the pointing device is determined to be significant, the recording is not performed any more for a maximum of ΔT4 hours.

The recording times .DELTA.T3 and .DELTA.T4 are set so that a certain period of time is sufficient to specify the target of one instruction word. This is because the information is regarded as meaningless information. For example, ΔT
Assuming that 3 = 3 seconds and ΔT4 = 10 seconds, the trajectory of the pointing operation of the pointing device is recorded for at least 3 seconds and 10 seconds.
After the second, the pointing operation is considered to be completed.

FIG. 20 is a flowchart showing an example of the flow of the process of determining the end of the instruction operation. First, step 30
At 1, it is determined whether the next instruction word has been detected. If the next instruction has been detected, the process proceeds to step 302. On the other hand, if the next instruction word has not been detected, the process proceeds to step 307.

In step 302, it is evaluated whether the elapsed time ΔTc from the instruction word utterance start time Ts to the current time Tc does not exceed the minimum recording time ΔT3. If a time equal to or longer than ΔT3 has elapsed, true is returned as the return value in step 303, and the process ends. If ΔT3 has not elapsed, that is, if the recording time is less than the minimum recording time, the process proceeds to step 304, and it is determined whether the pointing device corresponding to the next instruction word is the same as the current pointing device number PD. If they are the same, ΔT
In order to proceed to the processing of the next designated word without waiting for the elapse of three hours, true is returned as a return value in step 305.

If the pointing device corresponding to the next instruction word is not the same as the current pointing device number PD, the next instruction word has been detected, but the instruction operation time has not reached the shortest instruction operation time. It is determined that the processing has not been completed, and in step 306, false is returned as a return value.

In step 307, it is first determined whether or not the recording elapsed time ΔTc of the instruction operation is shorter than the shortest recording time ΔT3. Elapsed recording time of instruction operation ΔTc
Is less than or equal to ΔT3, it is determined that the instruction operation has not been completed, and in step 308, the return value is false.
Is returned.

The recording elapsed time ΔTc of the instruction operation is ΔT3
If the elapsed time has elapsed, it is determined in step 309 whether or not the elapsed time ΔTc has exceeded the longest recording time ΔT4. When the elapsed time ΔTc exceeds ΔT4, the instruction operation is determined to be ended, and true is returned as the return value in step 310. The elapsed time ΔTc is the longest recording time ΔT
If not, go to step 311 and
The amount of change in the pointing position of the pointing device is determined.

As a result of the determination in step 311, the end condition Rate of the variation of the pointing position of the pointing device is set.
If the condition of (Δd ≦ ΔD) ≧ R1 is satisfied, it is determined that the pointing operation of the pointing device has been completed, and the process proceeds to step 312 to return true as a return value. When the variation amount of the pointing position of the pointing device does not satisfy the termination condition, the process proceeds to step 313,
Return value false is returned. This is the end of the instruction operation end determination process.

As described above, in this embodiment, when one of the conference participants utters the instruction word, in addition to this, the instruction operation for instructing the instruction target of the instruction word is actually performed. If the instruction word is present, it can be determined that the instruction word is an instruction to refer to the conference material. Therefore, as the image information at that time, an image including the conference material to be instructed is taken in. Therefore, when an instruction word is merely uttered on a word and does not indicate a meeting material, image information is not recorded, so that useless recording of information can be avoided.

Further, since the image information is taken in with the utterance of the instruction word as a trigger, the materials to be referred to during the meeting can be efficiently recorded, and the contents of the meeting can be sufficiently reproduced to accurately reproduce the contents of the meeting. It can be said that

Furthermore, in the case of the above-described embodiment, the image information to be recorded is a still image, and the amount of recorded information may be smaller than in the case of recording a moving image. Also, since the same still image is not taken in, the amount of recorded information is also small in that respect.

Further, since the trajectory of the pointing operation to the pointing object when the indicating word is uttered is recorded in association with the utterance time of the indicating word, it is possible to accurately grasp the indicating object of the indicating word. We can expect what we can do. In addition, since the trajectory of the instruction operation is recorded as far back as the utterance of the instruction word, the trajectory of the instruction operation is also accurate. In addition, the end of the instruction operation trajectory is determined based on three conditions of time, the movement amount of the pointing device, and detection of the utterance of the next instruction word, so that information that is meaningless in relation to the instruction word is determined. It is possible to record a locus necessary and sufficient as an instruction operation by eliminating as much as possible.

In the above-described embodiment, the personal computer 1 electronically takes in the image information stored in the memory. However, the image displayed on the screen 2D is taken by the camera including the surrounding conference scenery. You may take a picture and take it. In that case, the image capturing unit 64 has a configuration of a video camera, and captures and captures an image triggered by a detection notification from the instruction word detection notification unit 62 as a trigger.

[0205] When capturing image information including the scenery of a conference by photographing with the camera in this way, a descriptive word is uttered, and the image information at the time of the corresponding instruction operation is transferred from the start to the end of the instruction operation. In this section, not only the target of the instruction operation but also the instruction operation is recorded if the image is taken by the camera and recorded as a moving image, so that it is not necessary to record the instruction operation trajectory.

Next, a description will be given of a conference information reproducing process for reproducing a conference by using the voice information, the image information, and the information of the trajectory of the instruction operation recorded as in the above-described embodiment. .

FIG. 21 is a block diagram showing a system configuration diagram relating to a reproducing system of an embodiment of a conference information recording / reproducing apparatus according to the present invention. This is an example in the case of the reproducing system of the conference information recording / reproducing apparatus whose recording system has been described with reference to FIGS.

That is, in this embodiment, the personal computer 1 includes a reproduction control unit 400 in addition to the recording control unit 60 shown in FIG. FIG.
The reproduction control section 400 mainly shows the processing control function as a block, and each block is actually realized by software.

As shown in FIG. 21, in this example,
As described above, the storage area 70 stores the audio data file 7 associated via the link information file 71.
2, an image data file 73 and an instruction target file 74 are stored. Then, while the content of each data file in the storage area 70 is controlled by the reproduction control unit 400,
The content of the conference is reproduced almost exactly by being reproduced.

[0210] However, the audio information is only used in the reproduction control section 400 for reproduction control processing of images and instruction operations, and the reproduction is performed in the audio reproduction device 430. That is, the audio information from the start to the end of the conference stored in the audio data file 72 is sequentially supplied to the audio reproducing device 430 and is sequentially reproduced. In this embodiment, the reproduction control unit 400 also has a function of performing reproduction instruction control such as reproduction start, stop, rewind, and jump on the audio reproduction device 430.

[0211] Although not shown, the audio reproducing device 430 includes an audio data acquisition unit that extracts audio data from the audio data file 72 and an analog digital audio data from the acquired audio data file 72 under the control of the reproduction control unit 400. A D / A converter for converting to an audio signal;
An amplifier for amplifying the analog audio signal and a speaker. The amplifier is provided with a sound quality adjustment function and a volume adjustment function.

[0212] The display device 440 is for displaying image information, instruction operation information, and the like for complementing the conference audio reproduced by the audio reproducing device 430. In this embodiment, in order for an individual user to reproduce and view the conference information, there is no need to project the information on the large display device 2 and display it in an enlarged manner. It is sufficient to display the information on the display of the personal computer 1. Display device 4
Reference numeral 40 corresponds to a display of the personal computer 1. Of course, the display may be enlarged on the screen 2D.

[0213] The reproduction control section 400 includes a timer 401, a display time determination section 402, a voice information display control section 410, and an instruction target display control section 420 as its functional blocks.

[0214] The reproduction control unit 400 includes a sound reproduction device 430.
While playing back the voice data, the display unit 440 displays the image information on the corresponding pointing target object in synchronization with the utterance time of the pointing word.

At the time of reproducing the audio information, the audio information display control unit 410 acquires the audio data from the audio data file 72, acquires the link information (designated word detection time) from the link information file 71, and performs the current reproduction. A display for displaying a change in the reproduction position is performed based on the position.
In this example, a change in the progress of the playback position uses a playback audio waveform. In this way, the audio information is reproduced and displayed as visual information, and the time at which the instruction word is detected is also reproduced and displayed as visual information. Further, the audio information display control unit 410 has a function of displaying a button icon for drive control of the audio reproduction device 430.

[0216] The audio information display control section 410 comprises an audio-waveform conversion section 411, an instruction word link display control section 412, and a reproduction instruction button display control section 413.

The audio-waveform converter 411 converts the frequency of the audio information into a visual waveform shape and displays it on the screen. In the instruction word link display control unit 412, the instruction word exists in a portion corresponding to the instruction word in the reproduction position change display indicated by the audio waveform, and a link to image information associated with the instruction word exists. This is visualized and displayed.

The playback instruction button display control unit 413 displays a button icon for instructing playback to the audio playback device 430. When a button icon is selected by a user with a mouse or the like, the playback instruction button display control unit 413 displays the button icon. The reproduced instruction control signal is supplied to the audio reproducing device 430.

The display device 440 has a so-called multi-window format. In this case, as described later,
A waveform information display window and an image window are displayed. A waveform display window is formed by the audio information display control unit 410.

FIG. 22 is an image diagram of the waveform information display window on the reproduction screen. That is, FIG.
2, reference numeral 451 denotes a frame of the waveform information display window. In the waveform display window, an audio waveform display area 452, an instruction word link display area 453,
A reproduction instruction button display area 454 is provided.

[0221] In the audio waveform display area 452, audio waveform information as shown in the figure generated by the audio-waveform converter 411 is displayed in conjunction with audio reproduction by the audio reproducing device 430. In FIG. 22, the vertical line position indicated as Tc indicates the position currently being reproduced. This current playback position T
In this case, c is a fixed position on the display screen. Therefore, in this example, the sound waveform moves from right to left over time. The area on the left side of Tc is the audio that has been reproduced, and the area on the right side is the audio information to be reproduced in the future. In this example, the current playback position Tc is not the center position of the window 451, but
The window 451 is set at a position closer to the left side of the window 451 so that the future portion can be displayed more widely in time.

For this reason, the audio information display control section 410
At least the audio data corresponding to the portion of the window 451 that is on the right side of the current reproduction position Tc in the future from the point in time of reproduction by the audio reproduction device 430 is prefetched and acquired from the audio data file 72. I have.

The instruction word link display area 453 links the section of the utterance part of the instruction word in the audio information with respect to the current reproduction position Tc in conjunction with the movement of the audio waveform display area 452 in accordance with the change in the reproduction position of the audio waveform. Demonstration word link mark 45
5 is a display area indicating that a link to image information is present.

As described above, the audio waveform display area 45
2 is set so that the future portion is displayed more widely in time, so the indicator link mark 455 displayed so as to correspond to this waveform display corresponds to audio information to be reproduced in the future. Is displayed. That is, the user can foresee the utterance of the instruction word and the existence of image information linked to the instruction word.

[0225] In the reproduction instruction button display area 454, buttons for instructing an operation for reproducing audio information in the audio reproducing device 430 are displayed. In this example, the play button 45
6, a stop button 457, a rewind button 458,
A jump button 459 is provided.

Then, when the play button 456 is selected,
The reproduction of the audio / video information is started. When the stop button 457 is selected, the reproduction is stopped. When the rewind button 458 is selected, the playback position is returned. Jump button 45
When 9 is selected, a pop-up window for designating a time is opened on the screen of the display device 440, and by designating the time, audio-video information at an arbitrary time can be reproduced.

By using the jump button 459, it is possible to predict the time from the position where the indicator word link mark 455 as the preview information exists, and to jump to the position where the image information linked to the indicator word exists. Become.

The means for visualizing the change in the playback position is not limited to using the waveform of the playback sound. Any display may be used as long as it can be understood.

The display time determination unit 402 monitors the time of the timer 401, detects the time at which the designated object is displayed from the designated word detection time from the link information file 71, and detects the designated time when the display time is detected. The control unit 420 is notified of this.

[0230] The image display control section 421 of the indication target display control section 420 displays an image including the indication target corresponding to the indication word on the display device 440. Instruction target display control unit 420
Are an image display control unit 421 and an instruction operation reproduction unit 422
And a preview information display control unit 423 and an afterimage information display control unit 424.

The instruction operation reproducing section 422 reproduces and displays the locus of the instruction operation by the pointing device of the participant on the screen image displayed by the image display control section 421. The preview information display control unit 423 displays attention information on the start point of the instruction operation before reproducing the instruction operation. The afterimage information display control unit 424 displays the attention information at the end point of the pointing operation for a time sufficient to recognize the pointing target. The processing of the instruction operation reproducing unit 422, the preview information display control unit 423, and the afterimage information display control unit 424 will be described later in detail.

The screen image window is displayed by the pointing target display control section 420. FIG. 23 shows an example of this screen image window. In FIG.
Reference numeral 461 denotes a window frame of the screen image window.

This screen image window is a window for displaying the instruction target corresponding to the instruction word.
Descriptive word link mark 45 that has reached the time point Tc during the current playback
The screen image extracted from the image data file 73 corresponding to No. 5 is to be displayed.

An instruction operation is reproduced in order to display an instruction target in the displayed screen information. The instruction operation is displayed as the movement of the arrow cursor icon 462. In FIG. 23, the movement locus of the arrow cursor icon 462 is represented by a thick solid line 463.

When displaying image information on the display screen, various display methods can be considered. In this embodiment,
A method of displaying in a reduced mode is adopted. In the reduction mode, the entire captured screen image is reduced and displayed at an arbitrary size so that both the waveform information display window and the screen image window fit within the screen of the display device 440.

As another method of displaying image information on the display screen, there is, for example, a full screen mode in which the image is displayed in the same size as the original screen. In the full screen mode, since the waveform information display window and the screen image window overlap with each other, the waveform information display window is iconified or the reproduction instruction button display area 4 is displayed.
A display method of making the entire screen as visible as possible, for example, using only 54, is also adopted.

As in the window frame mode, the screen size is not changed from the original size, and a rectangular area of a certain size including a portion where the pointing operation is performed is cut out and displayed, or the periphery of the pointing operation is displayed. May be provided in detail, and a fisheye lens mode or the like for roughly displaying other areas may be provided.

Next, details of the processing of the instruction target display control unit 420 will be described. FIG. 24 is a flowchart illustrating an outline of the instruction target display control process. This processing includes preview information display processing 471 executed by preview information display control section 423, operation instruction reproduction processing 472 executed by instruction operation reproduction section 422, and residual image information display executed by residual image information display control section 424. This is a process of sequentially executing the three steps of the process 473. In the following, the preview information display processing 47
1. Operation instruction reproduction processing 472, residual image information display processing 473
Each process will be described.

First, the operation instruction reproducing process 472 will be described. FIG. 25 is a flowchart illustrating the flow of the instruction operation reproducing process 472.

At step 481, the sequence data of the position coordinates of the pointing device corresponding to the instruction word which activated the current processing is read from the instruction target file 74,
Assign to the variable pdData. In the next step 482, the variable
One position coordinate is extracted from pdData and assigned to a variable pd.

At the next step 483, it is determined whether or not display of all position coordinates has been completed, based on whether pd = nil.
If not, go to the next step 484,
Display processing is continued. When all position coordinates have been completed, the instruction operation reproducing process ends.

In the processing from step 484 to step 486, the display position of the arrow cursor icon 462 is
This is a process for displaying the image by moving it by a unit. In step 484, the currently displayed arrow cursor icon is deleted. The variable Cposition is a variable that holds the value of the current display position of the arrow cursor icon. In step 485, the cursor position Cposition is set to the next position.
Change to pd. In step 486, the new display position Cp
Display arrow cursor icon on osition. Cursor
Is a variable that holds the bitmap of the cursor display icon. After step 486, the process returns to step 482 to continue the processing.

As described above, the instruction operation as indicated by the solid line 463 by the arrow cursor icon 462 shown in FIG. 23 is reproduced.

FIGS. 26 and 27 are diagrams for explaining the outline of the preview information display and the afterimage information display. FIG.
It is a figure explaining the concept of preview information display time and afterimage information display time. In the preview information display process, a screen corresponding to the instruction word is displayed prior to the start time of the instruction operation, and attention information such as a cursor is set in advance at a location corresponding to the start point of the instruction operation. This is a process of displaying for a given time ΔTα. Thus, information for foreseeing the start of the operation instruction is provided to the person who reproduces the conference record.

[0245] The afterimage information display processing is performed after the end of the instructing operation.
This is a process of displaying afterimages of the trajectory of the instruction operation and attention information such as a cursor icon at the instruction end point for a predetermined time ΔTβ. This means that by displaying the instruction target for a certain period of time or more, sufficient time is ensured for recognition of the target and comprehension of the content, and even after the end of the instruction operation,
Provide information that can accurately recognize the pointing target.

FIG. 27 is an image diagram of a display screen for preview information, instruction operation, and afterimage information. In this example, for example,
The case where ΔTα = 3 seconds and ΔTβ = 10 seconds will be described.

In the case of this example, the preview information display process is started three seconds before the utterance of the instruction word, and the preview information display screen 501 as shown in FIG. 27A is opened. On the preview information display screen 501, an arrow cursor icon 462 is displayed in a bar graph in the document. The position indicated by the arrow cursor icon 462 is the start point of the operation instruction. This allows the reproducing user to foresee the instruction target portion of the instruction word.

When the instruction operation start time comes, the instruction operation reproducing process is started, and the state of the instruction operation display screen 502 as shown in FIG. 27B is obtained. On the instruction operation display screen 502, as time elapses, an arrow cursor icon 462 is displayed along the position coordinates of the pointing device.
The display moves. The trajectory of the movement of the arrow cursor icon 462 is indicated by a solid line 463.
Do not show. With the movement of the arrow cursor icon 462,
An instruction target on the display image is specified.

Upon completion of the instruction operation reproducing process, FIG.
The afterimage information screen 503 shown in FIG. On the afterimage information screen 503, a pointing operation trajectory 464 of the pointing device is displayed as afterimage information, and an arrow cursor 462 is displayed at the end point of the instruction operation. The pointing operation trajectory of the pointing device is indicated by a thin solid line or a wavy line that does not make the image information difficult to see. This afterimage information screen 503 is displayed for 10 seconds in this example, and then the screen image window is automatically closed.

FIG. 28 is a flowchart showing the flow of the preview information display control process.
The flow of this processing will be described.

At step 521, the instruction word detection time Tr
Then, the instruction operation start time (instructive word utterance time) Ts before ΔT which is the start time correction time is obtained. In the next step 522, the preview information start display time Tα is calculated by obtaining the time before the preview information display time ΔTα from the instruction operation start time Ts.

In the next step 523, the current time Tc
Determine whether the preview information display start time Tα has elapsed. If not, the loop is repeated until the time has elapsed. If the preview information display start time Tα has elapsed, the process proceeds to step 524, and the image information of the screen image linked with the instruction word being processed is displayed as a screen image window.

Subsequently, in step 525, the first data of the position coordinate series data of the corresponding pointing operation is read from the pointing target file 74, and a variable representing the cursor display position is read.
Assign to Cposition. This is data indicating the starting point of the instruction operation. In the next step 526, the variable Cposit
An arrow cursor icon 462 is displayed at the position of ion.

At step 527, it is determined whether or not the current time Tc has passed the instruction operation start time Ts. If not, the loop is repeated until the time has elapsed. When the current time Tc has passed the instruction operation start time Ts, the above preview information display processing ends.

Next, FIG. 29 is a flowchart showing the flow of the display processing of the afterimage information. The flow of this processing will be described with reference to FIG.

First, in step 541, the instruction operation time ΔTi is added to the instruction operation start time Ts = instruction word detection time Tr−ΔT, and the instruction operation end time Te is calculated. Here, ΔTi = Δt × N, and therefore, Te = Tr−ΔT + Δt × N. Note that Δt is the data collection interval of the operation instruction, and N is the number of position coordinate series data of the operation instruction recorded in the instruction target file 74.

In the next step 542, an after-image information display end time Tβ is calculated from the instruction operation end time Te and the after-image information display time ΔTβ.

At the next step 543, the current time Tc
Determines whether the instruction operation end time Te has elapsed. If not, the loop is repeated until the time has elapsed.
When the current time Tc has passed the instruction operation end time Te, the process proceeds to step 544, and first, the movement trajectory of the pointing device, which is the first information of the afterimage information, is displayed. In this example, as shown in FIG. 27C, a solid line 464 connecting the series data of the position coordinates of the pointing target is displayed.

In the next step 545, the last data of the position coordinate series data of the corresponding pointing operation is read from the pointing target file 74, and a variable representing the cursor display position is read.
Assign to Cposition. This is data indicating the end point of the instruction operation. Then, in the next step 546, an arrow cursor icon 462 is displayed at the position of the variable Cposition.

At the next step 547, it is determined whether or not the current time Tc has passed the afterimage information display end time Tβ. If not, the loop is repeated until the time has elapsed. If the afterimage information display end time Tβ has elapsed, the screen image window is closed in step 548, and the above-described afterimage information display processing ends.

As described above, according to the conference information recording / reproducing apparatus of this embodiment, the information about the external reference corresponding to the instruction word, which is ambiguous only by the voice information, is supplemented, and a more accurate conference Information can be reproduced. In addition, since the timing at which the information of the instruction target of the instruction word is fetched and the specification of the instruction target are automatically performed and recorded, the labor of the person is not bothered and the labor is reduced. Further, since the minimum necessary data for supplementing the audio information is recorded, the storage capacity can be reduced.

Also, the conflict between a plurality of pointing devices can be resolved, and the most appropriate pointing device that points to the pointing target referred to by the indicating word can be selected.
In particular, respond appropriately when the speaker does not always point to the target, when the speaker or the person who was near the display device instead of the speaker points, or when multiple persons point at the same time. This makes it possible to select a pointing device in accordance with the actual situation of the conference, thereby improving the accuracy of selecting the instruction target.

Further, the detection time point at which the instruction operation is started due to a delay in voice recognition can be corrected, and the instruction operation of the instruction target can be more accurately recorded. Furthermore, in reproducing the instruction operation, the starting point of the operation plays an important role in the sense that it first attracts attention, but since the starting point of the instruction operation can be determined more accurately, The recognition of the information is improved.

Further, since the end of the instruction operation can be automatically determined, it is not necessary for the operator of the pointing device to explicitly give an explicit instruction of the end of the instruction operation, and the labor can be reduced. Further, even when the instruction word is detected continuously within a short time, the visibility of the instruction operation can be guaranteed by recording the instruction operation for a certain time or more. Further, by stopping the recording within a certain period of time, unnecessary movement of the pointing device is not recorded more than necessary. This allows
When reproducing a recording, unnecessary information is not confused by a viewer of the recording, and only necessary information can be presented.
Further, the storage capacity can be reduced.

[0265] Further, by visualizing the voice information and the link information, it is possible to foresee the existence of some remarkable information, and there is an effect that oversight can be reduced. In addition, by reproducing image information and instruction operations that complement audio information at appropriate timing and ending presentation of information at the required timing, excessive information can be given to a person who refers to reproduction of conference information. This can prevent confusion and improve the efficiency of grasping the contents of the meeting.

Also, by displaying an image linked to the instruction word a little earlier and displaying the attention information, it is possible to foresee the existence of the instruction target. As a result, in an actual meeting, where the user is trying to point can be inferred from the behavior of the moderator and participants, but the point of interest is played back for a short time without any foresight during playback. Then, it is possible to alleviate the problem that there is a possibility of overlooking what is really pointed out, and to reduce the oversight of the point of interest.

[0267] Also, after the instruction operation is completed, the information on the instruction target is not immediately erased, but after-operation, the afterimage information on the instruction target can be displayed for a certain period of time, so that oversight of the point of interest can be reduced. . Furthermore, after the instruction operation is reproduced, even once the user turns his / her eyes away from the information, the instruction target can be easily recognized again, and the accessibility of the information is improved.

Further, since there is a means for predicting the utterance of the instruction word and the existence of the image information or the instruction operation information to be linked, it is possible to reproduce only the main conference recording part by using the means. .

In the above example, a pointing device that emits infrared rays is used. However, the pointing device is not limited to this.
For example, like a pen or a mouse, the pointing operation position may be indicated as a cursor on a display of a personal computer. In this case, it is not necessary to provide the large display device 2 with a function of recognizing the pointing operation position. The personal computer 1 can easily determine which pointing device has been operated, and the pointing device of each pointing device can be easily determined. The coordinate data of the operation position can be easily obtained.

In the above example, a case has been described in which a pointing device is assigned to each conference participant and an arbitrary participant performs an instruction operation corresponding to the instruction word. The present invention can also be applied to a case where an arbitrary conference participant performs an instruction operation using a smaller number of pointing devices than the number. That is, in this case, the pointing operation of the pointing device exhibiting a significant movement is set to the instruction operation corresponding to the corresponding instruction word, so that the image at the time of the utterance of the simple instruction word without the instruction operation is not captured. In addition, only necessary image information can be efficiently recorded as conference information.

Further, the present invention is not a system in which a plurality of participants collectively hold a conference in a conference room as shown in FIG. 2, but is connected through a network, and each conference participant is displayed on an individual display. The present invention can also be applied as a conference information recording / reproducing device of a network conference system in which a common display image is displayed and the user views the common display image while viewing the common display image.

Although the above embodiment is directed to a device for recording and reproducing conference information, the recording function and the reproduction function are separated so that the configuration of the conference information recording device alone and the conference information reproducing device alone It is also possible to adopt a configuration of However, the conference information reproducing device in that case has a configuration dedicated to reproducing the conference information recorded by the conference information recording device according to the present invention.

[0273]

As described above, according to the conference information recording method and apparatus of the present invention, image information for supplementing audio information of a conference and instruction information of a conference participant are provided.
Since only appropriate timing and appropriately selected ones are recorded, it is possible to record information that can accurately reproduce the contents of the conference with a small amount of information.

Further, since the image information for supplementing the audio information of the conference and the instruction information of the conference participants are automatically recorded at an appropriate timing, the conference participants are conscious of the recording of the conference. And there is no need to specifically reserve a person for meeting recordings.

Further, according to the conference information recording / reproducing apparatus of the present invention, it is possible to record information capable of accurately reproducing the content of a conference with a small amount of information, and to reproduce the audio of the conference during reproduction. While reproducing the information, the image information or the instruction operation linked to the instruction word is automatically displayed when the instruction word is uttered, so that it is easy to grasp the contents of the conference accurately.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a system configuration diagram of a recording system of an embodiment of a conference information recording / reproducing apparatus according to the present invention.

FIG. 2 is a conceptual diagram of a face-to-face meeting in an electronic meeting room to which an embodiment of the present invention is applied;

FIG. 3 is a diagram for describing an example of recorded contents of conference information recorded according to the embodiment of the present invention.

FIG. 4 is a diagram for explaining an example of a data structure of recording data of conference information recorded according to the embodiment of the present invention.

FIG. 5 is a diagram showing a data structure of a directive word table for describing an outline of a directive word detection process according to the embodiment of the present invention.

FIG. 6 is a flowchart illustrating an example of a descriptive word detection process according to the embodiment of the present invention.

FIG. 7 is a diagram illustrating an instruction operation.

FIG. 8 is a flowchart illustrating an outline of an example of an instruction target detection process according to the embodiment of the present invention.

FIG. 9 is a diagram showing an example of a participant attribute table used for an example of a selection process of an instruction operation unit corresponding to an instruction word in the embodiment of the present invention.

FIG. 10 is a flowchart showing a part of an example of a flow of an instruction operation unit selecting process corresponding to an instruction word in the embodiment of the present invention.

FIG. 11 is a flowchart showing a continuation of the example of the flow of the instruction operation unit selection processing corresponding to the instruction word in the embodiment of the present invention.

FIG. 12 is a flowchart illustrating an outline of an example of a significance determination process of an instruction operation of an instruction operation unit according to the embodiment of the present invention.

13 is a diagram for explaining a significant movement amount of an instruction operation unit in the significance determination processing of FIG.

FIG. 14 is a flowchart illustrating an example of a process of determining the significance of the movement amount of the instruction operation unit according to the embodiment of the present invention.

FIG. 15 is a diagram for describing an example of a process for correcting an instruction operation start time of the instruction operation means according to the embodiment of the present invention.

FIG. 16 is a diagram illustrating an example of a data structure of a buffer that holds position coordinate data of the instruction operation unit according to the embodiment of the present invention.

FIG. 17 is a flowchart illustrating an example of a flow of an instruction operation start point correction process of the instruction operation means according to the embodiment of the present invention.

FIG. 18 is a diagram for describing an end determination condition relating to the movement amount of the instruction operation means in the embodiment of the present invention.

FIG. 19 is a diagram showing an end condition relating to time of the instruction operation means in the embodiment of the present invention.

FIG. 20 is a flowchart showing a flow of an instruction operation end determination process in the embodiment of the present invention.

FIG. 21 is a block diagram showing a system configuration diagram of a playback system of a conference information recording / playback apparatus according to an embodiment of the present invention.

FIG. 22 is an image diagram of an example of a playback screen according to the embodiment of the present invention.

FIG. 23 is an image diagram of an example of a playback screen according to the embodiment of the present invention.

FIG. 24 is a flowchart showing an outline of an example of an instruction target display control process in the embodiment of the present invention.

FIG. 25 is a flowchart illustrating an example of an instruction target display control process according to the embodiment of the present invention.

FIG. 26 is a diagram illustrating an overview of display of preview information and display of residual image information according to the embodiment of the present invention.

FIG. 27 is a diagram showing an image of a display screen of preview information, instruction operation, and afterimage information in the embodiment of the present invention.

FIG. 28 is a flowchart showing an example of a flow of a display process of preview information in the embodiment of the present invention.

FIG. 29 is a flowchart illustrating an example of the flow of a process of displaying afterimage information according to an embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Personal computer 2 ... Large display device 10-14 ... Conference participant 20-24 ... Pointing device (instruction operation means) 30-34 ... Microphone 60 ... Recording control part 61 ... Instruction word detection part 62 ... Instruction word detection notification part Reference numeral 63: instruction target detection unit 64: image capturing device 65: timer 66: file storage unit 67: instruction history recording unit 68: A / D conversion unit 70: recording area 71: link information file 72: audio data file 73: image Data file 74: Target file 400: Reproduction control unit 401: Timer 402: Display time determination unit 410: Voice information display control unit 411: Voice-waveform conversion unit 412: Directed word link display control unit 420: Directed target display control unit 421: Image display control unit 422: Instruction operation reproduction unit 423: Preview information display control unit 424: afterimage information display control unit 430: audio playback device 440: display device

Claims (12)

[Claims] 1. A method for recording audio information when a plurality of conference participants hold a conference while watching the contents of a common display screen, and displaying the contents of the display screen at an appropriate timing to supplement the audio information. In the method of recording together the image information including and the instruction operation information of any conference participant on the display screen as needed, when the conference participant speaks the instruction word, Detecting whether there is an instruction operation on the display screen of any conference participant who performs the instruction operation corresponding to the instruction word, at least image information including the instruction target, A conference information recording method, wherein information is recorded in association with an utterance portion of a descriptive word of information. 2. When an instruction operation corresponding to the instruction word is performed, information of an operation trajectory of the instruction operation corresponding to the instruction word is transmitted in addition to image information including the instruction target. 2. The conference information recording method according to claim 1, wherein the recording is performed in association with the audio information of the part. 3. A display device for displaying image information contributing to a conference, instruction operation means for a conference participant to indicate an arbitrary position on a screen of said display device, and said instruction operation means Position recognition means for recognizing the position on the screen designated by the above, voice information input means for receiving voice information input during the meeting, and voice information of the designated word from voice information from the voice information input means. Instruction word detection means for detecting; instruction operation detection means for detecting an instruction operation by the instruction operation means corresponding to the instruction word detected by the instruction word detection means; and a voice during the meeting from the voice information input means. When the instruction operation is detected by the instruction operation means corresponding to the instruction word, at least the image information including the instruction target of the detected instruction operation is recorded. The conference information recording apparatus characterized by comprising a recording means for recording in association with the voice information of the detected utterance portion of the instruction word in the instruction word detecting means. 4. The conference information recording apparatus according to claim 3, wherein the recording unit is configured to store the position information from the position recognition unit in addition to the image information including the pointing target detected by the pointing target detection unit. A conference information recording apparatus for recording information of a used instruction operation trajectory of said instruction operation means in association with voice information of an utterance part of said instruction word detected by said instruction word detection means. 5. The conference information recording apparatus according to claim 3, wherein a plurality of said instruction operation means are provided corresponding to a plurality of conference participants. A participant attribute determining means for determining the priority of each conference participant, and whether or not the amount of change in the indicated coordinate position on the display screen by the instruction operating means satisfies a defined significant moving amount condition, By comparing the instruction operation significance determining means for determining whether the instruction operation is significant or not and the amount of change in the indicated coordinate position on the display screen by the plurality of instruction operation means, the superiority of the instruction operation means Instruction operation means superiority determination means for determining, based on the determination result of the participant attribute determination means, the determination result of the instruction operation significance determination means, and the instruction operation means superiority determination result , A conference information recording apparatus, wherein an instruction operation corresponding to the instruction word by the instruction operation means is detected. 6. The conference information recording apparatus according to claim 4, further comprising: instruction history recording means for recording coordinate data of an instruction operation position for a predetermined period of time for each instruction operation means; The operation detecting means detects a start time of the instruction operation by correcting a time delay of a detection time point of the instruction word with respect to an utterance time of the instruction word using the coordinate data of the instruction history recording means. Conference information recording device. 7. The conference information recording apparatus according to claim 4, wherein the instruction operation detecting means terminates the instruction operation by the instruction operation means based on a magnitude of a change in an indicated coordinate position by the instruction operation means. End position determining means for determining, and end time determining means for determining the end of the instruction operation by the instruction operation means by monitoring an elapsed time from a point in time when the instruction operation is started; A notifying unit for notifying the detection, and detecting an end point of the instruction operation by the instruction operation unit based on the end position determination unit, the end time determination unit, and the notification unit. Information recording device. 8. A display device for displaying image information contributing to a conference, instruction operation means for a conference participant to indicate an arbitrary position on a screen of said display device, and said instruction operation means A position recognition means for recognizing a position on the screen designated by; a voice information input means for receiving voice information input of the speech of the conference participant; and voice information from the voice information input means, Instruction word detection means for detecting voice information; instruction operation detection means for detecting an instruction operation by the instruction operation means corresponding to the instruction word detected by the instruction word detection means; during a meeting from the audio information input means And when the instruction operation detecting means detects an instruction operation by the instruction operation means corresponding to the instruction word, at least, the image information including the instruction target, Recording means for recording in association with audio information of the utterance part of the instruction word detected by the sign word detection means; audio reproduction means for reproducing the recorded audio information; and, when reproducing the audio information, the instruction word A meeting information recording / reproducing apparatus, comprising: image information display means for displaying image information corresponding to the instruction word at the time of utterance. 9. The conference information recording / reproducing apparatus according to claim 8, wherein said recording means includes, in addition to image information including the pointing target detected by said pointing target detection means, position information from said position recognition means. The information of the instruction operation trajectory of the instruction operation means using is recorded in association with the voice information of the utterance portion of the instruction word detected by the instruction word detection means, by the information of the instruction operation trajectory, When sequentially reproducing the voice information, the instruction operation trajectory of the instruction operation means is displayed on the image information corresponding to the instruction word displayed at the time of utterance of the instruction word, and the instruction corresponding to the instruction word is displayed. A conference information recording / reproducing device for identifying an object. 10. Audio information during a conference and, as information supplementing the audio information, when a conference participant speaks a directive, an arbitrary conference participant corresponding to the directive is displayed on the display screen. When there is an instruction operation, the image information including the instruction target and the information of the trajectory of the instruction operation corresponding to the instruction word are used as the link information at the time of the utterance of the instruction word as the link information. A playback device of the conference information recorded in association with, a display device for displaying the image information and the operation trajectory of the instruction operation, on the screen of the display device, the change of the playback position of the audio information, First display control means for displaying on a screen display corresponding to the current reproduction position; and referring to the link information, utterance of an instruction word of a reproduction position in the future from the current reproduction position and its related You A display to predict the presence of image information, conference information reproducing apparatus characterized by comprising a second display control means to perform in connection with the change display of the playback position. 11. Audio information during a conference and, as information supplementing the audio information, when a conference participant speaks a directive, an arbitrary conference participant corresponding to the directive is displayed on the display screen. When there is an instruction operation, the image information including the instruction target and the information of the trajectory of the instruction operation corresponding to the instruction word are used as the link information at the time of the utterance of the instruction word as the link information. A playback device for meeting information recorded in association with a display device for displaying an operation trajectory of the image information and the instruction operation, and a display device for an instruction operation corresponding to the instruction word determined based on the link information. Means for calculating a predetermined time before a predetermined time from the start time, and displaying the corresponding image information on the screen of the display device from the time before the predetermined time, and Between, on the screen of the display means, the instruction at the beginning of the operation and means for displaying visual attention information, and characterized in that a foreseeable for instructions object meeting information reproducing apparatus. 12. The audio information during a conference and, as information supplementing the audio information, when a conference participant speaks a directive, an arbitrary conference participant corresponding to the directive is displayed on the display screen. When there is an instruction operation, the image information including the instruction target and the information of the trajectory of the instruction operation corresponding to the instruction word are used as the link information at the time of the utterance of the instruction word as the link information. A playback device for displaying the image information and the operation trajectory of the instruction operation, an audio playback unit for playing back the recorded audio information, and the audio information. At the time of reproduction of the instruction word, at the time of the utterance of the instruction word, image display control means for displaying image information corresponding to the instruction word on the screen of the display device; Instruction operation reproducing means for visually reproducing the corresponding instruction operation based on the information of the trajectory of the instruction operation, superimposed on the image information corresponding to the instruction word displayed in the step (c), and ending the reproduced instruction operation Means for specifying a time after a predetermined time has elapsed from a point in time; means for displaying afterimage information of the trajectory of the instruction operation for the predetermined time; and visual attention to an end point of the instruction operation. A conference information reproducing apparatus, comprising: means for displaying information; and means for deleting the display of the attention information after the lapse of the predetermined time.