JP4096132B2 - Time-series information specific section determination device, information storage device, and information storage / playback device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
This invention stores and accumulates information such as conversation voices in meetings and interviews, images of meetings and interview scenes, and conference memos and interview memos related to them, such as a conference record recording system and a coverage recording system. apparatus About.
[0002]
[Prior art]
Conventionally, recordings such as conferences, lectures, interviews, interviews, telephone and videophone conversations, TV images, surveillance camera images, etc. are recorded and played back on digital disks, digital still cameras, video tapes, semiconductor memory, etc. A device has been proposed. When information is accumulated using these devices, there is an advantage that it is possible to record audio and images as input information without leaking, compared with a method in which a recorder records only the main points of information to be recorded by writing. .
[0003]
These devices include those that record digital signals transmitted via a computer network on a storage medium, those that record analog input signals from a video camera or microphone on a storage medium as they are, and those that are encoded into digital signals. There are things to convert and record.
[0004]
However, there is a problem in that it is impossible to instantly search for a desired portion from recorded voice and images.
[0005]
In order to solve this problem, there has been proposed an apparatus for storing, recording, and reproducing audio signals or image signals that are sequentially input and user input information that is input by a recorder at any timing in association with user input using a pen or a keyboard. . If these devices are used, the recorder performs input with a pen or a keyboard so as to take notes, and if an audio signal or an image signal is recorded, the audio can be read later by referring to the input notes. It is possible to easily select and reproduce a portion where a signal or an image signal is desired to be reproduced.
[0006]
For example, in JP-A-7-182365 and JP-A-6-343146, audio signals or image signals are associated with user input information based on time stamps at the time of recording, and at the time of reproduction, An apparatus is described in which one of user input information displayed on the screen is designated to reproduce an audio signal or an image signal recorded at the time when the designated user input information is recorded.
[0007]
Japanese Patent Laid-Open No. 6-205151 discloses an input audio signal or an input image signal with an index added at the timing when it is detected that the user input has been interrupted for a certain period of time, and is displayed on the screen during playback. An apparatus is described in which one of specified specific user input information is designated to reproduce an audio signal or an image signal from an index portion corresponding to the designated user input information.
[0008]
However, the information storage device described in the above-mentioned Japanese Patent Laid-Open Nos. 7-182365 and 6-205151 has a mechanism for recording all input audio signals or image signals without compression. Therefore, there is a problem that it is difficult to record a long-time input audio signal or input image signal in a limited recording capacity. This is because, in general, when recording time-series data such as sequentially input audio signals or image signals over a long period of time, the necessary storage capacity becomes enormous.
[0009]
As a known method, a method of storing audio signals and image signals in a storage medium while always compressing them has been proposed. However, it is general that all input audio signals or image signals are stored at the same compression rate. It is. In the case of this method, it is not possible to record only important parts with high sound quality / high image quality to save storage capacity, and record a large amount of information that is unlikely to be referenced later, In spite of important information, there was a problem that it could not be recorded due to the storage capacity.
[0010]
For example, in the case where an interview scene is recorded for a long time using “Video for Windows” (“Microsoft Video for Windows 1.0 User's Guide” pp. 57-59, pp. 102-108), the storage capacity is increased. For the purpose of saving, it is assumed that the thinning compression rate is set so that only one frame is stored for 5 seconds. At this time, even if the recording person wants to reproduce the part that is important at the time of recording later, only one frame of image signal can be reproduced in 5 seconds. ), Talking, and subtle nuances cannot be reproduced. On the other hand, if all the input image signals are stored in 30 frames per second, the storage capacity becomes enormous as described above in order to store a long interview, which is very difficult to realize. is there.
[0011]
Therefore, in Japanese Patent Laid-Open No. 7-15519, when it is recognized that the free space of the storage medium is less than a certain amount, the free space of the storage medium is reduced by recompressing the already stored audio information. A voice information storage device to be secured is described.
[0012]
Further, in Japanese Patent Laid-Open No. 5-64144, when the usage amount (data storage amount) of the image storage medium exceeds a predetermined amount, the image information already stored is compressed in order from the old frame, An information storage device is described that attempts to save storage capacity by thinning out frames. By regarding the information stored later as important information, the previously stored information is overwritten with new input information, or the information stored earlier is increased in compression rate. It is a device that saves storage capacity.
[0013]
In addition, the moving image recording apparatus described in Japanese Patent Laid-Open No. 6-149902 performs automatic scene change detection, and considers that a longer scene is a more important scene. This is an apparatus that extracts in order from scenes having a high importance so as to be long. If only the scene included in the digest generated by the apparatus described in this publication is left and the scene not included in the digest is deleted, the storage capacity can be saved without losing important information.
[0014]
Further, there has been proposed an apparatus that selects information at the time of recording and records only information recognized as important or changes the compression rate. For example, Japanese Patent Application Laid-Open No. 6-343146 describes a method of recording a video signal for a predetermined time at a timing when a user input is made. Furthermore, some commercially available tape recorders have a silent section detection function that does not store voice in silent sections.
[0015]
Various systems have also been proposed in the past for retrieving important sections from time series information when all time series information such as image information and audio information is recorded as conference information.
[0016]
These conventional systems record time-series information such as audio information and video information such as conferences, while recording user input information such as pen input and keyboard input of conference participants and their input time, and later Thus, there is provided a mechanism for reproducing time series information related to each user input information by using the input time.
[0017]
For example, in Japanese Patent Application Laid-Open No. 6-343146, when an arbitrary portion of a pen input handwriting is pointed, a pop-up window displaying an audio waveform near the time when the pen input is performed is displayed. It is possible to control playback of audio information and image information. The following are examples of specific playback modes.
[0018]
{Circle around (1)} A certain time width tw is reproduced from the pen input time T. However, tw can be changed by the user. For example, the playback time is changed by sliding a bar displayed on the pop-up window.
[0019]
(2) Playback from the pen input time T to the end. However, buttons such as stop, rewind, and fast forward are provided on the pop-up window, and the user can operate at any timing.
[0020]
{Circle around (3)} Playback starts from a point slightly before the pen input time T. Reproduction is performed for a time width ta + tb from time T-ta to time T + tb. ta and tb may be determined in advance, or may be determined after the user slides a bar on the pop-up window.
[0021]
As described above, Japanese Patent Laid-Open No. 6-343146 provides a user with several reproduction methods when reproducing multimedia information corresponding to pen input.
[0022]
[Problems to be solved by the invention]
However, in the related art relating to the recording apparatus, for example, the apparatus described in Japanese Patent Application Laid-Open No. 7-15519 is an apparatus that recompresses a stored audio signal at the same compression ratio throughout and records the audio signal. There is a problem that it is impossible to record only high-quality sound by partially reducing the compression rate of only important parts of the content.
[0023]
Further, in an information storage device for storing and storing conferences, lectures, interviews, interviews, etc., as described in Japanese Patent Laid-Open No. 5-64144 or Japanese Patent Laid-Open No. 5-134907, new records are simply recorded as important information. If the old record is erased as unnecessary information, the record of important meetings and important interviews will be overwritten by new input information just because it was recorded first. was there. This is because, in general, it is not possible to determine the importance of the content of the meeting or the content of the coverage based only on the date and time when the conference or the coverage was held.
[0024]
In addition, with respect to the apparatus described in Japanese Patent Laid-Open No. 6-149902 that determines the importance of a scene based on the length of the scene, when a meeting or lecture is taken with an unmanned camera, the scene is changed by a cut change or a scene change. There is a problem that it is very difficult to cut and the length of the scene cannot be detected. In addition, if you are shooting a meeting or lecture, important statements may be included even in a short scene, so based on the length of the scene alone, There was a problem that the importance could not be determined.
[0025]
As a further problem, the above-described conventional devices do not take into account the input status such as the type of input device used by the user for input or the user's intention to input, so that depending on the input status when there is user input. There was a problem that proper selective recording was not possible.
[0026]
For example, if one of the conference participants writes the discussion process or conclusion on the shared electronic whiteboard device, and other conference participants take notes on their personal electronic notebook devices, the shared A writer who writes on the electronic whiteboard device tends to write related information while writing, or to write it on the shared electronic whiteboard device as soon as it is spoken by a meeting attendee other than the writer. Therefore, there is a feature that the time point at which the memo is written on the shared electronic whiteboard is close to the time point when the related information is spoken.
[0027]
On the other hand, when a conference participant writes on a personal electronic notebook device, he / she can input the hand memo after chewing the words once heard in his / her hand, or write the hand memo after replacing it with his / her own words. In many cases, there is a feature that a point of time when a memo is written on a personal electronic notebook device and a point of time when related information is spoken are relatively separated.
[0028]
As described above, it is considered that the timing and length of the corresponding voice information and image information sections change depending on the situation in which the user input is performed.
[0029]
However, this type of conventional device records a signal for a certain period of time at the time of user input, as in the case of the device described in Japanese Patent Laid-Open No. 6-343146, for example. The timing and length of recording an audio signal or image signal cannot be changed depending on the input status such as the type of user input device and the input intention, and therefore information can be recorded appropriately and selectively. There wasn't. In addition, there is a problem in that it is not possible to selectively reproduce a portion of information appropriately corresponding to a user input during reproduction.
[0030]
In addition, the above-described prior art relating to retrieval at the time of reproduction records multimedia information (time series information) such as voice and video, while recording user input information such as memo input and the input time thereof. This is to facilitate reproduction of time series information related to the input information.
[0031]
However, the time when the user inputs information and the time when the corresponding audio information and image information exist are generally different from each other, and the degree of deviation differs in each case. For this reason, in order to know the exact time interval of the time-series information corresponding to each user input information at the time of playback, the search is based on its own memory and intuition, and the memory becomes ambiguous / inaccurate. For example, even if the user input information and the corresponding position of the voice information and image information are close in time, there is a problem that it takes a lot of time for the search.
[0032]
In addition, the user's memory regarding the existing time interval of multimedia information such as appropriate audio information and image information corresponding to the user input information becomes thinner as time passes, and a long time is required for the search. In addition, when a third party other than the user who has input the user input information searches, not only does it further require time, but there is no certainty that it is the correct time interval position.
[0033]
Therefore, how to appropriately reproduce the multimedia information corresponding to the user input information according to the situation, or how to present appropriate information to help search for the multimedia information even if it cannot be reproduced, and In order to search for efficient information, it becomes a big problem to provide the user with information that ensures that the multimedia information corresponds to the input information.
[0034]
However, in the conventional playback method described above, multimedia information is played back from a certain time before the time when user input information is input, or information on a time interval near the time when user input information is input is displayed. This is a method of presenting a pop-up window and selecting a playback location.
[0035]
As described above, the time relationship between the input time of the user input information and the corresponding multimedia information section varies depending on the situation, and the appropriateness of the playback section and the display section varies depending on the situation.
[0036]
In the example shown in the embodiment of Japanese Patent Laid-Open No. 6-343146, there is no means for displaying only an appropriate section of the speech waveform displayed on the pop-up window, and in some cases The display section may be too short or too long. If the display section is too short, it is not only possible to specify an appropriate starting point for multimedia information on the display, but there is not enough information for the user to recall and infer the location of multimedia information. In addition, there is a problem that it takes a lot of time to search for multimedia information. In addition, when the display section is too long, the time section in which the multimedia information exists is displayed, but considering that the display area is finite, only unnecessary information is displayed. There is a problem that an efficient use of the display area results.
[0037]
The present invention solves the above-described problem, and detects an important period of an audio signal or an image signal in which a characteristic event occurs in the input audio signal or the input image signal in relation to the user input. In addition to being able to be identified, a large number of audio signals or image signals in this important period are stored in a limited storage medium, and even if the audio or image signals are not in the important period, they can be used for a long time with a small amount of data. The problem is to be able to memorize.
[0038]
Another object of the present invention is to make it possible to selectively and reliably reproduce or easily search an important part of an audio signal or an image signal related to a user input.
[0039]
[Means for Solving the Problems]
In order to solve the above-mentioned problem, the invention described in claim 1
Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each The user input information is received from a plurality of user input devices that output the user input information in a state where an input situation can be specified. User input information input means;
Input from the user input information input means Said User input information detecting means for detecting user input information;
The user input information detected by the user input information detection means Is provided with a function for specifying at least the input status of user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
The user input information detected by the user input information detection means and specified by the user input situation specifying means Said Based on the input status, a specific section of the audio information or image information from the time-series information input means is determined. And determining a time-series information section in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A specific section determining means;
A specific section determination device for time-series information comprising:
[0044]
Claims 2 In the invention of
Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers. The user input information is received from a plurality of user input devices that output the user input information in a state where an input situation can be specified. User input information input means;
Input from the user input information input means Said User input information detecting means for detecting user input information;
The user input information detected by the user input information detection means Is provided with a function for specifying at least the input status of user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
The user input information detected by the user input information detection means and specified by the user input situation specifying means Said Based on the input status, a specific section of the audio information or image information from the time-series information input means is determined. And determining a time-series information section in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A specific section determining means;
Time series information storage means for storing the audio information or the image information input from the time series information input means;
Means for compressing the amount of audio information or image information stored in the time-series information storage means, Special mention Fixed interval determination Determined by means A compression unit that compresses the audio information or the image information stored in the time-series information storage unit by changing a compression rate or a compression method in a section and another section;
An information storage device is provided.
[0045]
Claims 3 In the invention of
Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each The user input information is received from a plurality of user input devices that output the user input information in a state where an input situation can be specified. User input information input means;
Input from the user input information input means Said User input information detecting means for detecting user input information;
The user input information detected by the user input information detection means Is provided with a function for specifying at least the input status of user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
The user input information detected by the user input information detection means and specified by the user input situation specifying means Said Based on the input status, a specific section of the audio information or image information from the time-series information input means is determined. And determining a time-series information section in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A specific section determining means;
Time series information storage means for storing the audio information or the image information input from the time series information input means;
in front Special mention Fixed interval determination means Correspondence relationship storage means for storing correspondence information between the section information indicating the section determined in (1) and the storage position in the time-series information storage means of the audio information or the image information corresponding to the section information;
Means for compressing the data amount of the audio information or the image information stored in the time-series information storage means, and a section defined by the section information stored in the correspondence storage means and another section; A compression unit that compresses the audio information or the image information stored in the time-series information storage unit by changing a compression rate or a compression method;
An information storage device is provided.
[0048]
Claims 4 In the invention of
Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each The user input information is received from a plurality of user input devices that output the user input information in a state where an input situation can be specified. User input information input means;
Input from the user input information input means Said User input information detecting means for detecting user input information;
The user input information detected by the user input information detection means Is provided with a function for specifying at least the input status of user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
The user input information detected by the user input information detection means and specified by the user input situation specifying means Said Based on the input status, a specific section of the audio information or image information from the time-series information input means is determined. And determining a time-series information section in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A specific section determining means;
Temporary storage means for temporarily storing the audio information or image information;
Of the audio information or image information temporarily stored in the temporary storage means, Special mention Fixed interval determination Determined by means Time-series information storage means for storing audio information or image information of a section as information corresponding to user input information detected by the user input information detection means;
in front Special mention Fixed interval determination Determined by means Correspondence relation storage means for storing correspondence relation between section information of a section and a storage position in the temporary storage means of the audio information or image information corresponding to the section;
The means for compressing the data amount of the audio information or the image information stored in the time-series information storage means among the audio information or the image information stored in the temporary storage means, the correspondence relationship storage Before memorized in the means Special mention Fixed interval determination means A compression unit that compresses data of the audio information or the image information by changing a compression rate or a compression method in a section defined by
An information storage device is provided.
[0051]
Claims 5 In the invention of
Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each The user input information is received from a plurality of user input devices that output the user input information in a state where an input situation can be specified. User input information input means;
Input from the user input information input means Said User input information detecting means for detecting user input information;
The user input information detected by the user input information detection means Is provided with a function for specifying at least the input status of user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
The user input information detected by the user input information detection means and specified by the user input situation specifying means Said Based on the input status, a specific section of the audio information or image information from the time-series information input means is determined. And determining a time-series information section in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A specific section determining means;
Time series information storage means for storing the audio information or the image information input from the time series information input means;
Display means for displaying the user input information detected by the user input information detection means on a display screen;
For each of the user input information, information for specifying a display position of the user input information on the display screen, and the user input information, Special mention Fixed interval determination Determined by means Correspondence storage means for storing a correspondence relationship between the audio information or the image information corresponding to the section and the storage position in the time-series information storage means;
Designating means for designating a display portion related to the user input information displayed on the display screen;
The audio information or the image information of the section specified based on the storage position in the time series information storage means stored in the correspondence storage means specified by the user input information specified by the specification means is reproduced. Reproduction means;
An information storage device is provided.
[0054]
Claims 6 In the invention of
Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each The user input information is received from a plurality of user input devices that output the user input information in a state where an input situation can be specified. User input information input means;
Input from the user input information input means Said User input information detecting means for detecting user input information;
The user input information detected by the user input information detection means Is provided with a function for specifying at least the input status of user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
The user input information detected by the user input information detection means; Said Information about the input time of user input information, Said Specified by the user input status specifying means with respect to user input information Said User input information recording means for recording the input status;
Time-series information input means for inputting time-series information of audio information or image information;
Time series information recording means for recording the audio information or the image information input from the time series information input means;
Display means for displaying the user input information recorded in the user input information recording means;
Designation means for selecting and instructing one of the user input information displayed by the display means;
A specific section of the audio information or image information from the time-series information input unit based on at least information related to the input time and the input status of the user input information specified by the specifying unit Decide And determining a time-series information section in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A specific section determining means;
Time series information recording means Time-series information reproducing means for reproducing the time-series information corresponding to the specific section determined by the specific-section determining means among the time-series information recorded in
An information storage / playback apparatus is provided.
[0063]
Claims 7 In the invention of
Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each The user input information is received from a plurality of user input devices that output the user input information in a state where an input situation can be specified. User input information input means;
Input from the user input information input means Said User input information detecting means for detecting user input information;
The user input information detected by the user input information detection means Is provided with a function for specifying at least the input status of user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
The user input information detected by the user input information detection means, and information related to the input time of the user input information; The input status specified by the user input status specifying means with respect to the user input information; User input information recording means for recording
Time-series information input means for inputting time-series information of audio information or image information;
The audio information or the image information input from the time-series information input means Record Time-series information recording means,
Display means for displaying the user input information recorded in the user input information recording means;
Designation means for selecting and instructing one of the user input information displayed by the display means;
A specific section of the audio information or image information from the time-series information input unit is determined based on at least information related to the input time of the user input information specified by the specifying unit. And determining a time-series information section in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A specific section determining means;
By the specific section determining means Decision A specific section correction unit that corrects the determined section based on an input time of user input information immediately before the user input information selected and instructed by the specifying unit;
Time-series information reproducing means for reproducing the time-series information corresponding to the specific section determined by the specific-section determining means out of the time-series information recorded in the time-series information recording unit;
An information storage / playback apparatus is provided.
[0066]
[Action]
In the first aspect of the present invention, when the user inputs user user input information such as a memo input by the user input means, this is detected by the user input information detection means, and the user input is, for example, The input status such as whether it is input for personal use or input for common use is specified by the user input status specifying means. Then, according to the specified input situation, the time series information section corresponding to the user input information is specified by the time series information section specifying means.
[0067]
That is, a section having a timing and a length corresponding to an input situation when user input information is input can be specified, and an information section corresponding to a user input can be specified.
[0071]
That is, Claim 1 Invention In The user input device to which user input information is input can be specified to correspond to different information sections depending on whether the user input device is a shared input device or a personal input device. The time series information section can be specified.
[0072]
Claims of the above configuration 2 In the present invention, the section detected by the specific section detecting device for time-series information is compressed as, for example, an important information section with a compression rate or compression method different from other sections. In this case, different compression ratios or compression methods include cases where “compress” and “not compress”.
[0073]
For example, the voice information or the image information of the important section can be stored and stored with high quality, and the other section can store the voice information or the image information with a higher compression rate than that. Therefore, it is possible to reduce the used capacity of the time-series information storage means while keeping the important section information while maintaining high quality.
[0074]
In the invention of claim 7, section information indicating a section defined by the specific section detection device for time series information, and a storage position of the voice information or the image information corresponding to the section information in the time series information storage means Are stored in the correspondence storage means. For this reason, when compressing by the compression means, for example, the section information that should maintain high quality can be easily associated with the audio information or the image information of the section according to the stored contents of the correspondence relation storage means.
[0077]
Claim 4 In the present invention, not all of the audio information or the image information input from the time series information input means is stored as it is in the time series information storage means. That is, the audio information or the image information input from the time series information input unit is sequentially stored in the temporary storage unit. Then, the audio information or the image information stored in the temporary storage means includes a compression rate or a difference between a section determined by the specific section determination device for the time series information stored in the correspondence storage means and another section. The compression method is changed and compressed and stored in the time-series information storage unit. As a result, the storage capacity of the time-series information storage means can be smaller than when all the time-series information is stored as it is.
[0080]
Claim 5 In the information storage device according to the invention, when the user input information displayed on the display screen is instructed at the time of reproduction, the audio information or the image information of the section determined by the specific section determination device of the time series information corresponds. It is reproduced as sequence information. That is, audio information or image information in a section considered to be related to user input information is reproduced from the beginning to the end.
[0083]
Claim 6 In the information storage / reproduction apparatus, for example, in the case of a conference system, all of the time-series meeting information of audio information or image information is recorded in the time-series information recording means without any exception. Further, the user input information recording means includes the user input information detected by the user input information detection means, information related to the input time of the user input information, and the user input information specifying means related to the user input information. The input status is recorded.
[0084]
Then, the user input information recorded in the user input information recording means is displayed on the display screen, and when one of them is designated, the designated user input information recorded in the user input information recording means Information on the input time and the input status are taken out and passed to the specific section determining means.
[0085]
The specific section determining means corresponds to the specified user input information in the audio information or the image information recorded in the time series information recording means based on the input time information and the input situation of the user input information. Determine a specific section. Upon receiving this determination, the reproducing means extracts and reproduces the time series information of the specific section recorded in the time series information recording unit determined by the specific section determining means.
[0094]
Claims 7 In the information storage / playback apparatus of the present invention, all the time-series conference information of the audio information or the image information is recorded in the time-series information recording means. The user input information recording means records user input information detected by the user input information detection means and information related to the input time of the user input information.
[0095]
Then, the user input information recorded in the user input information recording means is displayed on the display screen, and when one of them is designated, the designated user input information recorded in the user input information recording means Information on the input time is taken out and passed to the specific section determining means.
[0096]
The specific section determining unit is configured to select a specific section corresponding to the specified user input information from the audio information or the image information recorded in the time series information recording unit based on the input time information of the user input information. decide.
[0097]
The specific section correcting means checks whether there is user input information immediately before the user input information selected and designated by the specifying means in the specific section determined by the specific section determining means. For example, the specific section determined by the specific section determination means is corrected based on the input time. If there is no previous user input information in the specific section, the specific section determined by the specific section determination means is not corrected.
[0098]
The reproduction means extracts time series information corresponding to the specific section determined by the specific section determination means or the specific section corrected by the specific section correction means from the time series information recording unit and reproduces it.
[0099]
And Claim 7 In the invention, the user input information specifying means is configured to input the user input information as follows: When the user input device to which the user input information is input is a shared input device, and when the user input device is a personal input device Is specified. Then, the specific section determining means determines the specific section related to the user input information instructed to be selected based on not only the input time of the user input information but also the input status of the user input information.
[0101]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
[0102]
[First embodiment of information storage apparatus]
The first embodiment is a case where the information storage device according to the present invention is applied to conference recording.
[0103]
In general, it is very unlikely to refer to a conference that took place a month ago compared to the possibility of referring to a conference that took place a few days ago. Keeping conference information such as video information that is less likely to be referenced in high quality is very inefficient in terms of saving memory space, and can be deleted or deleted at an appropriate time. It is desirable to reduce the amount of information by performing thinning compression.
[0104]
However, even in the case of old conference records, there are demands to reproduce the movements made by the speaker while speaking (gestures, etc.), talking, and subtle nuances. Therefore, it is required that the audio information or the image information in the important section where such a characteristic phenomenon occurs is stored in high quality.
[0105]
In this case, the problem is how to identify the important section of the time-series information of the audio information or the image information. In this embodiment, the detection time of user input information such as instruction input or memo input of a conference attendee (including a conference chairperson or facilitator), and the situation (input status) when the user input is made, The important section is determined based on the section that matches the predetermined condition for the audio information or the image information.
[0106]
In this example, there are two input situations, for example, when a user inputs to a shared input device such as a shared electronic whiteboard device and when a user inputs to a personal input device such as a personal electronic notebook device. It is assumed that the user input situation specifying means identifies the situation and distinguishes it.
[0107]
Further, in this example, the section that matches the condition is a speech section of one of the meeting attendees. Therefore, it is necessary to detect the alternation between the speech section and the speaker.
[0108]
In the first embodiment, audio information and video information are recorded for the conference, and when one month has elapsed from the recording time, only the important part of the recorded conference video is left, and the other parts are An example of performing an information amount reduction process (compression process) of erasing will be described. Further, in the first embodiment, as for the audio information, only the audio information of the important part in the recorded conference audio is left, and the other parts are erased.
[0109]
According to the first embodiment, as will be described later, since only the important scenes of the conference can be left and other portions can be erased, the amount of information to be stored and saved becomes very small. That is, in this example, it is assumed that the time when a conference participant takes a note is an important scene of the conference. By capturing this important scene, only the area around the time when the meeting participant took a note is saved, and the rest is erased, thereby greatly reducing the amount of information required to save the meeting video and meeting audio. be able to.
[0110]
FIG. 2 shows a meeting scene in this embodiment. Reference numeral 20 denotes an electronic conference apparatus constituting an information storage / reproduction apparatus. This electronic conference device 20 also has a function as a shared whiteboard device, and can be freely written on the projector screen using the electronic pen 23 with the same feeling as writing on the whiteboard with a marker pen. The process of discussion and conclusions can be stored and stored as an electronic file. For this reason, the electronic conference apparatus 20 incorporates a personal computer (not shown).
[0111]
Also, the attendees of the conference (including the conference presenter and facilitator) are equipped with a small computer 30 that functions as a personal electronic notebook device, and personal memos and the like are stored in the small computer during the meeting. Input can be made.
[0112]
In this embodiment, the computer built in the electronic conference apparatus 20 and the small computer 30 possessed by each individual are connected via a wireless network, and user input information input from a plurality of small computers 30 User input device information to be described later is transmitted to a computer built in the electronic conference device 20. Therefore, although not shown, each small computer 30 and the electronic conference device 20 are each provided with a wireless transmission / reception unit.
[0113]
In the following description, when the electronic conference device 20 is used as a shared input device, this is referred to as a shared whiteboard device 20, and the small computer is referred to as a personal electronic notebook device 30.
[0114]
As shown in FIG. 2, the electronic conference device 20 includes a display screen 21. The display method of the image on the display screen 21 is a so-called projection method in which a screen is projected and displayed from the back. Further, the electronic conference device 20 has a function of detecting the contact input coordinate position of the electronic pen 23 on the display screen 21, and the detected contact input coordinate position is used as user input information from the shared whiteboard device. Are input to the personal computer and stored and stored in the storage medium 27.
[0115]
As will be described later, the personal computer of the electronic conference apparatus 20 uses the electronic pen 23 to input user input information (user input information from the shared whiteboard apparatus) on the display screen 21 using the electronic pen 23. It is detected and stored separately from the user input information from.
[0116]
In addition, the electronic conference device 20 includes an audio input terminal and an image input terminal for inputting time-series information such as audio information and image information in a conference. In this embodiment, a microphone 25 is assigned to each of a plurality of conference attendees 29, and a speech signal of the speech of the conference attendant 29 collected by each microphone 25 is temporarily sent to the speech signal analyzer 26. Entered. The output terminal of the audio signal analyzer 26 is connected to the audio input terminal of the electronic conference apparatus 20.
[0117]
The audio signal analyzer 26 analyzes audio signals input from the plurality of microphones 25, identifies which microphone the input audio signal is input from, and sends the identification result to the electronic conference apparatus 20 together with the audio signal. Output.
[0118]
In addition, a paper document photographed by the video camera 24 and an image signal of the meeting scenery are input to the image input terminal of the electronic conference apparatus 20 and displayed on the display screen 21 of the electronic conference apparatus 20.
[0119]
The time-series information that is sequentially input may be an analog signal input from a video camera / microphone / video deck / tape recorder / sensor or the like, or a digital signal obtained by encoding the analog signal. Further, it may be a digital signal input through a computer network / computer bus. That is, any information that is sequentially input with the passage of time corresponds to time-series information in the present invention.
[0120]
On the display screen 21 of the electronic conference device 20, an image based on image information from the video camera 24 connected to the image input terminal of the electronic conference device 20 and an electronic document input through the personal computer of the electronic conference device 20 Are displayed as separate window images 22A and 22B, respectively, as shown in FIG. The image information displayed on the display screen 21 is also stored and accumulated in association with the user input information and voice information.
[0121]
In addition, the personal computer built in the electronic conference apparatus 20 is functionally provided with an information compression / storage apparatus. The information compression / accumulation device stores and accumulates the user input information, image information from the video camera 24 that captures a meeting scene, and audio information from the microphone 25 in a storage medium 27, and stores the information in the storage medium 27. It has a function of compressing the stored audio or image information as will be described later.
[0122]
The image information stored in the storage medium 27 is displayed on the display screen 21 in response to a playback request from the user, and the audio information stored in the storage medium 27 is played back from the speaker 28. The electronic conference device 20 also includes a playback unit for this purpose. The reproduction unit is configured to record audio information recorded at a time when the specified user input information is input when the user specifies arbitrary user input information from the user input information displayed on the display screen 21. Alternatively, the image information is read from the storage medium 27 and reproduced.
[0123]
When the user specifies arbitrary user input information, the image displayed on the display screen 21 at the time when the specified user input information is input is read out from the storage medium 27 and reproduced to be displayed on the display screen. 21 may be displayed.
[0124]
The electronic conferencing apparatus 20 is connected to a network by ISDN, for example, via a personal computer, so that audio information and image information in the course of the conference can be shared at the same time between remote locations, as if the conference was held in the same room. It is also possible to realize such an environment.
[0125]
FIG. 1 is a block diagram showing a functional part as an information storage / reproduction device of the electronic conference device of this embodiment. That is, the information storage device of this embodiment has a voice information input unit 1, an image information input unit 2, a condition matching section detection unit 3, a time series information storage unit 4, and a user input information input unit 5 with respect to the system bus. , User input information detection unit 6, user input device identification unit 7, important section determination unit 8, correspondence storage unit 9, compression unit 10, time information storage unit 11, reproduction unit 12, reproduction designation unit 13, display unit 14, The control unit 15 is configured to be connected to each other. In the case of this example, the output terminal of the voice information input unit 1 is also connected to the condition matching section detection unit 3.
[0126]
Each unit in FIG. 1 may be configured as a separate block, or one block may be configured to include several units. Moreover, one part may be divided into several blocks and mounted.
[0127]
The voice information input unit 1 receives a voice signal from the microphone 25, converts it into a digital voice signal, sends it to the system bus, and sends it to the condition matching section detection unit 3.
[0128]
The image information input unit 2 receives an image signal from the video camera 24. If the image signal from the video camera 24 is a digital signal, it is received and sent to the system bus. If the input image signal is not a digital signal, the image information input unit 2 converts the input image signal into a digital image signal and outputs it to the system bus.
[0129]
The condition matching section detection unit 3 monitors the digital voice signal from the voice information input unit 1 and detects a voice section that matches a predetermined condition. In this example, a condition matching section is detected on the condition that there is an input of an audio signal of a predetermined level or higher and that a change of speaker is detected from the input audio signal. In the condition matching section detection unit 3, the voice signal analyzer 26 and a part of the electronic conference apparatus 20 play the roles.
[0130]
As a method for detecting the presence / absence of an audio signal of a predetermined level or higher, as shown in FIG. 3, the condition matching section detection unit 3 detects that the input audio level is higher than the predetermined level and detects a speaker. And a detection function for recognizing the end point of the speaker's speech by detecting that the voice level is below a predetermined threshold level.
[0131]
However, as shown in FIG. 3, if the speech level change point F101 itself at which the speech level intersects the threshold level is the speech start point or end point, the first part and the last part of the speech are included in the condition matching section. Therefore, when the speech level changes from the high level to the low level, the time point F100 before the change point F101 when the audio level changes from the low level to the high level is set as the speech start point, and the audio signal level changes from the high level to the low level. The point of time F102 after a certain time T2 from the change point F101 of the above is defined as the speech end point.
[0132]
In this embodiment, the sound level at a certain time is a value obtained by smoothing the sound level before and after the time, for example, an average value of instantaneous sound levels for 2 seconds before and after the certain time.
[0133]
In this embodiment, as shown in FIG. 2, a microphone 25 is installed for each speaker, and the voice signal analyzer 26 compares the voice input level from the microphone of each speaker. 26 identifies the speaker that sent the input voice signal. Then, the change of the speaker is detected by specifying the speaker.
[0134]
In addition to this, as a method for specifying a speaker, a speaker may be specified from the characteristics of an audio signal (such as a voiceprint), or the speaker may be specified from a face or mouth movement based on image information. . In that case, it is not necessary to provide a plurality of microphones corresponding to all the meeting attendees, and one or a plurality of microphones smaller than the number of meeting attendees may be used. It is also possible to install a plurality of microphones, analyze the phase difference of the audio signals input from these microphones, detect the position of the sound source, and specify the speaker.
[0135]
The time-series information storage unit 4 stores the audio information input from the audio information input unit 1 and the image information input from the image information input unit 2 as digital information. As a storage device for storage information, a disk device, a magnetic tape device, or a semiconductor memory device is used.
[0136]
The user input information input unit 5 receives user input information from the shared electronic whiteboard device and also receives user input information from the personal electronic notebook device. As described above, in this embodiment, user input information from the personal electronic notebook device 30 is wirelessly communicated, and therefore this user input information input unit includes a wireless reception unit for this purpose.
[0137]
In this embodiment, the user input information from the shared electronic whiteboard device is added with identification information indicating that it is a shared input device in its header part. On the other hand, the discrimination information is not added to the header portion of the user input information from the personal electronic notebook device 30. Based on the presence or absence of the discrimination information in the header part, it can be discriminated whether the input device is a shared input device or a personal input device.
[0138]
The user input information from the shared electronic whiteboard device is given identification information indicating that it is a shared input device, and the header of the user input information from the personal electronic notebook device 30 It is of course possible to discriminate both by adding identification information indicating the above.
[0139]
The user input information detection unit 6 detects user input information from the user input information input unit 5 and transfers the detected user input information to the correspondence storage unit 9 and the display unit 14.
[0140]
The user input information includes display position information. The user input information detection unit 6 also outputs information at the time of detection of user input information to the system bus.
[0141]
The user input device specifying unit 7 constitutes an example of a user input situation specifying unit, and whether the user input information detected by the user input information detecting unit 6 is from a shared electronic whiteboard device or a personal electronic Whether it is from the notebook device 30 is specified from the header part included in the user input information. Then, the user input device information of the user input device identification result is output to the system bus.
[0142]
The important section determination unit 8 constitutes a specific section determination unit that specifies a specific section as described later in the time series information of audio information and image information as an important section. This is a partial function of the relationship storage unit 9. The important section determination unit 8 is configured to detect the time point when the user input information is detected from the user input information detection unit 6, the user input device information from the user input device specifying unit 7, and the condition matching section from the condition matching section detection unit 3. Based on the information, for the audio information and the image information, a specific section related to the user input information, that is, an important section is determined. A detailed example of how to determine this important section will be described later.
[0143]
The correspondence storage unit 9 includes user input information input by the user using, for example, the electronic pen 23, and information for specifying the display position of the user input information on the screen (for example, absolute coordinates or relative coordinates on the XY coordinate axes). And the storage address in the time-series information storage unit 4 of the voice information or the image information input in the important section determined by the important section determination unit 8 are stored in association with each other.
[0144]
In this embodiment, as the information of the important section, the correspondence storage unit 9 stores the start address and the end address of the important section including the time point when the user input information is input. That is, the correspondence relationship storage unit 9 sets a section specified by each user input information, condition matching section information, and user input device information as an important section, and the time-series information of the audio information or the image information of the important section. The storage address in the storage unit 4 and each user input information are stored in association with each other. The storage device of the correspondence storage unit 9 is also composed of, for example, a disk storage device or a semiconductor memory device.
[0145]
In this embodiment, the compression unit 10 performs a data thinning process on audio information and image information stored in the time-series information storage unit 4. In this case, the compression unit 10 is based on the important section information determined by the important section determination unit 8 of the correspondence storage unit 9 based on the timing when the user input information is input, the condition matching section information, and the user input device information. Thus, the audio information and the image information in the sections other than the important section can be deleted.
[0146]
The time information storage unit 11 is used to store the time when the input audio signal and image signal start recording in the time-series information storage unit 4, and is configured by, for example, a disk storage medium or a semiconductor memory.
[0147]
Further, the time information storage unit 11 has a function of measuring an elapsed time from the recording start time. For this reason, the time information storage unit 11 is supplied with current time information from a clock circuit unit (not shown). In this embodiment, the time information storage unit 11 uses the compression unit 10 to store the image of the time series information storage unit 4 when the elapsed time from the recording start time is equal to or longer than a predetermined time. A compression trigger timing signal that triggers the start of compression of information as described above is output.
[0148]
As described above, the reproduction unit 12 is a functional unit that reproduces the audio signal and the image signal stored in the time-series information storage unit 4.
[0149]
The reproduction designating unit 13 is used when the user input information, audio information, and image information stored and stored are designated and reproduced. As will be described later, when the user designates through the tablet from the user input information displayed on the display screen 21 or the personal electronic notebook device 30, the reproduction designation unit 13 reproduces the designated portion as a reproduction part. Is detected.
[0150]
The display unit 14 is a projection-type display device portion including the display screen 21 described above. In this example, the pen / tablet integrated input / output device is also used. For this reason, the display unit 14 includes a display device portion of the personal electronic notebook device 30 owned by the user. Therefore, the reproduction designation unit 13 also includes reproduction designation through the display screen 21 of the electronic conference device 20 and reproduction designation through the display screen of the personal electronic notebook device 30. In addition, as this display part 14, it can also be comprised with a CRT monitor and a liquid crystal monitor, for example.
[0151]
The user input information detection unit 6 of the electronic conference apparatus is configured by, for example, a tablet attached to the display screen 21, detects a user input by the electronic pen 23, and outputs the pen handwriting information as user input information. At this time, a pen locus corresponding to the pen handwriting information is displayed on the display screen 21 of the display unit 14.
[0152]
Since the user input information at this time includes identification information indicating that it is user input information of the shared input device, the user input device specifying unit 7 detects this and outputs the user input device information. .
[0153]
Similarly, pen handwriting information input from the personal electronic notebook device 30 at hand is displayed on the display device section of the personal electronic notebook device 30 that has been input. Furthermore, the personal electronic notebook device 30 that has detected the input from the user transmits user input information including the identification information to the electronic conference device 20. The user input information detection unit 6 receives and detects user input information from the personal electronic notebook device 30. Further, the user input device specifying unit 7 detects identification information included in the user input information and outputs user input device information.
[0154]
As user input information, in addition to handwriting or graphics (objects such as lines, squares, circles, etc.) from a pen (or mouse / trackball / touch panel), code information obtained by character recognition of handwriting data, or from a keyboard Code information may be used. Also, editing information indicating that the displayed user input information has been moved / copied / deleted, information indicating that the page has been switched, information indicating that the shutter of the still camera has been pressed, information indicating that the user has been sitting on a chair with a sensor, User input information such as information that a virtual eraser has been used may not be displayed. In this case, a predetermined indicia indicating the existence of the user input information is displayed on the display unit 3. That is, any input that a user makes to a device having calculation processing capability while input time-series information (for example, an audio signal or an image signal) exists corresponds to the user input information referred to in the present invention. To do.
[0155]
[Determination of important sections]
The important section determination unit 8 of the correspondence relationship storage unit 9 generates voice information from the timing when the user input information is detected, the specified user input device information, and the condition matching section detected by the condition matching section detection unit 3. And determining important intervals for image information. In this determination, the following points are considered in this embodiment.
[0156]
As described above, a writer who writes on a shared electronic whiteboard apparatus often speaks related information while writing. In addition, when a writer of the shared electronic whiteboard writes down what is said by a meeting attendee other than the writer, if the related information is spoken, it tends to be immediately written on the shared electronic whiteboard device. Therefore, there is a feature that the time point at which the memo is written on the shared electronic whiteboard is close to the time point when the related information is spoken.
[0157]
On the other hand, when a conference participant writes on a personal electronic notebook device, he / she can input the hand memo after chewing the words he / she hears in himself / herself, or write the hand memo after replacing it with his / her own words. Many. Accordingly, there is a feature that the time point at which the memo is written on the personal electronic notebook device and the time point when the related information is spoken are relatively separated.
[0158]
According to the results of experiments conducted by the inventors of the present application, for example, in the case of memo writing as a user input, the start point of the speech necessary for the memo writing input is included in the determined important section In order to make the establishment 95% (risk rate 5%) or more, when writing on a personal electronic notebook device, it is necessary to start recording at least 200 seconds before the time of writing. It was found that it is necessary to start recording at least 130 seconds before the time of writing.
[0159]
In addition, in order to make the end point of the speech necessary for the memo writing input included in the determined important section 95% (risk rate 5%) or more, when writing on a personal electronic notebook device It was found that recording had to be continued until at least 180 seconds after writing, while when writing on the shared electronic whiteboard device, it was necessary to continue recording until at least 70 seconds after writing.
[0160]
In consideration of the above, the important section determination unit 8 of this embodiment determines the important section from the time series information of the audio information and the image information as follows.
[0161]
That is, in FIG. 4, the identification of the important section when the handwriting input is made at the time tp by the input pen as the user input information on the display screen 21 of the shared electronic whiteboard device 20 or the display screen of the personal electronic notebook device 30 is as follows. It is made as follows.
[0162]
First, a time point t1 that goes back a predetermined time from the time point tp when the input of user input information is detected is specified, and a speaker change time point ts (t1) that is a time point that goes back further than this time point t1. The start point ts) of the condition matching section closest to is the beginning of the important section IP. In this case, as shown in the table of FIG. 5, if the user input is writing on the shared electronic whiteboard device, the time interval (tp−t1) between the time tp when the user input is detected and the time t1. Is 130 seconds, and the time t1 is determined so that the time interval (tp-t1) is 200 seconds if writing is performed on a personal electronic notebook device.
[0163]
Next, a time point t2 after a predetermined time from the time point tp when the input of the user input information is detected is specified, and the speaker change time point te (t2) that is a time point after the time point t2 and closest to the time point t2. The end point te) of the closest condition matching section is the end of the important section IP. In this case, as shown in the table of FIG. 5, if the user input is writing on the shared electronic whiteboard device, the time interval (t2-tp) between the time tp when the user input is detected and the time t2. Is 70 seconds, and if the writing is on the personal electronic notebook device, the time t2 is determined so that the time interval (t2-tp) is 180 seconds.
[0164]
Further, for example, when the sections to be determined to be important overlap each other due to the fact that the input from the user is continuously performed without a sufficient interval, the determination is made by connecting the sections.
[0165]
For example, as shown in FIG. 6, a case is considered where user input from the shared electronic whiteboard device is made at time tp1, and user input from the personal electronic notebook device 30 is made at time tp2 close to this. In this case, since the important interval determined for the user input time point tp1 and the important interval determined for the user input time point tp2 overlap in time, the time point tp2 from the start point of the important interval determined for the time point tp1 Up to the end point of the important section determined for is determined as the important section IP to be obtained.
[0166]
That is, in FIG. 6, a time point t1 that goes back a predetermined time before the user input time point tp1 from the shared electronic whiteboard device is specified, and the speaker change that is the time point that goes back further than this time point t1 and is closest to this time point t1. Let time ts be the beginning of an important interval. In this case, since the writing is on the shared electronic whiteboard device, as shown in the table of FIG. 7, the time interval (tp1-t1) between the time tp1 when the user input is detected and the time t1 is 130 seconds. A time point t1 is determined.
[0167]
Also, a time point t2 that is a predetermined time after the user input time point tp2 from the personal electronic notebook device is specified, and a speaker change time point te that is later than the time point t2 and is closest to the time point t2 is an important section. The end of In this case, since the writing is on the personal electronic notebook device, as shown in the table of FIG. 7, the time interval (t2-tp2) between the time tp2 when the user input is detected and the time t2 is 180 seconds. To time t2. In FIG. 6, WP indicates a section in which important sections determined for individual user inputs overlap.
[0168]
As shown in FIG. 6, when user input information input by a plurality of user input devices is combined and detected by the user input information detection unit 6, user input information from a plurality of users is combined and user input information is combined. When the detection unit 6 detects the voice information or the image information, the voice information or the image information is stored individually in the apparatus by storing the input voice information or the image information in the same time-series information storage unit 4. This eliminates the need for storage capacity.
[0169]
[Operation at the time of recording in the first embodiment of the information storage device]
Next, an operation at the time of recording in the electronic conference apparatus 20 as an example of the information storage apparatus having the above configuration will be described. FIG. 8 is a diagram for explaining the recording operation in this embodiment together with the flow of various information and the output flow of each unit at that time.
[0170]
When the conference is started and the audio signal from the microphone 25 and the image signal from the camera 24 are supplied to the electronic conference device 20, the audio signal and the image signal are sequentially stored in the time-series information storage unit 4. The audio signal is also input to the condition matching section detection unit 3.
[0171]
In this case, the audio signal analyzer 26 functions as the condition matching section detection unit 3. In other words, as described above, the microphone 25 is installed for each speaker, and the voice input level from the microphone of each speaker is compared with the threshold level in the voice signal analyzer 26, so that the voice signal analyzer At 26, the speaker that sent the input voice signal is identified and the start and end points of one utterance of one speaker are detected. In this embodiment, one utterance section of one speaker is a condition matching section. Information on the condition matching section is supplied to the important section determination unit 8 of the correspondence storage unit 9.
[0172]
FIG. 9 is a flowchart for explaining the operation of the condition matching section detection unit 3.
[0173]
When the voice signal from the voice information input unit 1 is supplied as a digital signal to the condition matching zone detection unit 3, the above-described speech zone detection and speaker identification are performed in step S100. As described above, the speaker identification method is performed by comparing the voice input level from the microphone 25 installed for each speaker by the voice signal analyzer 26.
[0174]
After this step S100, in step S101, it is recognized whether or not the speaker has changed, and if the change of speaker is detected, the process proceeds to step S102, and the condition matching interval specified in step S101 is selected as the important interval. It outputs to the determination part 8, and returns to step S100 after that, and the detection of a new condition matching area is started. Also, in step S101, when it is recognized that the speaker has not changed, the process returns to step S100 and detection of a new condition matching section is started.
[0175]
On the other hand, when the user input information detection unit 6 detects an input of pen handwriting information (user input information), the detected pen handwriting information is displayed on the display unit 14. The detected pen handwriting information is supplied to the correspondence storage unit 9 and used as one of the elements for determining the important section by the important section determination unit 8 and is stored and accumulated.
[0176]
FIG. 10 is a flowchart for explaining the operation of the user input information detection unit 6. The user input information detection unit 6 detects a coordinate position where the electronic pen 23 is in contact with the display screen 21 as the user input information, or a user having a coordinate position from the personal electronic notebook device 30 that is in contact with the display screen. The input information is detected (step S200), and the detected user input information is sequentially output and displayed on the display unit 14 and is also output to the correspondence storage unit 9, and the detected user is stored in the correspondence storage unit 9. The input information is stored (step S201).
[0177]
Further, based on the identification information of the user input device in the user input information from the user input information detection unit 6, the user input device specifying unit 7 determines whether the user input information is from the shared whiteboard device 20 or a personal electronic notebook. User input device information indicating whether it is from the device 30 is generated. This user input device information is supplied to the important section determining unit 8 of the correspondence storage 9 and used as one of the elements for determining the important section.
[0178]
FIG. 11 is a flowchart illustrating the operation of the user input device specifying unit 7 in this embodiment. As described above, in order to be able to determine whether the user input information is information input from the shared electronic whiteboard device 20 or information input from the personal electronic notebook device 30, the shared electronic whiteboard device 20 Adds the shared whiteboard device identification information to the header of the user input information.
[0179]
In other words, shared whiteboard device identification information indicating that it has been input from the shared electronic whiteboard device is added to the header of the user input information output from the shared electronic whiteboard device, while it is input from the personal electronic notebook device. The shared whiteboard device identification information is not added to the header of the user input information. Thereby, both input devices can be easily discriminated.
[0180]
As shown in FIG. 11, when there is an input from the user input information detection unit 6, this is detected in step S210, and the process proceeds to step S211. In step S211, it is detected whether or not the shared whiteboard device determination information is added to the header of the user input information from the user input information detection unit 6. If the shared whiteboard device determination information is not added, In step S212, user input device information that is input from the personal electronic notebook device 30 is supplied to the correspondence storage unit 9. On the other hand, if it is determined in step S211 that the shared whiteboard device identification information is added, the process proceeds to step S213, and the user input device information that is input from the shared electronic whiteboard device 20 is associated with the corresponding relationship. The data is supplied to the storage unit 9.
[0181]
In addition to the shared electronic whiteboard device 20, the device that generates the shared whiteboard device determination information may be an electronic pen 23 for writing to the shared electronic whiteboard device 20.
[0182]
As described with reference to FIGS. 4 to 7, the important section determination unit 8 of the correspondence storage unit 9 uses the above user input information, condition matching section information, and user input device information as described above. Determine important intervals for time series information.
[0183]
FIG. 12 shows an important section based on the pen handwriting input information that is the detection result of the user input information detection unit 6, the detection result of the condition matching section detection unit 3, and the detection result of the user input device of the user input device specifying unit 7. It is a figure which matches and demonstrates the important area defined by the determination part 8, and the memory state (state before performing a compression process) of the time series memory | storage part 4 which memorize | stored image information and audio | voice information. The correspondence information of these pieces of information is stored in the correspondence storage unit 9 as described above.
[0184]
As described above, the important section determination unit 8 of the correspondence storage unit 9 stores the user input information in the section determined from the detection result of the condition matching section detection unit 3 and the detection result of the user input device specifying unit 7. A section including the input time point is stored as an important section of input voice information or image information. That is, even if it is a section detected by the condition matching section detection unit 3, if no user input information is detected within the section, it is not recognized as an important section.
[0185]
And regarding image information and audio | voice information, from the time ts to the time te including the input time of the pen handwriting as user input information is specified as an important area. The image information and audio information of the important section are not deleted from the time-series information storage unit 4 when the compression process described later is performed, but the image information and audio information of the section other than the important section are subjected to the compression process described later. Sometimes erased.
[0186]
FIG. 13 is an example of correspondence information stored in the correspondence storage unit 9. As shown in FIG. 13, as pen handwriting information input by the user, an identifier for uniquely specifying user input information and information for specifying the type of user input information are stored. As display positions on the display screen 21, three XY coordinates (20, 30) (30, 40) (60, 10) and nil indicating the end of the coordinate information are stored.
[0187]
In addition, as storage addresses on the time-series information storage unit 4, an address as (start point address) and an address ae (end point address) determined by the important section determination unit 8 are stored.
[0188]
Finally, user input device information is stored to indicate whether the user input information is input from a shared input device or input from a personal input device.
[0189]
What is stored in the correspondence storage unit 9 is not pen handwriting information, display coordinates, and storage address itself, but may be specific information for identifying them, and the data storage structure of the correspondence storage unit 9 is a table. Instead of a format, it may be composed of other storage structures such as a list structure.
[0190]
In addition, as in this example, if a set of coordinate point sequences expressing multiple lines or the like is stored as one pen handwriting information, if any of the coordinate point sequences is specified during reproduction, The pen handwriting information corresponding to the coordinate point sequence can be specified, and the voice or image can be reproduced from the same storage address.
[0191]
FIG. 14 and FIG. 15 are flowcharts for explaining the operation of the correspondence relationship storage unit 9, but the steps involved in the recording operation described above are steps S300 to S304. Steps S305 and S306 are parts related to the operation at the time of compression described later, and steps S307 to S310 in FIG. 15 are parts related to the operation at the time of reproduction described later.
[0192]
That is, at the time of recording, in step S300, it is detected whether or not information indicating the condition matching section is input from the condition matching section detection unit 3. If the input of the condition matching section is not detected, The process returns to step S300 via S305 and S307, and the presence / absence of input of information indicating a condition matching section is detected.
[0193]
In step S300, when the input of the condition matching section from the condition matching section detecting unit 3 is detected, the process proceeds to step S301. In step S301, it is determined whether or not user input information is detected in the condition matching section. If it is determined that user input information is not detected in the condition matching section, step S305 and step S307 are performed. Returning to S300, the presence or absence of input of information indicating the condition matching section is detected again.
[0194]
On the other hand, if it is determined in step S301 that user input information is detected within the condition matching section, the process proceeds to step S302. In step S302, the important section is determined as described above. This step S302 is a step executed by the important section determination unit 8. In step S303, the time-series information is acquired in order to obtain the storage address in the time-series information storage unit 4 of the voice information or the image information stored in the time-series information storage unit 4 corresponding to the important section. Information indicating the important section and a storage address inquiry request are output to the storage unit 4 and a response is awaited.
[0195]
When a response from the time-series information storage unit 4 is returned, the process proceeds to step S304, in which the user input information, the display position of the user input information on the display unit 14, and the audio information and the image information that are important sections are displayed. A storage address (storage address of an important section) in the sequence information storage unit 4 and user input device information are stored in association with each other. The important section information of the voice information and the image information stored in the time series information storage unit 4 is thereby stored in correspondence with the user input information, the condition matching section information, and the user input device information. Become.
[0196]
After step S304, the process returns to step S300 via steps S305 and S307, and the presence or absence of input of information indicating the next condition matching section is detected.
[0197]
Next, the operation of the corresponding time-series information storage unit 4 at this time will be described with reference to the flowchart of FIG. Steps S400 to S404 of this processing routine are portions for operations during recording, and steps S405 and S406 are portions for operations during reproduction.
[0198]
That is, the operation at the time of recording of the time-series information storage unit 4 will be described with reference to FIG. 16. First, when this recording operation is started, the time-series information storage unit 4 stores audio information and image information in step S400. The start time is output to the time information storage unit 11 and recorded. Next, step S401 and step S402 are sequentially performed to receive input image information and sound information and store them sequentially.
[0199]
In the next step S403, it is determined whether or not a request for a storage address corresponding to the important section has arrived from the correspondence relationship storage unit 9. If it is detected that the request has arrived, the process proceeds to step S404. In step S404, the storage address of the voice information and the image information corresponding to the important section is returned to the correspondence storage unit 9.
[0200]
After it is determined in step S403 that a request for a storage address corresponding to the important section has not arrived, and after step S404, the process returns to step S401 via step S405 to continue storing image information and audio information. .
[0201]
The time information storage unit 11 receives the information of the storage start time by the process in the step S400 of the time series information storage unit 4, and stores the storage start time.
[0202]
FIG. 17 is a flowchart for explaining the operation of the time information storage unit 11, and FIG. 18 is a diagram for explaining the storage structure of the time information storage unit 11. In FIG. 17, steps S500 and S501 are processing at the time of recording. In step S500, the storage start time of audio information and image information supplied from the time-series information storage unit 4 is detected, and this storage is performed in step S501. The start time is stored in the time information storage unit 11.
[0203]
As will be described later, the time information storage unit 11 is configured such that the elapsed time (that is, the information storage time) from when the audio information and the image information are recorded in the time series information storage unit 4 is equal to or longer than a predetermined time. A compression processing start instruction is output to the correspondence relationship storage unit 9. Steps S502 and S503 in FIG. 17 are the processing units, and the compression start instruction process will be described later.
[0204]
The time information storage unit 11 manages the relationship between the name of the file storing the audio information and the image information and the storage start time using a table as shown in FIG. In this example, one conference record is recorded in one file. The file name is a file name given to each conference record, and the ID in FIG. 18 is an identifier (number in this example) for identifying each conference record file.
[0205]
The storage format of the storage start time is not limited to the table format, and may be a list structure or a stack structure. Further, information specifying the storage start time may be stored in a file or file name storing audio information and image information.
[0206]
As described above, in this embodiment, when a conference is started and conference recording is started, the time at the start time is stored in the time information storage unit 11 and the conference start time (storage start). The image information and the sound information are stored in the time-series information storage unit 4 from the time point).
[0207]
When user input information is detected by the user input information detection unit 6 during the conference, it is sequentially stored in the correspondence storage unit 9 together with position information on the display screen 21. Further, regarding the audio information in progress of the conference, when the change of speaker is detected by the condition matching section detection unit 3 and the user input information is detected by the user input information detection unit 6 within the condition matching section, the correspondence relationship In the storage unit 9, the condition matching section in which the user input is made is regarded as an important section, and information specifying this important section, the storage address of the corresponding time series information storage section, and user input device information are stored in association with each other. Is done.
[0208]
[Operation at the time of compression of the first embodiment of the information storage device]
Next, the operation during compression will be described. In the first embodiment, the image information and / or audio information stored in the time-series information storage unit 4 is information-compressed so that the degree of importance is reduced when a predetermined period has elapsed since storage, and the time-series information is stored. An empty capacity is formed in the memory of the information storage unit 4, but a neighboring section in which user input information is detected in the section is an important section, and this section is not compressed or the compression ratio is low. And keep high quality.
[0209]
In this case, the period from the storage to the start of compression may be a fixed period, but may be arbitrarily changed by the user.
[0210]
In addition, the period can be changed according to the file size. In this case, information specifying the storage start time may be stored in a file or file name storing audio information and image information. For example, as shown in FIG. 19, when the file size exceeds 5 Mbytes, the time to start compression processing is 1 month, and when it is less than 5 Mbytes, the time to start compression processing is 2 months. To do.
[0211]
If the file extension is .AVI, the time to start compression processing is 1 month. If the file extension is .mpg, the time to start compression processing is 2 months. As described above, the time until the compression process starts may be automatically set according to the file extension. In this case, there is no need to specify the time until the compression process starts for each file, and there is an effect that the user's trouble can be saved.
[0212]
FIG. 20 is a diagram for explaining the operation at the time of information compression in this embodiment, together with the flow of various information and the flow of output of each unit at that time.
[0213]
The time information storage unit 11 instructs the correspondence storage unit 9 to start the compression process when the elapsed time since the audio information and the image information are recorded in the time-series information storage unit 4 exceeds a predetermined time. Is output.
[0214]
That is, in step S502 of the processing routine of the time information storage unit 11 in FIG. 17, the current time supplied from a clock circuit unit (not shown) is compared with the storage start time stored in the time information storage unit 11, and the information It is determined whether or not a storage time of a predetermined time has passed. If it is determined that the predetermined time has elapsed, the process proceeds to step S503, and the correspondence storage unit 9 is requested to start the compression process.
[0215]
Then, after issuing this request or when it is determined in step S502 that the predetermined time has not elapsed, the process returns to step S500.
[0216]
For example, when the predetermined time is determined to be one month, the compression processing start request is generated one month after the storage start time, and the information newly accumulated in the time series information storage unit 4 is 1 After a month, compression will be applied. For example, the audio information and the image information of the file name “file10” recorded at 13:30 on April 25, 1996 shown in FIG. 19 are subjected to the above-described compression processing at 13:30 on May 25, 1996. Will be given.
[0217]
It should be noted that the start timing of the above compression processing may be in the vicinity of the set time, and the compression processing may be performed after the system enters the idling state.
[0218]
When the compression start instruction is input from the time information storage unit 11, the correspondence relationship storage unit 9 detects the input in step S305 in FIG. When the compression start instruction is detected, the process proceeds to step S306, where each of the information specifying the important section, the voice information stored in the time-series information storage unit 4 corresponding to each important section, and Information specifying the storage address of the image information in the time-series information storage unit 4 is output to the compression unit 10. That is, the information for each user input information shown in FIG. 13 is collectively output to the compression unit 10 for one conference.
[0219]
Of course, a set of information specifying each important section and a storage address corresponding to the important section may be sequentially output to the compression unit 10 one by one. Further, in the file of the time-series information storage unit 4 storing the voice information and the image information, each of the information specifying the important section, the voice information stored in the file corresponding to each important section, and The storage address of the image information in the file may be stored.
[0220]
The compression unit 10 that has received the input from the correspondence relationship storage unit 9 compresses the audio information and the image information accumulated in the time series information storage unit 4. In the case of this embodiment, high quality is maintained without performing data compression on information on important sections, and information on sections other than the important sections is deleted. For this reason, as shown in FIG. 20, the compression unit 10 deletes information of sections other than the important section from the information stored in the time-series information storage unit 4.
[0221]
FIG. 21 is a flowchart for explaining image compression processing of the operation of the compression unit 10.
[0222]
Hereinafter, the compression operation will be described in detail with reference to the flowchart and the explanatory diagram.
[0223]
The time information storage unit 11 has a predetermined time (ie, information storage time) after the audio information and the image information are recorded in the time-series information storage unit 4 in step S503 of the flowchart shown in FIG. When the time is exceeded, a compression processing start instruction is output to the correspondence storage unit 9.
[0224]
When the compression start instruction is input from the time information storage unit 11, the correspondence relationship storage unit 9 detects the input in step S305 in FIG. If a compression start instruction is detected, the process proceeds to step S306, where each piece of information shown in FIG. 9 of the important section and the voice stored in the time series information storage unit 4 corresponding to each important section are stored. The storage address of information or image information in the time-series information storage unit 4 is output to the compression unit 10.
[0225]
When receiving the compression start request from the correspondence storage unit 9, the compression unit 10 detects this in step S600, and proceeds to step S601. In step S601, each of the important sections input from the correspondence relationship storage section 9, and the time series of the audio information and the image information stored in the time series information storage section 4 corresponding to each of the important sections. The storage address in the information storage unit 4 is input and stored in a work memory (not shown) of the compression unit 10. The work memory uses, for example, a semiconductor memory as a storage medium.
[0226]
Note that a set of important sections and storage addresses corresponding to the important sections may be sequentially output to the compression unit 10 as one set. Further, in the file storing the audio information and the image information, each of the important sections and the storage address of the voice information and the image information stored in the file corresponding to each of the important sections are stored in the file. You may comprise so that it may memorize | store.
[0227]
The compression unit 10 refers to a plurality of sets of important intervals and storage addresses stored in the work memory, and performs thinning compression of image information and audio information stored in the time-series information storage unit 4. That is, in step S <b> 602, image information and audio information of sections other than the important section are deleted from the time series information storage unit 4.
[0228]
By the above compression processing, an empty memory as shown in FIG. 22 is generated. That is, FIG. 22 stores information (address as to address ae) recognized as an important section among image information and audio information stored in the time-series information storage unit 4 and erases information in other sections. It is the figure which showed the example in the case of having performed.
[0229]
In FIG. 22, the image data and audio data in the memory area between address as and address ae are data in the important section, so there is no change compared to before compression. On the other hand, the image data and audio data accumulated from the address a0 to the address as and from the address ae to the address a2 are not important sections and are deleted. Then, as the amount of information decreases, an empty memory area is generated in the storage medium of the time-series information storage unit 4 as shown in FIG.
[0230]
If it is desirable that the time series data is continuously stored in the storage medium, the generated empty memory portion is filled with the previous and next time series data, for example, so that there is no gap in the memory. May be eliminated.
[0231]
[Operation at the time of reproduction of the information storage device according to the first embodiment]
Next, the operation during reproduction will be described.
[0232]
In this example, the user designates (points) one of a plurality of pen handwriting information displayed on the display unit 14 using a pen, so that an audio signal stored in the time-series information storage unit 4 or Of the image signal, only the audio signal or the image signal input before and after the time when the pen handwriting information is input can be partially reproduced.
[0233]
In addition, the audio signal or the image signal can be reproduced by going back to the beginning of the important section of the audio signal or the image signal input before and after the time when the pen handwriting information is input.
[0234]
In this example, the reproduction designation unit 13 used for designation is also used as an input pen as in this embodiment.
[0235]
Note that a mouse, a trackball, a cursor key, a touch panel, or the like can be used as the reproduction specifying unit 13. Also, the pen handwriting information can be specified by pointing, specifying by an enclosure (enclosing by a closed curve or ellipse), specifying by identifier input, or specifying a line under the displayed item. Good.
[0236]
Further, when a plurality of pen handwriting information is selected as a result of designation by surrounding or underline, one can be specified based on some priority. For example, automatically select the pen handwriting information that is input earliest among the selected ones or the pen handwriting information that is displayed on the upper left, so that the candidates are displayed in a list and the user is asked to select again. Also good.
[0237]
Furthermore, still image images may be arranged in a specific order (for example, time series), and one of the table of contents images may be designated.
[0238]
Since the display position on the display screen 21 of the display unit 14 can be specified by XY coordinates, when a certain display position is specified by the reproduction specifying unit 13 such as a pen, the XY coordinates corresponding to the display position are set. Identified.
[0239]
In this way, when there is a reproduction request from the reproduction designation unit 13, it is detected in step S 307 in the flowchart of FIG. 15 of the operation in the correspondence relationship storage unit 9, and the process proceeds to step S 308 and obtained from the reproduction designation unit 13. XY coordinate group calculated from the designated coordinates and the XY coordinates stored in the correspondence storage unit 9 (all point coordinate groups constituting a multiple line having the pen handwriting coordinates in FIG. 13 as endpoints) And the corresponding pen handwriting information is specified.
[0240]
Subsequently, in step S309, the reproduction start address and reproduction end address of the audio signal or image signal corresponding to the pen handwriting information are acquired from the correspondence storage unit 9, and the process proceeds to step S310, where the reproduction start address and the reproduction end address are reproduced. The end address and the reproduction request are output to the time series information storage unit 4.
[0241]
Thereafter, the time-series information storage unit 4 that has received the reproduction start / end address and the reproduction request detects the input in step S405 of the flowchart of FIG. 16, proceeds to step S406, and outputs it to the reproduction unit 12.
[0242]
The reproduction unit 12 starts reproduction of audio information or image information from the reproduction start address to the reproduction end address of the time-series information storage unit 4 obtained by the above operation. For example, when a multiple line with ID 1000 in FIG. 13 is designated, audio information or image information from the storage address as to the storage address ae corresponding to the multiple line is reproduced.
[0243]
Note that if the coordinates are also matched with the slightly shifted coordinates in the coordinate comparison, a desired storage address can be acquired even if a slightly shifted coordinate point is specified at the time of specification. Even when the user input information is not displayed, a predetermined indicia indicating the presence of the user input information is displayed on the display unit 14, so that a desired storage address is obtained by the same method. it can. In addition, if a plurality of coordinate point sequences continuously input within a predetermined time interval are stored as one pen handwriting information, for example, a single character string can be regarded as one pen handwriting information. If any one of the coordinate point sequences constituting the sequence is designated, audio or image can be reproduced from the same storage address.
[0244]
During playback, the playback speed may be changed or the playback may be performed slowly, so that the playback unit 12 may be provided with a fast forward function, a rewind function, a slow playback function, and a pause function. An upper slide bar may be provided so that a playback position can be designated by displaying a pointer indicating the current playback time on the slide bar or by sliding the bar. Further, the speed to be reproduced does not necessarily have to be reproduced in accordance with the recorded time information, and the reproduction may be performed by increasing the speed while keeping only the recorded order relation.
[0245]
Furthermore, after the reproduction is paused, the instruction unit may be able to instruct the reproduction again, or new user input information may be additionally recorded.
[0246]
[Modification of First Embodiment]
In the above description of the first embodiment, each speech section of the conference attendee is detected as a condition matching section, but the condition matching section is not limited to this.
[0247]
For example, a single voice signal such as “door closing sound” can be detected by the condition matching section detection unit 3. In this case, a time point that is a predetermined time before the point in time when the single sound signal is detected is detected as the start point of the condition matching section, and a time point that is a predetermined time after the point in time when the single sound signal is detected is the condition match. It is configured to detect as the end point of the section. Further, the same configuration may be used when a single trigger signal from an external sensor is detected by the condition matching section detection unit 3.
[0248]
In addition to this, the event detected by the condition matching section detecting unit 3 includes the appearance of a pre-registered keyword in the audio signal, the appearance of a pre-registered audio pattern in the audio signal, and the advance in the audio signal. It may be the appearance of a registered character string, a state change in an image signal, a state change detected by an external sensor, a camera work or a camera work change.
[0249]
For example, characteristic voice patterns such as laughter patterns and applause patterns are registered, these patterns are recognized from an input voice signal, and a section including these patterns is detected as a condition matching section. You can also. In this case, the condition matching section detection unit 3 includes pattern recognition means for performing pattern recognition using a known pattern recognition technique, for example, a technique for analyzing the temporal transition of the power or frequency component of an audio signal. Provided.
[0250]
In other words, any device that detects a change in input time-series information (for example, audio information or image information) or a change in a signal detected by an external sensor while input time-series information exists is present. This corresponds to the condition matching section detection unit 3 in FIG. When the change point is used as a detection condition, the end point can be a predetermined time from the change point.
[0251]
In this embodiment, the compression unit 10 is configured to compress the image information and the audio information by performing thinning processing. However, when the image information is compressed, the storage time, the compression rate of the intra-frame compression, the frame Any device that dynamically changes at least one of the compression rate of intermittent compression, the time interval of intermittent recording, the color information thinning rate, the luminance information thinning rate, and the like may be used. In particular, methods for compressing moving image information include intra-frame compression methods and inter-frame compression methods, and intra-frame compression methods include a method using vector quantization and a method using discrete cosine transform. There is. As a compression method between frames, there is a method of recording only a difference between image information of previous and subsequent frames. That is, any device that converts the information amount per unit time into a smaller information amount corresponds to the compression unit 10 in the present invention.
[0252]
Further, when compressing audio information, there may be provided means configured to dynamically change at least one of the sampling frequency and the number of encoded bits.
[0253]
Further, the information on the important section and the information on the section other than the important section may be stored separately in different storage media. For example, when recording information, important section information and non-important section information are stored on the same magnetic disk, and when compressing information, only important section information is left on the magnetic disk. The information of the section other than the section is moved to the magneto-optical disk or the magnetic tape. In general, magneto-optical disks and magnetic tapes have a feature that a large amount of information can be stored at a low cost although the access speed to information is slower than that of magnetic disks. It is suitable for storing information of sections other than the important section.
[0254]
In the above example, the case where there is no information displayed on the display unit 14 in the initial state has been described. However, the scope of application of the present invention is not limited to this example, and for example, some user input information is present in the initial state. The case where it is already displayed and any additions or changes are made are also included in the scope of the present invention. However, in this case, the user input information that can be designated for reproduction is only the portion displaced from the initial state.
[0255]
As an application of the apparatus of the present invention, there is a method of sequentially reproducing user input information on a screen when reproducing recorded audio information or image information. The display on the display unit 14 is temporarily returned to the display at the time when the pen handwriting information specified by the reproduction specifying unit 13 is input, and the voice or image information and the pen handwriting information are reproduced in synchronization. As a method for returning the display screen to the display at that time, UNDO of the display screen may be repeated until that time, or after the screen is erased, the user input information stored in the correspondence storage unit 9 is used at that time. May be drawn sequentially at a high speed.
[0256]
Furthermore, in the above description, the compression process is started when the elapsed time (that is, the information storage time) after the audio information or the image information is recorded in the time-series information storage unit 4 becomes a predetermined time or more. However, instead of this method, it is recognized that the free space in the time-series information storage unit 4 is recognized as being less than a certain value or the storage amount in the time-series information storage unit 4 is greater than a certain value. The compression process may be started at the timing. In this case, the information storage device includes a storage amount detection unit, and when the storage amount detection unit detects that information has been recorded exceeding a pre-registered storage capacity, the correspondence storage unit 9 outputs a compression processing start instruction. The operation after this compression processing start instruction can be performed in the same manner as in the above-described embodiments.
[0257]
In the first embodiment, the example in which the shared electronic whiteboard device adds the shared whiteboard device identification information to the header of the user input information has been described. However, the inclination of the input plane of the user input device is detected. A personal electronic notebook device when it is determined that the user input surface is almost vertical and is a shared electronic whiteboard, or when the sensor detects that the device is placed on a desk or hand It may be determined that In this case, it is possible to determine whether the same device is used as a personal notebook device or a shared electronic whiteboard device. Furthermore, as a method for specifying a device to which information is input, a known method other than these can be used.
[0258]
[Second Embodiment of Information Storage Device]
In the first embodiment, all the sequentially input image information or audio information is once recorded in the time-series information storage unit 4, and the recorded conference video is recorded when a predetermined time has elapsed from the recording time point. In the above example, the compression processing is performed in which only the important part of the video is left and the other part is deleted. In the second embodiment, a case will be described in which only the video of the important part in the input conference video is selected and recorded in the time series information storage unit 4 when the time series information is recorded.
[0259]
FIG. 23 is a block diagram showing the function of the information storage device of the second embodiment. The time information storage unit 11 is removed from the block diagram of FIG. 1 and a temporary storage unit 16 is added. In the second embodiment, the important section determination unit 8 is provided in the temporary storage unit 16. Also in this case, each unit may be configured as a separate block, or one block may be configured to include several units. Moreover, one part may be divided into several blocks and mounted.
[0260]
The storage device of the temporary storage unit 16 is composed of, for example, a semiconductor memory or a magnetic disk. In the second embodiment, the audio information and the image information are temporarily stored before the time series information storage unit 4 stores them. In addition, the condition matching section information detected by the condition matching section detection unit 3 is temporarily stored in association with the input image information or the input voice information.
[0261]
As in the first embodiment, in the second embodiment, when a conference participant takes a note, it is assumed that it is an important scene of the conference. This is a case where the amount of information for storing the conference video and the conference audio is greatly reduced by storing only the periphery at the time of taking and erasing the other portions.
[0262]
Similarly to the first embodiment, a computer built in the information storage / reproduction device 20 and a small computer 30 possessed by each individual are connected via a wireless network, and input from a plurality of small computers 30 is possible. The inputted user input information and user input device information are transmitted to a computer built in the information storage / reproduction device.
[0263]
As in the first embodiment, the condition matching section detection unit 3 has detected that a voice signal input of a predetermined level or more has been input from the voice information input unit 1 and a change of speaker has been detected from the input voice signal. A condition matching section is detected as a condition. Similarly to the first embodiment, the user input device specifying unit 7 is written on the shared electronic whiteboard device 20 (electronic conference device) and on the personal electronic notebook device 30 (small computer). The user input device information for indicating the distinction is output by distinguishing from the case of writing.
[0264]
In the second embodiment, the compression unit 10 performs a data thinning process on audio information and image information stored in a temporary storage unit 16 to be described later. In this case, the compression unit 10 determines the sections other than the important sections based on the important section information specified by the important section determination unit 8 based on the timing when the user input information is input, the condition matching section information, and the user input device information. Audio information and image information are selected, and only the selected audio information and image information are stored in the time-series information storage unit 4.
[0265]
Next, the recording operation in the information storage / reproduction apparatus of the second embodiment will be described.
[0266]
FIG. 24 is a diagram for explaining the recording operation in this embodiment, together with the flow of various information and the output flow of each unit at that time. As shown in FIG. 24, voice information and image information are sequentially stored in the temporary storage unit 16, information on detection time of user input information (point of pen handwriting input), and user input device specifying unit 7. User input device information and condition matching section information from the condition matching section detection unit 3 are supplied.
[0267]
In this case, as shown in FIG. 25, the memory unit of the temporary storage unit 16 is configured in a ring buffer format, and sequentially indicates image information that is input and speaker change time points that are detection results of the condition matching section detection unit 26. The information is written in a ring shape by controlling the write address, while the old information is overwritten with new information.
[0268]
In addition, as this type of recording method, other known methods such as a queuing method and a FIFO method can be applied. In the example of FIG. 25, the number of memory (unit memory cells) in the ring buffer is 16, but of course there is no limit on the number of memories, and there is also a limit on the data storage capacity that can be accommodated in one memory. Absent. In the case of this embodiment, the ring buffer of the temporary storage unit 16 is a memory that can store input audio information and input image information for at least 380 seconds.
[0269]
In FIG. 25, the oldest data address F200 is a pointer that points to a memory in which the oldest information of image information or audio information stored in the temporary storage unit 16 is stored, and the latest data address F201 is a temporary data address. This is a pointer that points to a memory that stores image information or audio information that is most recently input to the storage unit 16.
[0270]
Further, the speaker utterance start points F202 and F204 and the speaker utterance end point F203 are obtained by storing the detection results of the condition matching section detection unit 3 in the temporary storage unit 16. Information indicating the speaker change timing may be stored in the memory of the ring buffer as in this example, or another memory may be prepared and stored therein.
[0271]
Then, as shown in the operations in the flowcharts of FIGS. 26 and 27 to be described later, the important section determination unit 8 of the temporary storage unit 16 determines the important section based on the user input information detection time, the user input device information, and the condition matching section information. When determined, in this embodiment, the information of the important section is transferred to the time-series information storage unit 4 and stored without being compressed.
[0272]
At this time, the address information (the leading address and the last address of the important section) of the important section information stored in the time-series information storage section 4 is the correspondence storage section 9. Is stored in the correspondence storage unit 9 in association with the user input information (pen handwriting information) at that time. At this time, as in the first embodiment, the user input device information can also be stored in association with the correspondence storage unit 9.
[0273]
26 and 27 are flowcharts for explaining the operation of the temporary storage unit 16.
[0274]
First, in step S700, the flag is reset to “0”. Next, the audio information and the image information input per unit time are accepted in step S701, and it is determined in step S702 whether there is an input of detection input of user input information from the user input information detection unit 6.
[0275]
If user input information is detected in step S702, the process proceeds to step S707, the flag is set to “1”, and the process proceeds to step S703. At this time, it is stored in a buffer memory (not shown) whether the user input information is input from the shared electronic whiteboard device or the personal electronic notebook device.
[0276]
When user input information is not detected, the flag remains “0” and the process proceeds to step S703. If it is determined in step S703 that the latest data does not overwrite the oldest data in the ring buffer, the process proceeds to step S704, where the input image sequence is stored in the memory unit indicated by the latest data address in the temporary storage unit 16. .
[0277]
Further, the process proceeds from step S704 to step S705, where it is determined whether or not information indicating a speech start point or a speech end point is input from the condition matching section detection unit 3, and information indicating a speech start point or a speech end point is input. If not, the process proceeds from step S705 to step S706, and the latest data address is advanced according to the stored data amount.
[0278]
If it is determined in step S705 that information indicating the utterance start point or the utterance end point has been input, the process proceeds from step S705 to step S710, where the information from the condition matching section detection unit 3 is the utterance start point. Or the end point of speech.
[0279]
If it is the speech start point, the process proceeds to step S711, where the speech start point mark is recorded at the memory location indicated by the latest data address, and then the process proceeds to step S706 to advance the latest data address according to the stored data amount. .
[0280]
If it is determined in step S710 that the speech end point is such that the voice level changes to a predetermined level or less, the process proceeds to step S712, and the speech end point mark is recorded in the memory location indicated by the latest data address. After this step S712, the process proceeds to step S713, where it is determined whether or not the flag is set to “1”. If it is not set to “1”, the process proceeds to step S706, and as described above, the stored data amount The latest data address is advanced according to. Thereafter, the process returns from step S706 to step S701.
[0281]
If the flag is set to “1” as a result of the determination in step SS713, the process proceeds to step S714, and whether the user input information detected in step S702 is input from the shared electronic whiteboard device, It is determined whether the input is from the personal electronic notebook device.
[0282]
If it is determined in step S714 that the input is from the shared electronic whiteboard device, it is determined in step S715 whether 180 seconds have passed since the user input information input time, and 70 seconds have been recognized. In that case, the process proceeds to step S716. In step S716, the voice information and image information for 200 (= 70 + 130) seconds are output to the time-series information storage unit 4 from the speech start point closest to the time point 130 seconds after the user input information input time point. Thereafter, the process proceeds to step S719. The compression unit 10 performs this thinning compression process.
[0283]
On the other hand, if it is determined in step S714 that the input is from the personal electronic notebook device, it is determined in step S717 whether 180 seconds have elapsed since the user input information input time, and it is recognized that 180 seconds have elapsed. If so, the process proceeds to step S718. In step S 718, the audio information and the image information for 380 (= 180 + 200) seconds are output to the time-series information storage unit 4 from the speech start point closest to the time point 200 seconds after the user input information input time point. Thereafter, the process proceeds to step S719. The compression unit 10 performs this thinning compression process.
[0284]
In step S719, the oldest data address is advanced to the latest data address.
[0285]
Then, the process proceeds to step S720, the flag is returned to “0”, and the process returns to step S701.
[0286]
If it is determined in step S715 that 70 seconds have not elapsed since the user input information input time point, or if it is determined in step S717 that 180 seconds have not elapsed since the user input information input time point, Returning to S701, new audio / image information is input.
[0287]
In step S702, the timing at which the user input information detection unit 6 does not detect the input of pen handwriting information is erased from the temporary storage unit 16 in step S703 (the latest data overwrites the oldest data). If so, in step S708, a predetermined amount of audio information and image information are erased. Thereafter, in the temporary storage unit 16, in step S709, the oldest data address is advanced by the deleted data amount. Thereafter, the process proceeds to step S704, and the above-described processing is repeated.
[0288]
As can be seen from the above description, when the pen input information is detected by the user input information detection unit 6, the flag is set to “1” in step S707 of FIG. However, the information thinning compression process has not yet started at this point. The compression process is started because the utterance end point is detected by the condition matching section detection unit 3 in step S710, and a predetermined time (70 seconds or 180 seconds) from the user input time point in step S715 or step S717. It is when it is recognized that it has passed.
[0289]
In the above description, the voice information and the time-series information storage unit 4 are transferred from the ring buffer of the temporary storage unit 16 after the user input information is detected to the last time specified as the important section. Since the image information is transferred, the ring buffer is a memory that can store at least 380 seconds of time-series information, but when user input information is detected, it is detected that it is within an important section. In addition, if the method starts to transfer the time series information to the time series information storage unit back to the time corresponding to the beginning time of the important section, the ring buffer has a capacity for storing at least 200 seconds of time series information. Just have it.
[0290]
In the above description of the second embodiment, the temporary storage unit 16 is configured to include the important section determination unit 8. However, in the second embodiment, the correspondence storage unit 9 also includes the important section. The determination unit 8 may be provided, and the important section information from the important section determination unit 8 of the correspondence relationship storage unit 9 may be given to the temporary storage unit 16 so that the same operation as described above can be performed.
[0291]
In the first embodiment to the second embodiment described above, the input status when the user input information is detected is whether the user input device in which the user input is made is a shared input device or a personal input device. The input device is assumed to be reflected in the determination of the important section. However, the present invention is not limited to this.
[0292]
For example, even when the difference between the user input devices is specified, it may be necessary to change the important section depending on the user of the personal input device. For example, since the situation is different between a clerk role attendee taking notes and another general meeting attendee taking notes, even if it is for personal use, the clerk and the general meeting attendee have important intervals. The timing and length may be changed. Considering the individual judgment of conference attendees and the difference in the speed with which ideas can be summarized, each individual input device is identified and the capabilities of each conference attendee are matched. The important section may be determined. In the latter case, when determining the important section, it is preferable to set the time going back and the time after the user input time for each meeting attendee.
[0293]
Further, the input situation is not limited to specifying the user input device. For example, as a user input situation specifying means, a means for specifying an input situation such as laughter or applause occurred between meeting attendees at the time of input of user input information is provided, and the specified situation is important. You may make it include in the determination factor of an area.
[0294]
In the above-described embodiment, the non-important sections are deleted as the compression method. However, the time-series information of the important sections is stored without being compressed, and the time-series information other than the important sections is compressed. May be stored. Further, time series information of important sections may be compressed and stored while maintaining high quality, and time series information other than important sections may be compressed and stored by a compression method having a higher compression rate.
[0295]
The first to second embodiments of the information storage device described above can be implemented in appropriate combinations in addition to those specified in the description of each embodiment.
[0296]
From the embodiment of the information storage device described above, the following effects can be obtained.
[0297]
(1) Compression interval or intermittent recording time interval based on section information determined by the timing when user input information such as pen handwriting is input and the information that determines which device the user input information is input from The audio information or the image information stored in the time series information storage unit is compressed to change the number of important information of the audio information or the image information in a limited storage medium. Even if audio information or image information other than the important part can be stored, it can be stored for a long time with a small amount of data.
[0298]
(2) Further, a correspondence relationship storage unit is provided, and the correspondence relationship between the segment information indicating the segment determined by the user input information and the storage position in the time series information storage unit of the audio information or the image information corresponding to the segment information is expressed. Since the information is compressed, there is no need to associate a section determined by user input information with a storage position in the time-series information storage unit of audio information or image information corresponding to the section at the time of information compression. Thus, there is an effect that the load on the system when compressing the audio information or the image information can be reduced.
[0299]
(3) Further, the user input information detection unit detects user input information input from the user, and dynamically changes the compression rate or the time interval of intermittent recording based on the timing at which the user input information is detected. By configuring the audio information or image information stored in the time-series information storage unit to be compressed, it is possible to store only important parts of the audio information or image information where characteristic events occur with high quality. In addition, there is an effect that even audio information or image information other than the important part can be stored for a long time with a small amount of data.
[0300]
(4) The user input information detection unit detects user input information input from the user, and based on the timing at which the user input information is detected, the audio information or the image information stored in the time series information storage unit Among these, by configuring so as to delete the audio information or the image information in the sections other than the important section, there is an effect that only the important part in which the characteristic phenomenon occurs in the voice information or the image information can be saved.
[0301]
(5) Further, a condition matching section detection unit is provided, and the condition matching section detection unit detects a condition matching section in which audio information or image information from the information input means matches a predetermined condition set in advance. Audio information stored in the time-series information storage unit by changing the compression rate or the compression method in the section determined by the timing when the input information is detected, the user input device information and the condition matching section, and other sections Alternatively, since the image information is configured to be compressed, there is an effect that the important part of the audio information or the image information can be reliably stored with high quality.
[0302]
(6) In addition, a correspondence relationship storage unit is provided, and from the user input information detected by the user input information detection unit, the user input device information, and the condition matching interval detected by the condition matching interval detection unit. Since the correspondence between the section information indicating the determined section and the storage position in the time series information storage unit of the audio information or the image information corresponding to the section information is stored, the user input information can be stored at the time of information compression. The processing for associating the section defined by the timing at which the signal is detected and the condition matching section with the storage position in the time-series information storage unit of the voice information or the image information corresponding to the section becomes unnecessary. There is an effect that the load of the system when compressing can be reduced.
[0303]
(7) Further, the condition matching section detection unit detects the condition matching section based on the sensor information, and based on the detection result, the detection result of the user input information detection unit, and the user input device information, the compression rate or the intermittent recording When the audio information or the image information stored in the time-series information storage unit is compressed by dynamically changing the time interval, the event that it is difficult to detect the state change of the audio signal or the image signal Even if an event occurs or if a state change occurs in information that is not included in the input audio signal or image signal, only important parts of the audio information or image information in which characteristic events occur are limited. In this case, a large amount of data can be stored in the stored storage medium, and even audio information or image information other than important parts can be stored for a long time with a small amount of data.
[0304]
(8) Further, the condition matching section detection unit detects a condition matching section based on the sensor information, and section information indicating a section determined from the detection result, the detection result of the user input information detection unit, and the user input device information. It is configured to compress the audio information or the image information stored in the time series information storage unit by dynamically changing the compression rate or the time interval of the intermittent recording based on the section information. If an event that makes it difficult to detect a change in the state of an audio signal or image signal occurs, or if a change in state occurs in information that is not included in the input audio signal or image signal, the audio information Alternatively, only a significant part of the image information in which a characteristic event occurs can be stored in a limited storage medium, and the load on the system when compressing audio information or image information can be reduced. There is a reduction can be effective.
[0305]
(9) Further, a temporary storage unit is provided, and the timing at which user input information such as pen handwriting is input, information that determines from which device the user input information is input, and voice information or image information are preset. Audio information or image information stored in the temporary storage unit is selectively stored in the time-series information storage unit based on the section information determined by the condition matching section information obtained by detecting the condition matching section that matches the predetermined condition. By being configured to output, it is possible to store only a significant part of the audio information or image information in which a characteristic event occurs in a limited storage medium, and further to store the audio information or image information. This has the effect of reducing the load on the system during compression.
[0306]
(10) In addition, the condition matching section detection unit detects a condition matching section based on the sensor information, the timing when the detection result and user input information such as pen handwriting are input, and from which device the user input information is input The state change of the audio signal or the image signal is configured to selectively output the audio information or the image information stored in the temporary storage unit to the time-series information storage unit based on the information that has been determined. Even when an event that is difficult to detect occurs or when a state change occurs in information that is not included in the input audio or image signal, a characteristic event occurs in the audio or image information. In this case, only a significant part can be stored in a limited storage medium, and the load on the system when compressing audio information or image information can be reduced.
[0307]
(11) Further, since the temporary storage unit temporarily stores the audio information or the image information input before the user input information detection unit detects the user input information, the timing before the user input occurs. Even for input audio information or image information, only important parts can be stored in a limited storage medium, and even audio or image signals other than important parts can be stored for a long time with a small amount of data. There is an effect that can be done.
[0308]
(12) In addition, the compression rate or the intermittent recording time interval is dynamically changed based on the timing when the user input information detection unit detects the user input information and the condition matching section information detected by the condition matching section detection unit. By compressing the audio information or the image information stored in the time series information storage unit or the temporary storage unit, the important part of the audio information or the image information in which a characteristic event occurs From the beginning to the end can be stored with high quality, and even audio information or image information other than the important part can be stored for a long time with a small amount of data.
[0309]
(13) Also, based on the timing when the user input information detection unit detects the user input information and the condition matching section information detected by the condition matching section detection unit, the voice information or the image stored in the time series information storage unit Since it is configured to delete voice information or image information of sections other than the important section of information, there is an effect that only important parts of the voice information or image information in which characteristic events occur can be stored. is there.
[0310]
(14) In addition, a display unit that displays user input information using a pen or a keyboard input by a recorder at an arbitrary timing, a designation unit that designates one of the user input information displayed on the display unit, and a time series A playback unit for playing back audio information or image information of a portion stored in correspondence with the user input information specified by the specifying unit out of the audio information or image information stored in the information storage unit When configured, if the recorder performs input with a pen or a keyboard so that a memo is taken, and the voice or image signal is recorded, the voice or image signal can be recorded later by referring to the inputted memo. There is an effect that a portion to be reproduced can be easily selected and reproduced.
[0311]
(15) Further, a display unit for displaying user input information, a designation unit for designating one of the user input information displayed on the display unit, and a memory corresponding to the user input information designated by the designation unit If a playback unit that plays back the audio information or image information of the portion that has been configured and a condition matching section detection unit are provided, the audio information or image information corresponding to the specified user input information There is an effect that the part can be reliably reproduced from the beginning to the end.
[0312]
(16) Further, the configuration is such that the compression process is started when the elapsed time (that is, the information storage time) after the audio information or the image information is recorded in the time-series information storage unit becomes a predetermined time or more. In this case, it is possible to store recent voice or image information that is highly likely to be referenced with high sound quality / high image quality, and to store voice or image information recorded in the past with a small amount of data. is there. In addition, when the audio information or the image information is configured to be compressed in stages according to the elapsed time (that is, the information storage time) since it was recorded in the time-series information storage unit, it can be referred to later. Since the information can be stored with a smaller amount of information when the performance becomes lower, the storage medium can be saved more effectively.
[0313]
(17) Further, the compression processing is performed at a timing when it is recognized that the free space in the time series information storage unit becomes a certain value or less, or at a timing when the storage amount in the time series information storage unit is recognized as a certain value or more. When configured to start, there is an effect that input can be continued even when newly input audio or image information is input exceeding the storage capacity of the time-series information storage unit.
[0314]
(18) Further, in the case where a specific caller of voice or a change of caller is detected from the input voice information or image information by the condition matching section detection unit, the voice of the specific caller or Image information can be saved from the beginning to the end with high sound quality / high image quality, and even the voice information or image information of other callers can be saved with a small amount of data, going back to the point when the caller changed This has the effect of starting playback.
[0315]
(19) Further, when it is configured to detect a specific caller or change of caller based on sensor information, an event that makes it difficult to detect a change in the state of the audio signal or the image signal has occurred. In case of a change in the state of information that is not included in the input audio signal or image signal, the voice or image information of a specific caller can be saved from the beginning to the end with high sound quality / high image quality. In addition, even voice information or image information of other callers can be saved with a small amount of data, and there is an effect that playback can be started retroactively until the caller changes.
[0316]
(20) When the user input device specifying unit is configured to detect a shared electronic whiteboard device shared by a plurality of persons and a personal electronic notebook device privately used by each individual, a comparison is made. The interval between the time of writing notes on a personal electronic notebook device that is far from the target and the time when related information is spoken, and the time of writing notes and related information on a nearby shared electronic whiteboard Since important sections corresponding to each interval can be specified, the voice or image information of the important sections can be saved from the beginning to the end with high sound quality / high image quality, and other caller's voice information or images Even information can be saved with a small amount of data, and the reproduction can be started back to the beginning of the important section.
[0317]
(21) Further, when the user input device specifying unit is configured to detect the installation location or installation state of the user input device and specify the user input device, even if it is the same device, It is possible to determine whether the device is used as a personal notebook device or as a shared electronic whiteboard device, and it is possible to identify an important section according to the use of the device.
[0318]
(22) Further, the user input device specifying unit detects that the user input device is placed on at least a desk, a hand, a wall, or a shelf, or an input plane of the user input device Even if the same device is used, the device can be easily used as a personal notebook device. It is possible to determine whether the device is used or as a shared electronic whiteboard device, and it is possible to specify an important section according to the use of the device.
[0319]
(23) Also, a user input situation detection unit for detecting a user input situation, and a time series information section or time input from the time series information input unit according to the situation detected by the user input situation detection unit. When it is configured to include a target section specifying unit that specifies and outputs the specified result, input processing, compression processing, storage processing, search processing, display processing for time-series information, according to the state of user input, The section of the output process can be specified, and processing according to the user input situation such as writing at hand or writing on the whiteboard is performed on a specific section of the time series information There is an effect that can.
[0320]
(24) A time-series information input unit for inputting audio information or image information, and a time-series information section or time is specified according to the detection result detected by the time-series information input unit, and the specified result And a target section specifying unit for outputting the time series information according to the situation of the time series information, input processing, compression processing, storage processing, search processing, display processing, output processing section Specific processing of time-series information, such as laughter occurred, applause occurred, discussion was actively conducted, there was a long remark, etc. There is an effect that can be applied to the section.
[0321]
[Embodiment of Information Storage / Reproduction Device]
In the embodiment of the information storage / playback device described below, the aim is to realize a device that can appropriately play back audio information and image information according to the input situation when a user input is made. Yes. In this embodiment, whether the user input information is performed for the purpose of transmitting to others (situation 1), or is performed for the purpose of recording information transmitted from others (situation 2). We try to make it happen by distinguishing between the two situations.
[0322]
That is, when considering a situation such as a meeting, in the situation 1, after the user input is performed, the user input is often explained, whereas in the situation 2, the user is input before the user input. In this embodiment, attention is paid to the situation dependency of human behavior characteristics that information related to input is spoken. In this embodiment, the situation 1 assumes a situation in which a so-called presentation is performed, and the situation 2 assumes a situation in which a lecture or explanation by the presentation is taken as a memo.
[0323]
The outline of the system configuration of the information storage / playback apparatus of this embodiment is almost the same as that of the electronic conference system as shown in FIG. However, in this embodiment, one of a plurality of conference participants makes a presentation on a predetermined theme, and the other conference participants take notes of the lectures and explanations.
[0324]
In the case of this embodiment, each conference participant has a system configuration capable of becoming a presenter or listening to the presentation while present. Each conference participant has a personal computer 30 for personal use connected to the shared electronic blackboard 41 as in the case of FIG. 2, but each of the personal computers 30 in this embodiment is in the shared mode. It has a function that can be set to use or personal mode.
[0325]
When the shared mode is set, the contents of the display screen of the personal computer are displayed on the shared electronic blackboard 41, and when the user performs an input operation of user input information through the display screen of his / her personal computer 30, this is the personal computer. It is reflected not only on the display screen of the computer 30 but also on the shared electronic blackboard 41, and a presentation can be made. Therefore, this shared mode is referred to as a presentation mode.
[0326]
When the personal mode is set, a user of the personal computer 30 can input user input information as a personal memo on his personal computer 30 as a participant who receives the presentation. The personal memo input information is only displayed on the screen of the personal computer and is not displayed on the shared electronic blackboard 41.
[0327]
FIG. 29 is a diagram for explaining a method in which the user sets the mode in the personal computer 30.
[0328]
In FIG. 29, 30D is a display screen of the personal computer 30. On this display screen 30D, a status setting button icon 31 is displayed. When the user presses the status setting button icon 31, a pop-up window 32 of the status setting menu is opened. In this embodiment, the user can select any of the presentation mode, the memo mode, and the auto mode in this situation setting menu.
[0329]
The presentation mode is selected when an explanation is given while writing in a shared writing area where other conference participants can see their opinions and the like. The memo mode is a mode in which a memo is taken with respect to information spoken by others. The memo mode is a mode that is set when taking a personal memo or writing down the opinion of another person as a clerk in the shared writing area. Which mode is set may be set before a meeting or the like, or may be set according to a change in the situation during the meeting.
[0330]
In this embodiment, the mode in which the system automatically recognizes the situation can be set as the auto mode. As will be described later, this auto mode is a mode for specifying a situation when an input operation of user input information is automatically performed from time-series information of audio information and image information.
[0331]
In FIG. 29, the user B sets his / her personal computer 30 to the presentation mode, and displays the same screen as the screen of his / her personal computer 30 on the electronic shared blackboard 41. The other users A, C, and D are set in the memo mode.
[0332]
FIG. 30 is a block diagram showing an information storage / playback apparatus according to an embodiment of the present invention in which each processing function unit is shown as a block.
[0333]
That is, the information storage / playback apparatus of this embodiment includes a user input unit 51 for a user to input information, a user input information control unit 52 for controlling information input by the user, information input by the user, and A user input information recording unit 53 that records related information, a display unit 54 that displays information input by the user, a time series information input unit 55 that inputs time series information such as audio information and video information, and a time series A time-series information processing unit 56 for processing information, a time-series information control unit 57 for controlling time-series information, a time-series information output unit 58 for outputting time-series information, and depending on the situation A time series information specifying section determining unit 59 for outputting a time section of time series information, a chart creating unit 60 for creating a chart from the time series information, and a time series information recording unit 61 for recording time series information, A situation specifying unit 62 for specifying a situation when the user inputs information, and a coordinate time conversion unit 63 for converting the coordinate information of the user input information when the user inputs information into time information. .
[0334]
The user input information input means corresponds to the user input unit 51, and the user input information control unit 52 is provided as a partial function of the user input information detection unit.
[0335]
The information storage / playback apparatus of this embodiment shown in FIG. 30 can be configured by each of the personal computers 30 of FIG. FIG. 31 shows an overview of a system centered on one personal computer 30.
[0336]
That is, in the information storage / playback apparatus of this embodiment, image information as time-series information is input from the video camera 42 to each personal computer 30. For this reason, each personal computer 30 includes a video input board, and the image information is digitized by the built-in video input board and is taken into the personal computer 30 in a form that can be processed.
[0337]
In addition, the audio information is transmitted from all the microphones 43 assigned to each of the users A to D who are each conference participant to each personal computer 30 from the microphone for each participant. The voice information can be identified and input through a voice input board built in the voice information. Therefore, the time-series information input unit 55 described above includes the video camera 42 and the microphone 43.
[0338]
Each of the personal computers 30 includes an interface that enables user input with an electronic pen, and can make notes during a meeting. This pen input interface is one component of the user input unit 51. The user input unit 51 can also be configured by a user interface using a pointing device such as a keyboard or a mouse.
[0339]
As recording media of the user input information recording unit 53 and the time-series information recording unit 61, specifically, a semiconductor memory or a hard disk built in the personal computer 30 is used. A recording medium such as an MO disk or a floppy disk may be used.
[0340]
The display unit 53 and the time-series information output unit 58 are realized by a display 30D and a speaker 30SP connected to the personal computer 30.
[0341]
The user input information control unit 52, the time series information processing unit 56, the time series information control unit 57, the time series information section specifying unit 59, the chart creating unit 60, the situation specifying unit 62, and the coordinate time converting unit 63 are added to the personal computer 30. It is realized by the installed software.
[0342]
The user input information control unit 52 receives user input information from the user input unit 51, sends it to the display unit 54, and displays it on the display screen. Then, processing corresponding to the received user input information is performed according to information recording (information storage) or information reproduction.
[0343]
As will be described later, the user input information control unit 52 records the user input information input at the time of recording time series information as conference information, the input start time and end time, the situation at that time, etc. Recorded in the user input information recording unit 53.
[0344]
Further, at the time of reproduction, the user input information control unit 52 reads information recorded in the user input information recording unit 53 in accordance with a user instruction, and sends the information to the display unit 54 and the time-series information specific section determination unit 59. To communicate. Further, when the user inputs an instruction while the chart is displayed on the display screen of the display unit 54, the instructed coordinate information is transferred to the coordinate time conversion unit 63.
[0345]
The display unit 54 displays the user input information passed from the user input information control unit 52 on the display screen. In addition, an utterance chart (described later) created by the chart creation unit 60 through the time-series information control unit 57 is displayed on the display screen.
[0346]
As will be described later, the user selects any one of the user input information displayed on the display screen of the display unit 54 for reproduction of corresponding time-series information or a chart creation instruction. can do. Further, in this system, in the message chart displayed on the display unit 54, the time series information corresponding to the designated reproduction location is reproduced by indicating a specific reproduction location, that is, the reproduction start point or the reproduction section. Can be made to do.
[0347]
The time-series information processing unit 56 corresponds to a condition-matching section detecting unit that detects a section that matches a predetermined condition with respect to time-series information. In this embodiment, each user of a conference participant in a meeting The speech sections A to D are detected as the condition matching sections when recording time-series information.
[0348]
The time series information control unit 57 controls to record the time series information in the time series information recording unit 61 at the time of recording. At this time, the speech section for each user detected by the time-series information processing unit 56 is also recorded in the time-series information recording unit 61.
[0349]
Further, at the time of reproduction, the time series information control unit 57 reads the specific section determined by the time series information specific section determination unit 59 from the time series information recording unit 61 and reproduces and outputs it by the time series information output unit 58. To. Further, the chart created by the chart creating unit 60 is displayed on the display screen of the display unit 54. Then, the time-series information recorded in the time-series information recording unit 61 is read from the time position corresponding to the coordinate position indicated on the chart, and is reproduced and output by the time-series information output unit 58.
[0350]
The time series information section specifying unit 59 operates in response to a command from the user input information control unit 52 at the time of reproduction. As described later, the time series information section specifying unit 59 corresponds to user input information instructed to be selected by the user. The section of the time series information of the conference information of the audio information is specified. Then, the time information of the identified section is passed to the time series information control unit 57 or the chart creation unit 60 in accordance with a user instruction.
[0351]
The chart creation unit 60 receives information on a specific section from the time-series information specific section determination unit 59 at the time of reproduction, and acquires information on the speech sections of the users A to D in the specific section from the time-series information recording unit 61. 41, a statement chart for a specific section is created. The created message chart is displayed on the display screen of the display portion 54 through the time-series information control unit 57. The speech chart will be described later.
[0352]
The situation specifying unit 62 responds to an inquiry from the user input information control unit 52 when the user input information is detected in the personal computer 30 at the time of recording the time series information. It is specified whether the 30 modes are the presentation mode or the memo mode, and information on the specified mode, that is, status information is returned to the user input information control unit 52.
[0353]
When the presentation mode or memo mode is set by the user, the setting state is stored in the buffer, and when the user input information control unit 52 makes an inquiry by referring to the buffer, It can be specified whether it is in the mode.
[0354]
However, in auto mode, it becomes a problem. Therefore, in this embodiment, the presentation mode or the memo mode is specified depending on whether or not the screen information of the personal computer is shared screen information. As will be described later, there is a method of specifying from the action of the input person.
[0355]
The coordinate time conversion unit 63 generates the reproduction time position of the time series information from the specified coordinates when a specific message section is specified by the user when the message chart is displayed on the screen of the display unit 54. Is to do.
[0356]
In the information storage / playback apparatus of this embodiment configured as described above, time-series information such as audio information of the speech of the conference participant and video image information of the state of the conference, and information input from the user during the conference are received. The time series information corresponding to the information input by the user can be appropriately reproduced after recording.
[0357]
[Recording in the embodiment of the information storage / playback apparatus]
First, the recording operation of the information storage / playback apparatus of this embodiment will be described.
[0358]
The time series information of the image information from the video camera 42 and the audio information from the microphone 43 is converted into a digital signal by the time series information input unit 55 and input to the time series information processing unit 56. As described above, the time-series information processing unit 56 detects a speech section as a condition matching section. Also in this embodiment, as in the above-described embodiment, since the microphone 43 is assigned to each conference participant, by detecting which microphone the input audio information is from Detect if it is a statement.
[0359]
The speech period of each participant is detected as shown in the flowchart of FIG. 32 and the explanatory diagram of FIG. In this example, when the audio information from a certain microphone continues for a certain time or more than a preset level, it is detected that a conference participant using the microphone is speaking. .
[0360]
That is, as shown in FIG. 33, when a sound signal of a certain level L1 or higher is output from the microphone, the process proceeds to step S800, and a time interval determined in advance as an appropriate unit time length for detecting the start of speech. It is monitored whether the audio signal level equal to or higher than the level L1 is maintained over Δt1 or higher. If it does not persist, it is not regarded as a speech, and the speech section detection is terminated. If it is determined that it has continued, the process proceeds to step S801, where the current time (T1) is detected, and T1-Δt1 is set as the speech start time.
[0361]
In step S802, in order to obtain the end time of the sound, the sound has a certain level L2 that is equal to or longer than a time Δt2 that is set as an appropriate unit time length for detecting the end of speech from a certain time T2. Monitor whether it has fallen below. If it falls below, the process proceeds to step S803, and T2-Δt2 is detected as the speech end time.
[0362]
In the above example, the level L1 for detecting the speech start time and the level L2 for detecting the speech end time are set to L1 = L2, but the level L1 and the level L2 are not necessarily equal. Also good.
[0363]
Information relating to the speech section for each conference participant detected as described above is collected in a data structure as shown in FIG. The time series information processing unit 56 sends information related to the speech section to the time series information control unit 57 together with the time series information of the image information and the voice information. The time-series information control unit 7 records time-series information of all image information and audio information from the start to the end of the conference in the time-series information recording unit 61, and also stores information related to the speech section as a time-series information recording unit. 61. Then, the time-series information control unit 57 sends the image information and audio information being recorded to the time-series information output unit 58 so that it can be monitored.
[0364]
The time series information recording start time is also recorded in the time series information recording unit 61. This is for calculating the time corresponding to each reproduction position during reproduction.
[0365]
As shown in FIG. 34, the recorded information related to the speech section includes speech identification data (speech ID) for identifying each speech section, the speaker, the speech start time, and the speech end time. As the speaker information, a conference participant name registered in advance is recorded. Note that a correspondence table between the speaker name and the speaker identification data may be prepared separately, and the speaker identification data may be recorded in the time-series information recording unit 61.
[0366]
When a user input is made with the recording of the time series information, necessary information is recorded in the user input information recording unit 53 as follows.
[0367]
FIG. 35 shows a processing routine at the time of recording by the user input information control unit 52. That is, when a user input such as a pen touch is detected, the process proceeds to step S900, and the user input information is displayed on the display screen of the display unit 54.
[0368]
In step S901, when user input is performed, the image information displayed on the screen of the personal computer on which the user input is performed (still image information displayed as a window or the like; The user input information is extracted, the start time and end time of the user input are obtained, and the information of the input status (presentation mode or memo mode) at that time is determined from the status specifying unit 62 To get.
[0369]
In this case, the situation information is inquired from the user input information control unit 52 to the situation specifying unit 62, and as a result, returned from the situation specifying unit 62. Each time a user input block is cut out, the user input information control unit 52 inquires of the situation specifying unit 62 about the situation and acquires the situation information.
[0370]
In step S902, an ID (identification data such as an identification number; the same applies hereinafter) is assigned to the worksheet specified in step S901, and an ID is assigned to the extracted user input information. In step S903, the information, that is, user input information including coordinate values, the user input information ID, the worksheet ID, the input time and end time of the user input information, and the status information are displayed. Then, it is recorded in the user input information recording unit 53.
[0371]
FIG. 36 is a detection process routine of the input start time and input end time of the user input information in the process of step S901. FIG. 37 is a diagram for explaining the input start and end time detection processing of the user input information.
[0372]
That is, when a user input such as a pen touch is detected, the process proceeds to step S1000, and the time at that time is set as an input start time (T1 in FIG. 37). In step S1001, it is determined whether there is no user input for a predetermined time Ti or more set in advance to detect that user input information does not exist.
[0373]
In step S1001, when there is a user input during a predetermined time Ti, it is determined that the user input information is continued. When it is detected that there is no user input during the predetermined time Ti, it is determined that the user input information has ended, and the process proceeds to step S1002, where the user subtracts the time obtained by subtracting Ti from the current time. It is assumed that the input end time (T2 in FIG. 37).
[0374]
As shown in the explanatory diagram of FIG. 37, the user input performed between the user input start time T1 and the user input end time T2 is regarded as one input unit (block). User input information is handled in units of blocks and recorded and reproduced.
[0375]
FIG. 38 shows the structure of information recorded in the user input information recording unit 53. That is, a user input information ID is assigned to each input block. The coordinate value represents the coordinate input by the pen. The worksheet ID represents the number of the worksheet on which pen input has been performed. The correspondence between the worksheet ID and the worksheet is managed separately.
[0376]
The input start time represents the time when input of the user input information is started. The input end time represents the time when the input of the user input information is ended. As will be described later, the link information represents the time of the time series information when a specific link time of the time series information is given to the user input information by the user. The status indicates the status (presentation mode or memo mode) when the user input information is input.
[0377]
FIG. 39 shows an example of the processing routine of the situation specifying unit 62. That is, first, in step S1100, it is determined whether there is an inquiry from the user input information control unit 52. If there is an inquiry, the process proceeds to the next step S1101. In step S1101, it is checked whether the screen information of the personal computer is displayed on the shared electronic blackboard 41 at that time. If it is determined that the screen information is displayed on the shared electronic blackboard 41, the process proceeds to step S1102. “Presentation” is returned to the user input information control unit 52 as information. When it is determined that it is not displayed on the shared electronic blackboard 41, the process proceeds to step S1103, and “memo” is returned to the user input information control unit 52 as status information.
[0378]
The situation specifying operation of the situation specifying unit 62 may be as follows. FIG. 40 shows a processing routine of another example of the situation specifying operation.
[0379]
First, in step S1200, it is determined whether there is an inquiry from the user input information control unit 52. If there is an inquiry, the process proceeds to the next step S1201. In step S1201, the duration of the speech by the input person and the number of instruction operations on the display screen during the predetermined time T from the input time are detected. Then, the process proceeds to step S1202, and it is determined whether the speech time by the input person is equal to or greater than a predetermined time Tp and the number of instruction operations on the display screen is equal to or greater than the predetermined number Np during the predetermined time T. .
[0380]
In step 1202, during a predetermined time T from the input time, the utterance by the input person is equal to or longer than a predetermined time Tp, and the number of instruction operations on the display screen is a predetermined time during the predetermined time T. If it is determined that the number of times is Np or more, the process advances to step S1203 to return “presentation” as the situation information to the user input information control unit 52. If not, the process proceeds to step S1204, where “memo” is returned to the user input information control unit 52 as status information.
[0381]
As an example of the time T, Tp and the number Np, the time T = 30 seconds, the time Tp = 10 seconds, and the number Np = 2. However, the instruction detection method calculates the minimum rectangle that encloses the input block of user input information, and determines that the instruction has been made when the area inside the rectangle is touched with a pen.
[0382]
[When playing back information storage / playback equipment]
Among the time series information as the conference information recorded as described above, the time series information of the specific section corresponding to the user input information recorded in the user input information recording unit 53 is specified as described below. Thus, it is possible to reproduce or display the message chart of the specific section.
[0383]
During reproduction in the information storage / reproduction apparatus of this embodiment, first, a plurality of user input information recorded in the user input information recording unit 53 is displayed on the screen of the display unit 54. When all the user input information cannot be displayed on one screen, it is displayed by scrolling.
[0384]
The user uses the user input information displayed on the display unit 54 as an index to designate a conference recording portion that he / she wants to reproduce. That is, the user estimates from the user input information such as a memo whether or not the user input information corresponds to the conference recording portion that he / she wants to reproduce, and when the user wants to reproduce the corresponding conference recording portion, the user input Information is selected and instructed.
[0385]
In the case of this embodiment, the automatic playback mode and the message chart of the specific section are displayed using the specific section determined by the time-series information section specific section determination unit corresponding to the user input information instructed to be selected. There is a chart display mode. Here, in the automatic reproduction mode, the reproduction is started from the time Ts, but the reproduction can be terminated at an arbitrary point designated by the user. In the chart display mode, as will be described later, the user can designate a specific speech section on the speech chart, and the time series information corresponding to the designated speech section can be reproduced.
[0386]
Before explaining the reproduction operation, the message chart will be described. FIG. 41 shows an example of a message chart displayed on the screen of the display unit 54. This message chart is composed of a speaker name area 71 for displaying a speaker name for identifying a speaker, and a message transition display area 72 for visually displaying a state of a message transition. In the message transition display area 72, the message structure in the specific section determined by the time-series information specific section determination unit 59 is displayed. In FIG. 41, Ts is the start time of the specific section, and Te is the end time of the specific section.
[0387]
In each column on the right side of each speaker in the message transition display area 72, a rectangle indicating when and how much time each conference participant (speaker) performed in the specific section during the conference. This is indicated by the display position and length of the speech section bar VB. The statement transition structure in the specific section indicating who has made a transition from who to speak by reading the speech structure displayed as the transition of the speech section bar VB of all conference participants in the speech transition display area 72 Can be read.
[0388]
A triangle mark 73 in the message transition display area 72 in FIG. 41 indicates a user input time point (input start time) on the message chart in the specific section.
[0389]
Next, operation processing during reproduction will be described. As described above, at the time of this reproduction, the user input information recorded in the user input information recording unit 53 is displayed on the display screen of the display unit 54. When the user gives an instruction to select any user input information on this display screen, the user input information control unit 52 detects that and starts playback.
[0390]
FIG. 42 is a processing routine of the user input information control unit 52 at the time of reproduction.
[0390]
First, in step S1300, when the user input information control unit 52 receives an instruction to select any of the user input information displayed on the screen of the display unit 54 through the user input unit 51, the process proceeds to the next step S1301. In this embodiment, user input information is selected by surrounding the displayed user input information with, for example, an electronic pen. The user input information control unit 52 determines the enclosed area and searches the user input information recording unit 53 for user input information existing in the area.
[0392]
In step 1301, the input start time and status information of the user input information instructed for selection is acquired from the user input information recording unit 53. Then, the process proceeds to the next step S1302, and an inquiry message is displayed to the user as to whether the user input is for automatic reproduction of time-series information or creation of a chart to be described later.
[0393]
In response to this, since the user makes a selection input for automatic reproduction or chart creation, the user selection input is accepted in step S1303. In step S1304, it is determined whether automatic reproduction is selected or chart creation is selected.
[0394]
If the automatic reproduction is selected, the process proceeds to step S1305, where the time series information specifying section determining unit 59 is instructed to input the input start time information of the instructed user input information, the situation information, and the automatic reproduction command. And an automatic reproduction command is sent to the time-series information control unit 57. If chart creation is selected, the process proceeds to step S1306, where the time series information specifying section determination unit 59 is instructed to input the input start time information of the instructed user input information, the situation information, the chart creation command, Is sent to the time-series information control unit 57.
[0395]
In the above description, only the input start time is used, but only the input end time can be used. Further, both the input start time and the input end time may be used. However, the numerical values in FIGS. 44 and 45 differ depending on how they are used.
[0396]
Next, the operation of the time-series information specific section determination unit 59 will be described with reference to the flowchart of FIG.
[0397]
First, in step S1400, an input time (input start time in the embodiment), status, and command are acquired from the user input information control unit 52. In step S1401, a specific section of time-series information corresponding to the user input information is calculated from the input time of the user input information and the situation information.
[0398]
In this case, the specific section is determined as described later based on the information on the relative start time with respect to the input time point of the user input information and the information on the relative end time. Since there is information on the input time of the user input information, the absolute start time Ts and the absolute end time Te of the specific section are determined.
[0399]
The time series information specific section determination unit 59 stores information on relative start time and information on relative end time with respect to the input time point of the user input information for determining the specific section. Using the information, a specific section of time-series information corresponding to the user input information is determined.
[0400]
Next, the process proceeds to step S1402, and it is determined whether the instruction acquired in step S1400 is an automatic reproduction instruction or a chart creation instruction. If the instruction is an automatic reproduction instruction, the process proceeds to step S1403, and the time-series information control unit 57 Information on the specific section (in this example, start time Ts) and information on the input time of the user input information are sent. If it is a chart creation command, the process proceeds to step S1404, and information on a specific section (start time Ts and end time Te) and information on the input time of user input information are sent to the chart creation unit 60. Thus, the processing routine of the time series information specific section determining unit 59 ends.
[0401]
Next, the determination of the specific section of the time series information corresponding to the user input information in step S1401 will be described in more detail.
[0402]
In this case, as described in the above-described embodiment, the time interval specified as the interval corresponding to the input time point of the user input information is different between the presentation mode and the memo mode.
[0403]
For example, FIG. 44 is a diagram for explaining the probability of the relative time with respect to the input time of the user input information at the start time of the specific section of the time-series information corresponding to the user input information. FIG. 44 shows a case where a specific user reproduces time series information and determines a specific section corresponding to the user input information by himself / herself. It shows the frequency of how much it has been determined as the start time of the specific section of the time series information corresponding to the user input information during the second section.
[0404]
For example, the relative time [−300, −250) indicates a section from 300 seconds to 250 seconds before the input time point of the user input information. In the example of FIG. In the presentation mode, the start time of the specific section is determined 0 times, that is, the start time of the specific section is not determined. Here, the minus before the numerical value of the relative time means that the time is before the input time of the user input information.
[0405]
The relative time [−50, 0) indicates a section from 50 seconds to 0 seconds before the user input information is input. In the example of FIG. 44, the relative time [−50, 0) is 8 times in the memo mode in this section. In the presentation mode, the start time of the specific section is determined 20 times in the presentation mode.
[0406]
The frequency distribution of FIG. 44 is as shown in FIG. 46 in the memo mode, for example, when the horizontal axis is the relative time with respect to the input time of the user input information and the vertical axis is the frequency. In this embodiment, the relative start time and end time with respect to the input time of the user input information in the specific section are determined so that, for example, 90% of the frequency number is included. That is, in this example, the relative start time and the relative end time of the specific section of the time series information are determined so that the start time of the specific section of the time series information with respect to the user input information is correctly determined with a probability of 90%. decide.
[0407]
The information on the relative time with respect to the start point of the user input information for determining the specific section of the time-series information corresponding to the user input information determined as described above can be seen from the probability distribution of FIG. It depends on the difference between the memo mode and the presentation mode.
[0408]
That is, FIG. 47 shows the probability distribution of the relative time with respect to the input time of the user input information at the start time of the specific section of the time series information corresponding to the user input information in both the memo mode and the presentation mode. In the memo mode, the relative start time is Ts (memo), and in the presentation mode, the relative start time is Ts (presentation). is there.
[0409]
Similarly, the relative end time of the specific section of the time series information with respect to the user input information is also determined so that the end time of the specific section of the time series information with respect to the user input information is correctly determined with a probability of 90%, for example.
[0410]
FIG. 45 is an example of a relative start time and a relative end time for determining a specific section set in accordance with the situation as described above. In this example, when the situation is in the memo mode, the user input information is from 200 seconds before the user input information start time to 30 seconds after the user input information start time. The specific section of the corresponding time series information is determined as. Further, when the situation is the presentation mode, a time series corresponding to the user input information from a time point 50 seconds before the start time of the user input information to a time point 20 seconds after the start time of the user input information. The specific section of information is determined as.
[0411]
For the information on the relative start time and the relative end time in FIG. 45, fixed data determined as described above for a plurality of users can be used in all personal computers 30.
[0412]
However, the information on the relative start time and the relative end time in FIG. 45 is generally different depending on the propensity of the user to use.
[0413]
Therefore, in this embodiment, the specific section of the corresponding time-series information is determined using the fixed data of the relative time with respect to the input time of the user input information corresponding only to the difference in the situation, regardless of the user's tendency. A mode for determining a specific section of the corresponding time-series information in consideration of the propensity of the user to determine the specific section in the past.
[0414]
In the latter mode, the user sets a specific section corresponding to the user input information while reproducing time-series information recorded in advance. In this case, the time-series information section specifying unit 59 receives from the user input information control unit 52 the input time of the user input information obtained by reproducing the specific section of the time-series information immediately before, the input status, and the start time of the time-series information. And record the frequency in the relevant relative time column of the mode of the relevant situation.
[0415]
Then, by setting by the user, frequency data similar to that shown in FIG. 44 is created, and each situation is determined based on a probability value for determining a specific section set by the user for this frequency data. A relative start time and a relative end time are determined in accordance with. The value of the probability set by the user is not limited to 90% described above, and the user can set an appropriate value.
[0416]
In the latter mode, when frequency data is newly generated by the user, the relative time information is updated. In this way, by allowing the user to add information for determining the relative time, personal information such as a note of the user's memo is specified according to the situation in the time-series information interval specifying unit 59. Can be reflected in the decision.
[0417]
In addition to the creation of the frequency data of the specific section, the link information described above is set as shown in FIG. 38 corresponding to the user input information, and the user records it in the user input information recording unit 53. Can be. That is, when it is determined that the user is related to the user input information, information relating to the immediately preceding reproduction portion is recorded as link information of the user input information recording unit 53. This makes it possible to reproduce the time-series information corresponding to the input information more accurately based on the link information when the part is reproduced again later.
[0418]
Therefore, when the user input information control unit 52 transmits a reproduction command to the time-series information control unit 57, the user input information control unit 52 holds the ID of the user input information selected for reproduction in the buffer, The sequence information control unit 57 holds the reproduction time information in a buffer.
[0419]
When linking after reproduction, the user input information control unit 52 obtains the reproduction time information regarding the immediately preceding reproduction part from the buffer of the time-series information control unit 57, and the corresponding link information of the user input information recording unit 53. So that it can be recorded in
[0420]
At the time of user input at the time of creating the frequency data for determining the relative time described above, the user input information control unit 52 and the time-series information are the same as when the link information is recorded in the user input information recording unit 53. The previous reproduction information in the buffer of the control unit 59 is used.
[0421]
Specifically, when the user commands the information update of the time-series information specific section determination unit 57, the user input information control unit 52 uses the user input information ID reproduced immediately before as a clue to the user input information recording unit 53. , The playback time information immediately before being held in the buffer of the time-series information control unit 59 is acquired, and the information is sent to the time-series information specific section determination unit 57.
[0422]
When the relative time information is newly added by the user, the same information as shown in FIG. 44 is obtained. In this case, the time series information section specifying unit 59 not only outputs the time information of the specific section, but also the time series on that section (Ts to Te) as shown in FIGS. Information existence probability information and the like can be output together.
[0423]
In the above description, the buffers of the user input information control unit 52 and the time-series information control unit 57 are configured to hold only the immediately preceding reproduction information. The reproduction information may be held.
[0424]
Next, the operation during reproduction of the time-series information control unit 57 will be described with reference to the flowchart of FIG.
[0425]
That is, when an instruction is acquired from the user input information control unit 52 in step S1500, the process proceeds to step S1501 to determine whether the instruction is chart creation or automatic reproduction. If it is chart creation, it will progress to step S1502 and will instruct | indicate chart creation to the chart creation part 60. FIG. In step S1503, the chart information is acquired from the chart creation unit 60, the output position of the chart information to the display unit 4 is determined, the output coordinates are held, and the chart information is output to the display unit 4.
[0426]
When the message chart information is displayed on the screen of the display unit 54, the user selects and instructs one of the message section bars VB displayed in the message chart, as shown in FIG. Requests playback of time-series information of a speech section. In FIG. 50, when the position indicated by the circle is designated, the designated coordinate (xp, yp) is converted into time information by the coordinate time conversion unit 63 and sent to the time series information control unit 57. In this example, the designated coordinates may be only the x coordinate xp.
[0427]
For this reason, in step S1504, the time-series information control unit 57 acquires time information designated by the user from the coordinate time conversion unit 63, and sequentially starts from the time-series information recording unit 61 from the time-series information of the corresponding time. Read out and output to the time-series information output unit 58. When time information (the link information described above) is output from the user input information control unit 52 according to the user input, the coordinate time conversion unit 63 is bypassed, and the time information specified by the user is The time series information control unit 57 directly receives from the user input information control unit 52. Whether or not to use link information can be specified when the user selects input information.
[0428]
On the other hand, if it is determined in step S1501 that the command is automatic reproduction, the process proceeds to step S1505, and the corresponding time is determined based on the time information (Ts, Te) of the specific section from the time-series information specific section determination unit 59. The time series information of the section is read from the time series information recording unit 61 and output to the time series information output unit 58. In this embodiment, playback is started from time Ts, and playback is performed until the user explicitly indicates the end of playback.
[0429]
Although omitted in the flowchart of FIG. 48, it is also possible to reproduce time-series information from the time or the time interval when the user inputs time without creating a chart. That is, when receiving the reproduction command and time, the time-series information control unit 57 outputs time-series information corresponding to the time from the time-series information recording unit 61 to the time-series information output unit 58.
[0430]
Next, the operation of the chart creating unit 10 will be described with reference to the flowchart of FIG.
[0431]
If it is determined in step S1600 that a chart creation command has arrived from the time-series information control unit 57, the process proceeds to step S1601, and the time-series information specific section determination unit 59 receives time information (Ts, Te) of a specific section and user input. The information input time is acquired, and information (see FIG. 34) regarding the speech section within the specific section is acquired from the time-series information recording unit 61. Next, the processing proceeds to step S1602, and a speech chart as shown in FIG. 41 and FIG. 50 is created from the acquired speech section information, and the created chart information is output to the time-series information control unit 57. As described above, the time-series information control unit 57 performs control so that the message chart is displayed on the display screen of the display unit 54.
[0432]
An example of the chart shown in FIG. 50 will be described. The size (W, H) of the pop-up window for displaying the chart according to the time information of the specific section output from the time-series information specific section determination unit 59, the position (x0, y0) of the screen indicating the speech status, and the size (Wc, hc) is determined. In this figure, speech sections of conference participants A, B, C, and D are expressed as rectangles.
[0433]
Note that when the distribution (probability) information related to the relative time as described above is also provided from the time-series information specific section determination unit 59, the probability density information is displayed near the time axis as shown in FIG. A graph 80 is displayed. In this case, the distribution (probability) information of the start time of the specific section of the time series information corresponding to the user input information is output together with the statement chart of the specific section. Reflecting this distribution (probability) information on the chart helps the user to reproduce the time-series information from which time.
[0434]
In the example of FIG. 51, it is shown that the part of the first speech section of the speaker B has the strongest correspondence as the start time of the time series information corresponding to the user input information at the input time indicated by the triangle mark 81. Will be.
[0435]
Next, the operation of the coordinate time conversion unit 63 for converting the information on the designated coordinate position into time information on the chart by the user will be described with reference to the flowchart of FIG.
[0436]
That is, the coordinate time conversion unit 63 acquires the coordinate information from the user input information control unit 2 in step S1700, proceeds to the next step S1701, acquires the display position information of the chart from the time series information control unit 57, and sets the coordinates. Calculate the corresponding time. In step S1702, the calculated time is sent to the time-series information control unit 57.
[0437]
FIG. 50 is an explanatory diagram of time calculation in the coordinate time conversion unit 63. If the coordinates (x0, y0) are the origin of the chart, the coordinates of the designated location are (xp, yp), the display time is [Ts, Te], and the display width on the screen in that time interval is wc, the designated time Tp Is
Tp = Ts + (Te−Ts) × (xp−x0) / wc
It becomes.
[0438]
In the above-described embodiment of the information storage / playback apparatus, the example in which the situation when the user inputs is a personal memo or a presentation has been described. However, other situations can be used.
[0439]
For example, meeting ups and laughs, and actions of a specific person are regarded as one situation information or user input information, and there is a highly correlated relationship between the time and the recording position of time series information related to it, You may apply to the information storage / reproducing apparatus of this invention.
[0440]
In the embodiment of the information storage / playback apparatus, the specific section (Ts, Te) is determined from the relationship between the time of memo input and the start time of time series information. However, the presence of memo input time and time series information exists. A specific section (Ts, Te) may be determined in the section.
[0441]
In the above-described information storage / playback device embodiment, the specific section is determined from the input time of the user input information and the relative time information held by the time-series information specific section determination unit. As in the case of the embodiment of the information storage device, the time series corresponding to the user input information is also taken into account, for example, the speech section information, which is the processing result of the time series information processing section as the condition matching section detection section. Of course, a specific section of information may be determined.
[0442]
Further, as in the case of the above-described information storage device embodiment, a sensor is provided, and a condition matching section detecting means for detecting a section in which the detection output information of the sensor matches a predetermined condition set in advance is provided. Of course, the specific section of the time series information corresponding to the user input information may be determined in consideration of the section information detected by the condition matching section detecting means.
[0443]
In addition, the chart has been described with respect to an example of a utterance chart that displays a utterance section as a condition matching section. However, for example, a condition matching section such as a laughter section or a clap section of a conference participant is displayed as a chart. But of course.
[0444]
[Other Embodiments of Information Storage / Reproduction Apparatus]
In presentation-type meetings such as those in the embodiment described above, when each conference participant takes notes individually, after viewing the contents of the presentation, enter the notes about the viewed contents, and write the notes. When the process is completed, the user repeats the operation of viewing the contents of the presentation again and taking notes on the new contents viewed.
[0445]
In the embodiment of the present invention, the user input information is detected by the user input information detecting means as a lump of a series of related input information such as continuous pen strokes and recorded in the user input information storage unit 53. . The break of user input information detected as a lump is detected, for example, when the user input is interrupted for a predetermined time, for example, 15 seconds or more.
[0446]
Therefore, in the case of presentation-type conference, in the case of user input information such as memo input for each conference participant as described above, time series information corresponding to the input time of user input information specified by the user at the time of playback The start time (start time of a specific section) often exists after the input time of user input information immediately before the specified user input.
[0447]
This was confirmed experimentally as follows. That is, as shown in FIG. 53A, the time interval from the input time Tp of the user input information designated for reproduction to the input time Tb of the user input information immediately before is X, and the designated user input information When the time interval from the input time Tp to the start time of the time-series information related to the specified user input information is set to Y, the relationship between the two is examined. The result shown in FIG. 53 (B) is obtained. .
[0448]
In FIG. 53B, the straight line L is a straight line of Y = X. From FIG. 53B, it can be understood that the start time of the time-series information related to the specified user input information is mostly after the input time of the immediately preceding user input information.
[0449]
In this case, the input time Tb immediately before the user input information indicates at least the input start time of the user input information. However, the above applies even when the input time Tb of the immediately preceding user input information is the input end time of the user input information. Further, a time between the user input start time and the end time can be used as a reference.
[0450]
By the way, in the time series information specific section determination unit 59 of the embodiment of the information storage / playback apparatus described above, the input of user input information designated by the user for the playback section of the time series information and the specific section which is the chart display section. Based on the time alone, it was decided with reference to the input status.
[0451]
For this reason, depending on the input time of user input immediately before the specified user input information, in the above-described embodiment, the specified section includes an unnecessarily long section. It may take time. That is, when the input time of the previous user input is included in the specific section, the time series information before the input time of the user input is related to the specified user input information as described above. This is because there are almost no cases.
[0452]
In view of such a situation, the embodiment described below determines a specific section related only to specified user input information by considering the input time of the user input immediately before the specified user input information. Is performed with high accuracy.
[0453]
FIG. 54 is a block diagram showing each processing function unit in a block form in the embodiment of the information storage / playback apparatus in this case. The functional block diagram of FIG. 54 is different from the functional block diagram of the above-described embodiment described with reference to FIG. 30 in that a specific section correction unit 64 is provided for the time series information specific section determination unit 59. Different.
[0454]
In this embodiment, the specific section determined by the time-series information specific section determination unit 59 is determined by the specific section correction unit 64 according to the input time of the user input information immediately before the specified user input information. To correct. In the case of this embodiment, when the user input information is the memo mode, it is detected from the input situation and a correction operation as described later is performed. In this embodiment, when the input state of the user input information is in the presentation mode, the specific section correction unit 64 does not execute the correction operation, and the specific section determined by the time series information specific section determination unit 59 is used as it is. Output.
[0455]
FIG. 55 is a diagram for explaining the correction operation in the specific section correction unit 64. In the example of FIG. 55, the input time of the user input information designated by the user is Tp, and as described above, the time-series information specific section determining unit 59 uses the input status at the time of this user input information. , The specific section of the start time Ts and the end time Te is determined.
[0456]
The specific section correction unit 64 determines whether or not the input time Tb of the user input information immediately before the specified user input information is within the specific section determined by the time series information specific section determination unit 59. As shown in FIG. 55A, when the input time Tb of the user input information immediately before the specified user input information is within the specific section determined by the time-series information specific section determination unit 59, The specific section is corrected so that the specific section starts from the input time Tb of the previous user input information.
[0457]
As shown in FIG. 55 (B), when the input time Tb of the user input information immediately before the designated user input information is before the specific section determined by the time-series information specific section determination unit 59. The specific section determined by the time-series information specific section determination unit 59 is used as it is.
[0458]
As a result, in the memo mode, the specified user input information is excluded by eliminating the time-series information portion that is assumed to be irrelevant to the specified user input information before the user input information immediately before the specified user input information. It is possible to reproduce the time-series information portion corresponding to exactly.
[0459]
In order to enable the above-described correction operation in the specific section correction unit 64, not only the above-described processing operation of the specific section correction unit 64 is necessary, but also in this embodiment, a user input information control unit 52 and the time series information specific section determining unit 59 are different from the above-described embodiment described with reference to FIG.
[0460]
The processing operations of these units will be described below using flowcharts. First, the operation of the user input information control unit 52 will be described with reference to the flowchart of FIG. In this case, the input time of the user input information is, for example, an input start time.
[0461]
First, in step S1800, when the user input information control unit 52 receives an instruction to select any of the user input information displayed on the screen of the display unit 54 through the user input unit 51, the process proceeds to the next step S1801. In this embodiment, user input information is selected by surrounding the displayed user input information with, for example, an electronic pen. The user input information control unit 52 determines the enclosed area and searches the user input information recording unit 53 for user input information existing in the area.
[0462]
In step 1301, the input time Tp of the user input information instructed to select, the status information, and the input time Tb of user input information immediately before the user input information instructed to select are acquired from the user input information recording unit 53. To do. Then, the process proceeds to the next step S1802, and an inquiry message is displayed to the user as to whether the user input is for automatic reproduction of time-series information or creation of a chart to be described later.
[0463]
In response to this, the user makes a selection input for automatic reproduction or chart creation, and accepts this user selection input in step S1803. In the next step S1804, it is determined whether automatic reproduction is selected or chart creation is selected.
[0464]
If automatic playback is selected, the process proceeds to step S1805, where the time series information specifying section determination unit 59 is instructed to input the input time Tp of the instructed user input information, the status information, and the instructed user. Information on the input start time Tb of the user input information immediately before the input information and an automatic reproduction command are sent, and an automatic reproduction command is sent to the time-series information control unit 57.
[0465]
If chart creation is selected, the process proceeds to step S1806, where the time series information specific section determination unit 59 is instructed to input the input time Tp of the instructed user input information, the situation information, and the instructed user. Information on the input time Tb of the user input information immediately before the input information and a chart creation command are sent, and a chart creation command is sent to the time-series information control unit 57.
[0466]
In the above description, as the input time Tb of the immediately preceding user input information, not only the input start time but also the input end time can be used. In addition, as the input time of the user input information immediately before, a time between the input start time and the input end time (for example, an intermediate time between the two) may be used.
[0467]
Also, as the input time Tp of the specified user input information, not only the input start time but also the input end time or an intermediate time between them may be used. Depending on how they are used, the time-series information specific section is determined. 44 and 45 used in the part 59 are different from each other.
[0468]
The input time Tp of the specified user input information and the input time Tb of the user input information immediately before the input time are any of the input start time, the input end time, and an intermediate time between them. Use is independent of each other, and any combination of the three types of time is possible.
[0469]
Next, the operation of the time-series information specific section determination unit 59 in this embodiment will be described with reference to the flowchart of FIG.
[0470]
First, in step S1901, the input time Tp, the situation, the input time Tb of the previous user input information, and a command are acquired from the user input information control unit 52. Next, the process proceeds to step S1902, and the specific section (Ts, Te) of the time series information corresponding to the user input information is calculated from the input time Tp of the user input information and the situation information as described above. .
[0471]
In step S1903, it is determined whether the input status related to the specified user input information is the memo mode or the presentation mode. If it is the memo mode, the process proceeds to step S1904, and the information on the input time Tb of the previous user input information and the information on the specific section (Ts, Te) are sent to the specific section correction unit 64.
[0472]
The specific section correction unit 64 performs correction processing on the determined specific section as described later, and returns information on the specific section of the correction result to the time series information specific section determination unit 59. In the next step S1905, the section determination unit 59 receives the corrected specific section information from the specific section correction unit 64. Then, the process proceeds to next Step S1906.
[0473]
If it is determined in step S1903 that the presentation mode is selected, the process jumps to step S1906 without performing steps S1904 and S1905 of correction by the specific section correction unit 64.
[0474]
In step S1906, it is determined whether the command acquired in step S1901 is an automatic playback command or a chart creation command. If the command is an automatic playback command, the process proceeds to step S1908, and the time-series information control unit 57 stores information on the specific section. (In this example, the start time Ts) and information about the input time Tp of the user input information are sent. If it is a chart creation command, the process proceeds to step S1907, and information on the specific section (start time Ts and end time Te) and information on the input time Tp of the user input information are sent to the chart creation unit 60. Thus, the processing routine of the time series information specific section determining unit 59 ends.
[0475]
Next, FIG. 58 is a flowchart showing a processing routine of the specific section correction unit 64.
[0476]
First, in step S2001, the input time Tb of the user input information immediately before the specified user input information and the information of the specific section (Ts, Te) are acquired from the time series information specific section determination unit 59. Next, proceeding to step S2002, the start time Ts of the specific section is compared with the input time Tb of the previous user input information to determine whether the start time Ts is before the previous input time Tb. To do.
[0477]
When the start time Ts is before the previous input time Tb, the process proceeds to step S2003, the start time of the specific section is changed and corrected from Ts to Tb, and the information of the specific section changed and corrected in the next step S2005 ( Tb, Te) is returned to the time-series information specific section determination unit 59.
[0478]
If the start time Ts is later than the previous input time Tb, the process proceeds to step S2004, the start time of the specific section remains Ts, and the information of the original specific section (in the next step S2005 ( Ts, Te) is returned to the time-series information specific section determination unit 59. Thus, the processing routine of the specific section correction unit 64 ends.
[0479]
In this embodiment, the time-series information specific section determining unit 59 determines the specific section by determining the situation, but it is not always necessary to determine the situation, only the input time Tp of the user input information. Even in the information storage / playback apparatus that determines a specific section from the above, by using the specific section correction unit 64, it is possible to obtain an effect of accurately extracting only the time-series information portion strongly related to the user input information. In particular, as described above, when the user input information is a situation where a personal memo is taken while viewing the presentation, the effect of providing the specific section correction unit 64 is significant.
[0480]
In the above description of the embodiment, the input time Tb of the user input information immediately before is used as it is, but a time obtained by adding or subtracting a certain amount of time (Δt) from the input time Tb may be used. Further, as described above, the improvement effect was confirmed whether the input time Tb was the time when the user input was started or the time when the input was completed.
[0481]
The effect of the specific section correction unit 64 varies depending on the result of the specific section of the time-series information specific section determination unit 59. That is, in the time-series information specific section determination unit 59, when the start time Ts of the specific section is set as a time that is relatively short before the input time Tp of the specified user input information, The effect is not significant. In that case, the section specifying accuracy of the time-series information specifying section determining unit 59 becomes the specifying accuracy.
[0482]
On the other hand, when the start time Ts of the specific section is set as a relatively long time before the input time Tp of the specified user input information in the time series information specific section determination unit 59. The effect of the correction by the specific section correction unit 64 is increased.
[0483]
【The invention's effect】
As described above, according to the information storage device of the present invention, for example, when conference information is stored, user input information, changes in sound and image, sensor detection signal, and user input are detected. The situation can be appropriately combined to identify an important section that is a section in which a characteristic event occurs.
[0484]
The identified important sections are stored with high quality while maintaining time series information such as voice information and image information, and the other sections are compressed with a high compression rate and stored in the time series information storage unit. Thus, even a memory with a small storage capacity can accumulate time-series information for a long time.
[0485]
In addition, since the timing and length of recording an audio signal or an image signal can be changed depending on the type of user input device such as a shared input device or a personal input device, the efficiency can be improved in various embodiments of the conference recording device. Thus, the information storage medium can be saved.
[0486]
Moreover, since the time-series information of the important section is stored with high quality, it is possible to grasp the information content appropriately and reliably when it is reproduced. Therefore, for example, when it is applied when recording and storing conference information with a large amount of information, the information necessary for appropriately grasping the content of the conference is stored with high quality as an important section. Even if it is reduced, the content of the meeting can be grasped properly.
[0487]
Further, by storing the user input information and the important section in association with each other, by specifying the user input information, it is possible to easily reproduce the audio information and the image information of the corresponding important section.
[0488]
Further, according to the information storage / playback apparatus of the present invention, when time series information such as a meeting and a memo are recorded together and the time series information corresponding to the memo is reproduced later using the memo as a clue, the memo is input. Using this situation, time-series information at an appropriate location can be automatically reproduced. This makes it possible to reproduce appropriate time-series information regardless of whether the person is a participant in the conference, the person who entered the memo, or the person who is not.
[0489]
In addition, it is also possible to know the speech status of conference participants in an appropriate range using the status at the time of memo input. Therefore, if it is a participant in the conference, it is easy to recall and determine the part to be reproduced. Further, even if the person is a non-participant in the conference, it is possible to easily guess the part to be reproduced.
[0490]
Further, when the probability information is presented regarding the section to be reproduced, it is possible to obtain certainty about whether necessary information can be obtained in the reproduced section.
[0491]
Further, by correcting the specific section based on the input time of the user input information immediately before the specified user input information, it is possible to accurately specify the section of time series information closely related to the specified user input information. For this reason, it becomes possible to search and acquire desired information efficiently. For example, according to the present invention, it is possible to make the work efficient when creating minutes of meetings, summarizing talks such as lectures, and the like.
[Brief description of the drawings]
FIG. 1 is a functional block for explaining a first embodiment of an information storage device according to the present invention;
FIG. 2 is a diagram illustrating an overview of a system to which an embodiment of the present invention is applied.
FIG. 3 is a diagram for explaining an audio level detection operation of a condition matching section detection unit according to the first embodiment.
FIG. 4 is a diagram illustrating pen stroke input timing, a condition matching section, and an important section in association with each other in the first embodiment.
FIG. 5 is a diagram illustrating an example of setting elements for an important section according to an input situation.
FIG. 6 is a diagram illustrating pen stroke input timing, a condition matching section, and an important section in association with each other in the first embodiment.
7 is a diagram for explaining FIG. 6; FIG.
FIG. 8 is a diagram showing a flow of operations during information recording according to the first embodiment.
FIG. 9 is a flowchart illustrating an example of a processing routine of a condition matching section detection unit according to the first embodiment.
FIG. 10 is a flowchart illustrating an example of a processing routine of a user input information detection unit according to the first embodiment.
FIG. 11 is a flowchart illustrating an example of a processing routine of a user input device specifying unit in the first embodiment.
12 is a diagram illustrating pen stroke input timing, an important section, and a storage state of a time-series storage unit in association with each other according to the first embodiment. FIG.
FIG. 13 is a diagram illustrating a storage state for one pen handwriting of correspondence relationship information stored in a correspondence relationship storage unit according to the first embodiment.
FIG. 14 is a flowchart illustrating a part of an example of a processing routine of a correspondence relationship storage unit in the first embodiment.
FIG. 15 is a flowchart showing a part of an example of a processing routine of the correspondence relationship storage unit in the first embodiment.
FIG. 16 is a flowchart of an example of a processing routine of a time-series information storage unit in the first embodiment.
FIG. 17 is a flowchart of an example of a processing routine of a time information storage unit in the first embodiment.
FIG. 18 is a diagram illustrating an example of a storage structure of a time information storage unit in the first embodiment.
FIG. 19 is a diagram illustrating another example of the storage structure of the time information storage unit in the first embodiment.
FIG. 20 is a diagram illustrating a flow of operation during information compression in the first embodiment.
FIG. 21 is a flowchart illustrating an example of a processing routine of a compression unit according to the first embodiment.
FIG. 22 is a diagram for explaining the comparison between the pre-compression storage state and the post-compression storage state of the image information stored in the time-series information storage unit in the first embodiment.
FIG. 23 is a functional block for explaining a second embodiment of the information storage device according to the present invention;
FIG. 24 is a diagram showing a flow of operations during information recording according to the second embodiment.
FIG. 25 is a diagram for explaining storage contents of a temporary storage unit according to the second embodiment;
FIG. 26 is a part of a flowchart of an example of a processing routine of a temporary storage unit according to the second embodiment.
FIG. 27 is a part of a flowchart of an example of a processing routine of a temporary storage unit according to the second embodiment.
FIG. 28 is a diagram for explaining a system to which an embodiment of the information storage / playback apparatus according to the present invention is applied;
FIG. 29 is a diagram showing a status setting screen in the embodiment of the information storage / playback apparatus according to the present invention;
FIG. 30 is a functional block diagram of an embodiment of an information storage / playback apparatus according to the present invention;
FIG. 31 is an overview of an embodiment of an information storage / playback apparatus according to the present invention;
FIG. 32 is a flowchart for explaining the operation of the time-series information processing unit in the embodiment of the information storage / reproduction device according to the present invention;
33 is a diagram for explaining the operation of the time-series information processing unit in FIG. 32;
34 is a diagram showing a data structure of a utterance situation of a processing result of the time-series information processing unit in FIG. 32. FIG.
FIG. 35 is a flowchart for explaining the recording operation of the user input information control unit in the embodiment of the information storage / playback apparatus according to the present invention;
36 is a flowchart of a user input information cut-out operation in the user input information control unit of FIG. 35;
FIG. 37 is a diagram for describing a user input information cut-out operation in the user input information control unit of FIG. 35;
FIG. 38 is a diagram showing an example of a data structure recorded in a user input information recording unit in the embodiment of the information storing / reproducing apparatus according to the present invention;
FIG. 39 is a flowchart of the operation of the situation specifying unit in the embodiment of the information storage / playback apparatus according to the present invention;
FIG. 40 is a flowchart of another example of the operation of the situation specifying unit in the embodiment of the information storing / reproducing apparatus according to the present invention;
FIG. 41 is a diagram showing an example of a statement chart in the embodiment of the information storage / reproduction device according to the present invention;
FIG. 42 is a diagram showing a flowchart of the operation at the time of recording by the user input information control unit in the embodiment of the information storage / reproduction device according to the present invention;
FIG. 43 is a diagram showing a flowchart of the operation of the time-series information specific section determination unit in the embodiment of the information storage / playback apparatus according to the present invention.
FIG. 44 is a diagram for describing information for determining a relative time for determining a specific section possessed by a time-series information specific section determining section in the embodiment of the information storage / reproducing apparatus according to the present invention;
FIG. 45 is a diagram showing an example of relative time for determining a specific section possessed by a time-series information specific section determining section in the embodiment of the information storage / reproducing apparatus according to the present invention;
FIG. 46 is a diagram illustrating an example of a frequency distribution of relative time with respect to an input time point of user input information in which a start time point of a specific section of time series information corresponding to the user input information exists.
FIG. 47 is a diagram illustrating, for each situation, an example of a frequency distribution of relative time with respect to an input time point of user input information in which a start time point of a specific section of time-series information corresponding to the user input information exists.
FIG. 48 is a diagram showing a flowchart of the operation at the time of reproduction of the time-series information control unit in the embodiment of the information storage / reproduction device according to the present invention.
FIG. 49 is a diagram showing a flowchart of the operation of the chart creating unit in the embodiment of the information storing / reproducing apparatus according to the present invention;
FIG. 50 is a diagram illustrating an example of an instruction by a user on a display example of a message chart.
FIG. 51 is a diagram illustrating an example in which a message chart and a frequency distribution of relative time with respect to an input time point of user input information in which a start point of a specific section of time series information corresponding to the user input information exists are displayed together; is there.
FIG. 52 is a diagram showing a flowchart of the operation of the coordinate conversion unit in the embodiment of the information storage / reproduction device according to the present invention;
FIG. 53 is a diagram for explaining another embodiment of the information storing / reproducing apparatus according to the present invention;
FIG. 54 is a functional block diagram of another embodiment of the information storage / reproduction device according to the present invention.
FIG. 55 is a diagram for explaining an operation of a main part of another embodiment of the information storing / reproducing apparatus according to the present invention;
FIG. 56 is a flowchart of operation of a user input information control unit in another embodiment of the information storage / playback apparatus according to the present invention;
FIG. 57 is a flowchart of the operation of a time-series information specific section determination unit in another embodiment of the information storage / playback apparatus according to the present invention.
FIG. 58 is a flowchart of the operation of a specific section correction unit in another embodiment of the information storage / playback apparatus according to the present invention.
[Explanation of symbols]
1 Voice information input section
2 Image information input section
3 Condition matching section detector
4 Time series information storage
5 User input information input section
6 User input information detector
7 User input device identification part
8 Important section determination section
9 Correspondence storage unit
10 Compression unit
11 Time information storage
12 Playback unit
13 Playback instruction section
14 Display section
15 Control unit
16 Temporary storage
20 Electronic conferencing equipment
21 Display screen of monitor device
23 Electronic pen
24 cameras
25 Microphone
26 Voice signal analyzer
27 Storage media
28 Speaker
29 meeting participants
30 Small computer (personal electronic notebook device)
41 Shared electronic blackboard
42 video camera
43 Microphone
51 User input section
52 User Input Information Control Unit
53 User input information recording section
54 Display
55 Time-series information input section
56 Time Series Information Processing Department
57 Time Series Information Control Unit
58 Time-series information output section
59 Time-series information specific section determination section
60 Chart generator
61 Time-series information recording part
62 Situation Identification Department
63 Coordinate time converter
64 Specific section correction section

Claims (7)

Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each User input information input means for receiving the user input information from a plurality of user input devices that output the user input information in a state in which an input situation can be specified ;
User input information detecting means for detecting said user input information input from the user input information input means,
The user input information detected by the user input information detection means has a function of specifying an input status of at least user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
And wherein the user input information detected by the user input information detecting means, based on said input state identified by said user input state specifying means, among the audio information or image information from the time-series information input means A specific section for determining a section of time-series information in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A determination means;
A specific section determination device for time-series information comprising: Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each User input information input means for receiving the user input information from a plurality of user input devices that output the user input information in a state in which an input situation can be specified ;
User input information detecting means for detecting said user input information input from the user input information input means,
The user input information detected by the user input information detection means has a function of specifying an input status of at least user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
And wherein the user input information detected by the user input information detecting means, based on said input state identified by said user input state specifying means, among the audio information or image information from the time-series information input means A specific section for determining a section of time-series information in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A determination means;
Time series information storage means for storing the audio information or the image information input from the time series information input means;
And means for compressing the data amount of the time series information audio information or image information stored in the storage means, before Kitoku at the interval determined by the constant interval determining means, and the other section, the compression ratio or compression A compression unit that changes the method and performs data compression of the audio information or the image information stored in the time-series information storage unit;
An information storage device comprising: Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each User input information input means for receiving the user input information from a plurality of user input devices that output the user input information in a state in which an input situation can be specified ;
User input information detecting means for detecting said user input information input from the user input information input means,
Wherein said user input information detected by the user input information detecting means, the user input information from at least the common input device, the user input information of the input-like status from the personal input device, a function of specifying each A user input situation specifying means comprising:
Time-series information input means for inputting audio information or image information;
And wherein the user input information detected by the user input information detecting means, based on said input state identified by said user input state specifying means, among the audio information or image information from the time-series information input means A specific section for determining a section of time-series information in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A determination means;
Time series information storage means for storing the audio information or the image information input from the time series information input means;
Correspondence storing the segment information indicating the pre Kitoku interval determined by the constant interval determining means, the correspondence between the storage locations in the time-series data storing means of the audio information or the image information corresponding to the section information Storage means;
And means for compressing the data amount of the time-series data storing means and the voice information or the image information stored in the region determined by the section information stored in the correspondence relation storage unit, and the other section The compression means for changing the compression rate or the compression method and compressing the audio information or the image information stored in the time-series information storage means,
An information storage device comprising: Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each User input information input means for receiving the user input information from a plurality of user input devices that output the user input information in a state in which an input situation can be specified ;
User input information detecting means for detecting said user input information input from the user input information input means,
The user input information detected by the user input information detection means has a function of specifying an input status of at least user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
And wherein the user input information detected by the user input information detecting means, based on said input state identified by said user input state specifying means, among the audio information or image information from the time-series information input means A specific section for determining a section of time-series information in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A determination means;
Temporary storage means for temporarily storing the audio information or image information;
Of the audio information or image information temporarily stored in the temporary storage means, before Kitoku audio information or image information of the determined period by the constant interval determining means, user input detected by the user input information detecting means Time-series information storage means for storing information corresponding to the information;
And section information of the determined period before Kitoku constant interval determining means, and correspondence storage means for storing correspondence between the storage locations in said temporary storage means of the audio information or image information corresponding to the section,
The means for compressing the data amount of the audio information or the image information stored in the time-series information storage means among the audio information or the image information stored in the temporary storage means, the correspondence relationship storage a section before being stored Kitoku determined by a constant interval determining means to the means, in the other section, by changing the compression ratio or compression method, compression means for performing data compression of the audio information or the image information When,
An information storage device comprising: Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each User input information input means for receiving the user input information from a plurality of user input devices that output the user input information in a state in which an input situation can be specified ;
User input information detecting means for detecting said user input information input from the user input information input means,
The user input information detected by the user input information detection means has a function of specifying an input status of at least user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
Time-series information input means for inputting audio information or image information;
And wherein the user input information detected by the user input information detecting means, based on said input state identified by said user input state specifying means, among the audio information or image information from the time-series information input means A specific section for determining a section of time-series information in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device. A determination means;
Time series information storage means for storing the audio information or the image information input from the time series information input means;
Display means for displaying the user input information detected by the user input information detection means on a display screen;
And the user input information, wherein said display information for specifying the display position on the screen of the user input information, for each of said user input information, before Kitoku the corresponding to the determined period by the constant interval determining means Correspondence storage means for storing the correspondence with the storage position in the time-series information storage means of audio information or image information;
Designating means for designating a display portion related to the user input information displayed on the display screen;
The audio information or the image information of the section specified based on the storage position in the time series information storage means stored in the correspondence storage means specified by the user input information specified by the specification means is reproduced. Reproduction means;
An information storage device comprising: Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each User input information input means for receiving the user input information from a plurality of user input devices that output the user input information in a state in which an input situation can be specified ;
User input information detecting means for detecting said user input information input from the user input information input means,
The user input information detected by the user input information detection means has a function of specifying an input status of at least user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
User who records and the user input information detected by the user input information detecting means, and information about the input time of the user input information, and said input status identified by the user input state specifying means with respect to the user input information Input information recording means;
Time-series information input means for inputting time-series information of audio information or image information;
Time series information recording means for recording the audio information or the image information input from the time series information input means;
Display means for displaying the user input information recorded in the user input information recording means;
Designation means for selecting and instructing one of the user input information displayed by the display means;
A specific section of the audio information or image information from the time-series information input unit based on at least information related to the input time and the input status of the user input information specified by the specifying unit Specific section determining means for determining a section of time-series information in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device ;
Time-series information reproducing means for reproducing the time-series information corresponding to the specific section determined by the specific-section determining means among the time-series information recorded in the time-series information recording means ;
An information storage / playback apparatus comprising: Whether the user input information is from the shared input device or the personal input device, including at least a shared input device shared by a plurality of users and a personal input device to which each user refers to each User input information input means for receiving the user input information from a plurality of user input devices that output the user input information in a state in which an input situation can be specified ;
User input information detecting means for detecting said user input information input from the user input information input means,
The user input information detected by the user input information detection means has a function of specifying an input status of at least user input information from the shared input device or user input information from the personal input device, respectively. User input status identification means;
A user who records the user input information detected by the user input information detection means, information related to the input time of the user input information, and the input status specified by the user input status specification means regarding the user input information Input information recording means;
Time-series information input means for inputting time-series information of audio information or image information;
Time series information recording means for recording the audio information or the image information input from the time series information input means;
Display means for displaying the user input information recorded in the user input information recording means;
Designation means for selecting and instructing one of the user input information displayed by the display means;
The user input information specified by the specifying means, at least, based on the information about the input time of said voice information or image information from the time-series information input means, be one that determines the specific section Specific section determining means for determining a section of time-series information in which the input status specified by the user input status specifying means is different depending on the shared input device or the personal input device ;
A specific section correcting means for correcting, based the determined period by the specific section determination unit, the input time of the user input information immediately before the user input information selected designated by the designation unit,
Time-series information reproducing means for reproducing the time-series information corresponding to the specific section determined by the specific-section determining means out of the time-series information recorded in the time-series information recording unit;
An information storage / playback apparatus comprising:

Images