JPH08235209A - Multi-media information processor - Google Patents

Abstract

PURPOSE: To retrieve the recording information of a first medium whose retrieval is relatively easy and to reproduce the recording information of a second medium corresponding to the time information of the recording information by managing multi-media information by the time information indicating the timewise relation of the recording information of plural media. CONSTITUTION: This processor is provided with an input part 1 for inputting information by the kinds of the media, a means 2 for adding the time information for indicating the timewise relation of the respective pieces of the information to the information of the medium kind, a storage part 3 for storing them, a retrieval means 4 for retrieving a storage position to start reproduction in the storage information of the medium of a first kind, a first information reproducing means 4 for performing the reproduction from the retrieval point and a second information reproducing means 4 for specifying the point of the time of starting the reproduction in the medium of a specified second kind based on the time information corresponding to the retrieval point of the storage information during the reproduction and reroducing the storage information of the medium of the second kind from the point of the time of starting the reproduction.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION The present invention relates to audio, moving images, still images,
The present invention relates to a multimedia information processing apparatus that compositely stores information in a plurality of types of media such as text, handwritten characters, and graphics, and that plays back the stored information for each media.

[0002]

2. Description of the Related Art In a multimedia information processing apparatus as described above, for example, when storing information about a lecture, it is difficult to reproduce the facial expression of the lecturer, the atmosphere of the venue, etc. by using the voice information alone. Image taken by the speaker with a video camera, still image presented by the speaker using an overhead projector (hereinafter abbreviated as OHP),
Information can be reproduced more accurately by storing the information in multiple media such as handwritten memos for the lecture contents.

[0003]

By the way, when searching and reproducing the stored information of the multimedia information processing apparatus,
If the information is monotonic and there are no salient features that can serve as a guide for the search, for example if it is audio information,
You have to search sequentially from the beginning of the information by fast-forwarding or skipping. Therefore, as the amount of information increases, it becomes difficult to find the necessary information, and it takes a long time to perform the search, and the search efficiency decreases.

[0004] For the purpose of easily retrieving information that is difficult to retrieve in this way, there has been disclosed an information device that retrieves voice information from character information or graphic information being reproduced without directly retrieving voice information ( JP-A-3-236089). However, for example, when the voice information is a conversation between two parties and the break of the conversation is clear by detecting a change in the voice, on the contrary, when the voice information has a salient feature that serves as a reference for retrieval. It is more efficient to search and reproduce character information, graphic information, etc. from voice information, but in the above conventional information equipment, the media to be searched and the media to be searched are fixed, No means for enabling a fluid search, such as searching for information that is difficult to search from information that has salient features that serve as a reference for searching, regardless of the medium is not presented.

The present invention has been made to solve the above problems, and search is compared by managing multimedia information with time information indicating temporal relationship of stored information of a plurality of media. With the object of providing a multimedia information processing apparatus which enables to retrieve stored information of a first medium which is easy to reproduce and to reproduce stored information of a second medium corresponding to time information of the stored information, thereby improving retrieval efficiency. To do.

[0006]

FIG. 1 is a diagram illustrating the principle of the present invention. In the figure, reference numeral 1 is a multimedia information input / output unit, which is an audio information input / output (I / O) device 1a including a microphone (input) and a speaker (output), and a video camera / CC.
D I / O device 1b for image information consisting of camera (input) and LCD (output) and keyboard / pen / mouse / scanner (input) and LCD (output). In addition, the first type of media and the first type
I / O device 1c that inputs the designation information of the second medium that should reproduce the information based on the reproduction information of the media of 1 type, and A / D converter 1d that digitally converts the analog information such as voice and image, 1f, 1h and D / A converters 1e, 1g, 1i for analog-converting digital information such as audio information and image information read from the secondary storage device 3 to be described later are provided, and the information is combined with these plural media. Input and output according to media type. The time information addition circuit 2 is an RTC (Real Time Closure) that can always operate by in-house power generation even when the system power is turned off.
ck) and the like, and time information indicating the temporal relationship of the multimedia information is added to the information of each medium input from the multimedia information input / output unit 1. This time information adding circuit 2, a secondary storage device 3 such as a hard disk for storing multimedia information together with time information, a break of a talk, a change of a speaker, a change point of an image, etc.
The search point indicating the storage position at which the reproduction of the stored information of the media of
From the search point of the storage information of the specified second type of media, based on the time information added to the storage information of the search point of the information being played back, while playing back the storage information of the media of the type A part of the second type of media such as a part where the silent state lasts for a predetermined time or more near the traced back, a turn of the speaker closest to the search point, or a point where the image largely changes near the traced point. A CPU 4 for reproducing stored information from an appropriate reproduction start point according to the characteristics of the stored information and a working memory 5 of the CPU 4 are connected to a multimedia information input / output unit 1 via a bus 6.

[0007]

The multimedia information processing apparatus of the present invention is such that information is compositely input by the I / O devices 1a, 1b and 1c of a plurality of media and digital information is output by the A / D converters 1d, 1f and 1h. The time information adding circuit 2 adds time information to the digitally converted multimedia information, and the multimedia information is stored in the secondary storage device 3 together with the time information.

FIG. 2 and FIG. 3 are conceptual diagrams of the storage status of multimedia information by the multimedia information processing apparatus of the present invention. Video, still image, voice, handwritten character (figure),
Each piece of information stored in a plurality of textual media and time-series is associated with time information.

FIG. 4 shows the status information (*) of each input device every one second.
**) management table. The state information is, for example, the start of a cluster in the case where the beginning and end of a voiced part of voice information using a microphone as an input means is one cluster,
Information about the end of the cluster, the number of the cluster, and the start of the cluster, the end of the cluster, and the number of the cluster in the case where the beginning and the end of the moving image information with the camera as the input means one cluster. , The coordinates of the pen in handwritten information using the pen as input means, the action information by the pen touch, the information about the state change due to the pen touch, the information about the key using the keyboard as the input means, and the mouse in the text information using the mouse as the input means. Coordinates, action information by mouse operation, information on state change by mouse operation, and the like.

The CPU 4 is the first designated by the I / O device 1c.
The storage information of the media of the type is read into the memory 5, and when the first type of the media is voice, breaks in the talk, transitions of the speaker, and the like, and in the case of the image, change points of the image, and the like. In the case of text, search points such as document names are searched, and the stored information from each search point is output from the I / O device 1a, 1b, or 1c, and the user can search the stored information of the first type of media by the search point. Search for. I / by user
When one search point is specified through the O device 1c and the second type of medium for which the information is to be finally reproduced is designated, the CPU 4 is added to the stored information of the second type of medium. The time information stored in the memory 5 is read out and compared with the time information of the specified search point so that the reproduction start time of the stored information of the second type medium is, for example, the same time as the search point or a fixed time from the search point. It is decided at the time of going back, etc., and the memory information of the second type of media from this playback start time is converted to analog by the D / A converter 1e, 1g or 1i and output from the I / O device 1a, 1b or 1c. To do.

[0011]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings showing its embodiments. The multimedia information processing apparatus of the present invention configured as shown in FIG. 1 is an I / O device for each of a plurality of media such as voice, image, text, and handwritten character.
1a, 1b, 1c are input in combination and analog information such as voice and image is digitally converted by A / D converters 1d, 1f, 1h, and RT
The time information adding circuit 2 including C and the like adds time information to the digitally converted multimedia information, and the multimedia information is stored in the secondary storage device 3 together with the time information. As shown in FIGS. 2 and 3, the multimedia information stored in the secondary storage device 3 is a moving image (continuous / intermittent), a still image, a sound (continuous / intermittent), a handwritten character (continuous / intermittent),
For each medium such as text (continuous / intermittent), it is stored in the secondary storage device 3 together with time information (time point on the time axis) indicating the temporal relationship of these information.

[0012] For example, at the lecture, the voice of the lecturer is recorded with a microphone, the appearance of the lecturer is photographed with a video camera, and the OHP still image used during the lecture is photographed with a CCD camera. Take notes with handwritten characters (figures) inside and enter the text of the related materials distributed with the keyboard or read with a scanner. The information of these multiple types of media is stored on the hard disk together with the time information added by the RTC.

FIG. 9 is a flow chart showing the procedure of information reproduction of the present invention. The CPU 4 reads the storage information of the first type medium designated by the I / O device 1c into the memory 5 (S1), and when the first type medium is voice, breaks in the talk and the speaker. Search points such as transition points of images, change points of images in the case of images, and document names in the case of text (S2), and store information from each search point in the I / O device 1a, 1b or 1c. Further output (S3), the user searches the stored information of the medium of the first type by the search point. When the user specifies one search point through the I / O device 1c and finally specifies the second type of medium for which information is to be reproduced (S4), the CPU 4
Reads the time information added to the storage information of the second type medium into the memory 5 and compares it with the time information of the specified search point to determine the reproduction start time of the storage information of the second type medium, For example, it is determined at the same time as the search point, when the search point goes back a certain time, and the like.
A converter 1e, 1g or 1i converts into analog and I / O device 1a,
Output from 1b or 1c (S5).

For example, after the lecture, when the user wants to hear the recording information of a certain topic, when the voice information is monotonous and has no remarkable feature,
The user does not directly search the recorded information, but first searches, for example, an OHP still image in which lecture contents are aggregated and is easy to search. The user searches the OHP still image output on the LCD for a still image that is closely related to the audio information to be played back, and the CPU uses the time information added to the still image,
For example, the audio information stored at the same time as this still image is taken out from the hard disk and reproduced by the speaker, and the user acquires the final objective audio information.

In this case, in order to retrieve the audio information to be finally reproduced, any one of the other types of media such as moving images, texts, and handwritten characters may be retrieved first.
As described above, by managing the storage information of a plurality of types of media by associating them with the time information, the second type of media that is difficult to search is based on the search results of the first type of media that is easy to search. The stored information of can be easily searched and reproduced.

The storage information of the first type of media is required to be easily searched. Therefore, the voice information that is generally difficult to search is, for example, when the voice information is not a monotonous voice in which one person talks with almost no breaks, and when a suitable interval is set at the change point of the topic, Alternatively, if a plurality of human voices are included, and if the plurality of human voices are voices that can be easily recognized, such as voices of men and women, the search is performed in the opposite manner to the case of the above-described embodiment. Since it becomes easier, the stored information of other types of media can be reproduced from the search result of the audio information.

In the following, each medium is the above-mentioned first.
When the type of media becomes, the examples of the search points that facilitate the search of the storage information of the first type of media are listed in the type of media. (1) Voice ・ Provides a means for detecting the volume and searches for a part where the silence level continues for a predetermined time or longer as a break of the talk ・ Provides means for detecting the pitch and talks about the point where the pitch changes by a predetermined level or more・ A voice recognition unit is provided, and at least one voice of a plurality of speakers is registered in advance, and only the beginning of the registered voice is searched. Search for only the beginning of the voice excluding the voice that exists ・ With the voice recognition means, register the keyword by voice input and search for the voice that matches the registered keyword ・ Mark in parallel with the storage of information Add points to information and search for marking points

(2) Moving image-Providing means for detecting the color and saturation of the image to search for a change point of the image where the color and saturation change by a predetermined level or more-Information on marking points in parallel with storage of information (3) Still image-Search the beginning of the stored information for each still image-Search the beginning of the stored information for multiple still images Is an example of a screen display. If the still image information is managed like an album and the number of still images that can be viewed on the screen at one time can be freely changed to one or more, the search efficiency is improved.

(4) Handwritten characters (figure) -A means for recognizing a group of handwritten information is provided to search for the start point of storage of a group of handwritten information. It is a conceptual diagram of the memory situation of a character. A unit for recognizing a group of handwritten information is, among handwritten characters written in a state as shown in FIG. 6, characters written in a range in which no other handwritten character is input within a predetermined distance around the handwritten characters, or As shown in FIG. 7, characters whose recording time is continuous, for example, “handwritten characters” are recognized as a group.

(5) Text: An important mark is attached to an appropriate position of text information (continuous text information) of one document, and the position where the important mark is attached is searched. The display color of the position is changed and the position where the display color is changed is searched. ・ In the text information of multiple documents (intermittent text information), the document name, subtitle, document save date, document size, etc. Search Using Elements FIG. 8 is an example of a screen display when searching for text information of a plurality of documents.・ Register a keyword and search for text that matches the registered keyword.

Further, when the stored information of the second type medium is reproduced, not only the reproduction from the search point and the simultaneous point in the stored information of the first type medium, but also a time point when a certain time is traced back from the search point May be set as the reproduction start time, and if the reproduction start time is determined according to the characteristics of the information of the second type of medium, the usability of the apparatus is further improved.

Below, other methods for determining the reproduction start time of the stored information of the second type of medium based on the search points will be listed in the medium type. (1) Voice ・ Provides a means to detect the volume and plays back to the break point of the talk that is closest to the search point and has a silence level that continues for a predetermined time or more ・ Provides a means to detect the pitch and is closest to the search point , Play back retroactively to the change point of the speaker whose pitch changes more than a predetermined level ・ Providing voice recognition means, pre-registering at least one voice of multiple speakers, and playing back only the registered voice・ Conversely to the above, only the voices other than the registered voices are played back ・ Voice recognition means is provided, and the keywords are registered by voice input so that they are closest to the search point and match the registered keywords. Play back to the audio that is played back.-Add marking points to the information in parallel with storing the information, and play back to the marking point closest to the search point. Do

(2) Moving image-Providing means for detecting the color and saturation of the image, and playing back to the change point of the image which is closest to the search point and whose color and saturation change by a predetermined level or more-Memory of information In parallel with this, a marking point is added to the information, and the trace is played back to the marking point closest to the search point. (3) Still image ・ One or more sheets from the beginning of the stored information of the still image closest to the search point. To play

(4) Handwritten characters (graphics) -A means for recognizing a group of handwritten information is provided, and the group of handwritten information that is closest to the search point is reproduced back to the start point of storage. As described above, the means for recognizing handwritten information is a means for recognizing handwritten information whose written time is continuous, or handwritten information in which no other handwritten information exists within a predetermined distance around it. May be any means for recognizing a group.

(5) Text: An important mark is added to an appropriate position of the text information of one document, and the data is reproduced back to the position where the important mark is closest to the search point. Change the display color at an appropriate position and play back to the position closest to the search point where the display color is changed.-Play one document including the search point among multiple documents.-Register a keyword. Every other, play back to the text that is closest to the search point and matches the registered keyword.

As described above, the multimedia information processing apparatus of the present invention uses the stored information of the first type of medium for searching and reproducing the stored information of the second type of medium. It is also possible to reproduce the storage information of the first medium and the storage information of the second medium searched using the storage information of the first medium in parallel.

The multimedia information processing apparatus of the present invention is not limited to a recording / playback dual-purpose machine, but may be a playback-only machine for playing back prerecorded multimedia information. Furthermore, the multimedia information processing device of the present invention
The I / O devices 1a, 1b, 1c may be integrated devices for input / output, or the input device and the output device may be separate devices.

[0028]

As described above, the multimedia information processing apparatus according to the present invention manages multimedia information with time information indicating the temporal relationship of stored information of a plurality of media, and is relatively easy to search. Since the stored information of the medium can be searched and the stored information of the second medium corresponding to the time information of the stored information can be reproduced, an excellent effect of high search efficiency can be obtained.

[Brief description of drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a conceptual diagram showing a storage status of multimedia information according to the present invention.

FIG. 3 is a conceptual diagram showing a storage state of multimedia information of the present invention.

FIG. 4 is a management table of time information / state information of the present invention.

FIG. 5 is a screen display example (still image) at the time of search.

FIG. 6 is an input screen for handwritten characters.

FIG. 7 is a conceptual diagram of a storage status of handwritten characters.

FIG. 8 is a screen display example (text / plural documents) at the time of search.

FIG. 9 is a flowchart showing a procedure of information reproduction of the present invention.

[Explanation of symbols]

 1 Multimedia information input / output unit 1a, 1b, 1c I / O device 1d, 1f, 1h A / D converter 1e, 1g, 1i D / A converter 2 Time information addition circuit 3 Secondary storage device 4 CPU 5 memory

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display location G11B 27/00 A 27/10 A

Claims (13)

[Claims] 1. A multimedia information processing apparatus for compositely storing information in a plurality of types of media and reproducing the information by media type, while inputting information in the media type,
An input unit for inputting information designating a medium for reproducing information, a unit for adding time information indicating temporal relation of each information to the information of the media type input from the information input unit, A storage unit that stores information together with time information, a point search unit that searches a search point indicating a storage position where reproduction should be started in the storage information of the first type of medium, and a first type from the searched search points. Specified by the information from the input unit based on the first information reproducing means for reproducing the stored information of the medium and the time information corresponding to the search point of the stored information being reproduced by the first information reproducing means. A reproduction start time point in the second type of media is specified, and a second reproduction time is determined from the reproduction start time point.
Multimedia information processing apparatus comprising: second information reproducing means for reproducing stored information of the media of the above types. 2. The multimedia information is stored in a plurality of types of media in a composite manner, and the multimedia type information, which is stored together with the time information indicating the temporal relation of each information, is reproduced in the media type. In a multimedia information processing apparatus, an input unit for inputting information designating a medium for which information is to be reproduced, and a point search for searching a search point indicating a storage position at which reproduction is to be started in the storage information of the first type of medium. Means, first information reproducing means for reproducing the stored information of the first type of medium from the searched search point, and time information corresponding to the search point of the stored information being reproduced by the first information reproducing means. Based on the information from the input unit, a reproduction start time point in the second type of medium specified by the information is specified, and a second type from the reproduction start time point is specified. And a second information reproducing means for reproducing stored information of the medium. 3. The second information reproducing means reproduces the storage information of the second type of medium from the time point corresponding to the search point at the same time point or a time point that is a predetermined time back from the time information corresponding to the search point. Claim 1 which is a means to do
Alternatively, the multimedia information processing device according to 2. 4. A means for inputting a keyword for storage information of a first type of medium is provided, wherein the point searching means stores storage information matching the input keyword in the storage information of the first type of medium. The multimedia information processing apparatus according to claim 1, which is a means for searching. 5. A means for inputting a keyword for the storage information of the second type of medium, and a means for searching the storage information of the second type of medium for storage information matching the input keyword are provided. 3. The multimedia information processing apparatus according to claim 1, wherein the second information reproducing means is means for reproducing the stored information of the second type of medium from the keyword stored closest to the search point. 6. A means for adding marking points to appropriate places in the storage information of the first type of medium, wherein the point searching means searches for the marking points added to the storage information of the first type of medium. The multimedia information processing apparatus according to claim 1 or 2, which is means for performing. 7. The storage information of the first type of medium is voice information, and means for detecting a silent portion which continues for a predetermined time or longer in the voice information is provided, and the point searching means is restarted after the silent portion. The multimedia information processing apparatus according to claim 1 or 2, which is a means for retrieving the beginning of the audio information. 8. The storage information of the second type of medium is audio information, and means for detecting a silent portion lasting for a predetermined time or more closest to a point traced back from a search point is provided.
3. The multimedia information processing apparatus according to claim 1, wherein said information reproducing means is means for reproducing from the beginning of the voice information restarted after the silent portion. 9. The storage information of the first type of medium is moving image information, and means for detecting a change in the moving image information is provided,
The multimedia information processing apparatus according to claim 1 or 2, wherein the point search means is means for searching a point where the moving image information changes. 10. The multi according to claim 1, wherein the storage information of the first type of medium is still image information, and the point searching means is a means for searching a leading time point of information storage of each still image. Media information processing device. 11. The storage information of the second type of medium is still image information, and the second information reproducing means is means for reproducing still image information having a time point closest to a search point as a storage head. The multimedia information processing apparatus according to claim 1 or 2. 12. The memory information of the first type of medium is handwritten information, and means for determining a set of handwritten information is provided, and the point search means is the first time point when the set of handwritten information is stored. The multimedia information processing apparatus according to claim 1, which is a means for searching for. 13. The storage information of the first type of medium is text information, and means for adding a mark to an appropriate portion of the text information is provided, and the point searching means searches for a mark added to the text information. The multimedia information processing apparatus according to claim 1 or 2, which is means for performing.