JP2005250317A - Information processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To appropriately and easily add an index to time series information in which picture and voice information or the like are time sequentially recorded. <P>SOLUTION: The scene of a user 12 operating a copy machine 10 is recorded by time series information including image information and voice information. The mouth of the user 12 who operates the apparatus (the copy machine)10 is photographed by a video camera 20 and user's uttering is detected by an uttering detection section 22 from the obtained image based on the shape of the user's mouth. An index is written by an index writing section 26 at the place (the time) of the time series information in which the scene of the user's operation is photographed corresponding to the user's uttering time. Moreover, based on the voice information during uttering is detected, voice recognition is conducted by a voice recognition section 28 and analyzed language information is converted into character information by a telop generating section 30 and written into the time series information by a telop writing section 32. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an information processing apparatus having a function of assigning an index to a predetermined point in time-series information such as images and sounds.

  A technique for providing an index for facilitating searching for time-series information (sometimes referred to as multimedia information) including at least one of image information and audio information, and possibly text information. It has been known. For example, in Patent Document 1 below, when recording time series information of a conference situation, the speech (utterance) of the attendee of the conference is acquired by voice information, the time when the speech is made is determined, and A technique for searching for the position (time point) of time-series information corresponding to is disclosed.

JP-A-11-53385

  The device described in the above-mentioned publication extracts speech based on voice information, and it may be difficult to separate speech and other speech (noise) and to identify the attendee who spoke.

  The present invention appropriately detects an utterance when assigning an index based on the utterance.

  According to the present invention, an utterance is detected based on the mouth shape of the person who made the utterance, and an index is assigned to a position (time point) on the time series at which the utterance was detected. Since the utterance is detected from the image information, the utterance can be detected without being affected by the surrounding noise.

  One aspect of the present invention is a device usability test apparatus. Recent office equipment and information equipment are designed to select functions while looking at the screen. When one of the displayed function groups is selected, more detailed settings and selection contents of the function groups are displayed. . This hierarchical structure is often multi-layered, and it is necessary to set a plurality of functions in order to execute one work. The usability test is to confirm whether the operation for selecting such a function is easy for the user to understand. In the apparatus for performing the test, information on the display screen of the device to be tested, image information obtained by photographing the user to be operated, and voice information obtained by collecting the user's utterance contents are recorded as time-series information. Further, the user's mouth shape is captured by the camera, and based on the mouth shape, it is detected whether the user is speaking or speaking. An index is written at the position (time point) of the time series information corresponding to the time point when the user speaks. By performing a search based on this index, access to time-series information is facilitated, and usability test analysis is facilitated.

  Further, voice recognition can be performed on voice information while the user is speaking, voice can be converted into character information, and this can be written in time-series information.

  Another aspect of the present invention is a recording device for presentations and meetings. Record attendees' utterances and utterances as time-series information, and later use them to create minutes of meetings. Also in this device, the speech of the person is detected from the mouth shape of the attendee, and an index is assigned to the time point of the time series information corresponding to the detected time point. By performing a search based on this index, access to time-series information is facilitated, which is helpful when confirming the contents of a meeting or the like later. In this apparatus, it is also possible to convert the utterance contents into character information and write it on the time-series information based on the voice information when the attendee is speaking. Further, since the mouth shape at the time of utterance is determined, the attendee who uttered can be identified, and this information can also be included in the index.

  Hereinafter, embodiments of the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings. FIG. 1 shows a schematic configuration of the present embodiment. This embodiment shows an example applied to a so-called usability test for verifying whether the operation and handling of the device can be performed smoothly. The device 10 that is the subject of the usability test is a copier, for example, and will be described below using the usability test of the copier as an example. The copying machine 10 requires various settings when performing copying, such as the number of copies, single-sided / double-sided selection, paper type selection, and enlargement / reduction ratio selection. In order to perform such settings, the copier 10 includes a user interface including a display unit and an input unit. This interface can be, for example, a touch panel sensor in which a display unit and an input unit are integrated. A display showing the selection / setting contents is displayed on the display section, and the user inputs his / her desired setting contents from the display section through the input section. This copier function selection and setting display are hierarchical. For example, when performing an enlarged copy, “select magnification” on the initial screen is selected, an arbitrary magnification is selected on the magnification selection screen changed by this selection, and predetermined settings are made. Further, a predetermined function is selected by sequentially displaying a setting screen such as tray selection for setting the paper type, double-sided copy instruction, and the like.

  As described above, in recent years, a copying machine has many functions, but an operation for selecting the function is complicated. The usability test verifies whether this operation can be performed appropriately, that is, whether the display content of the screen, the instruction content, etc. are appropriate.

  In the usability test, it is verified whether the user 12 can smoothly select and set the copy machine 10. For example, an operation content such as “copy 8 copies from a double-sided original to a single-sided original at a magnification of 70%” is given to the user (subject) 12, and the operation when the user is performing this operation Information on the display screen of the copying machine and its surroundings such as the history of the display screen, the state of the display screen, the state of the user, etc. is recorded in time series. This time series information is analyzed later to verify usability.

  Information on the screen of the copier 10, particularly the screen of the input interface, is input to the scan converter 14 and sent to the time-series information recording unit 16 as time-series image information, and a predetermined recording is performed by the time-series information writing unit 18. Written on a medium and stored. The state of the user 12 who is operating the copy machine 10 is photographed by the video camera 20, and sent to the time-series information recording unit 16 as image information and audio information and stored therein. The time series information of the surrounding situation taken by the copy machine 10 and the video camera can be reproduced later, and the reproduced image and sound are used for verification of usability. Further, from the copying machine 10, information indicating the contents of the operation performed by the user 12 is sent to the operation recording unit 22 and sequentially stored therein. The information recorded in the operation recording unit 22 is an operation flow according to the operation order, that is, a sequence of operations.

  The video camera 20 is arranged so as to photograph the mouth of the user 12. The photographed image information is sent to the utterance detection unit 22 where it is determined whether the user 12 is uttering words, that is, speaking from the shape of the user's mouth. For detecting the user's utterance from the mouth shape, for example, a technique described in Japanese Patent Laid-Open No. 2001-51693 can be used. When the utterance of the user 12 is detected, the index creating unit 24 instructs the time series information recording unit 16 to write the index at the time of the time series information corresponding to the time of the head portion of the utterance. . In the time series information recording unit 16, the index writing unit 26 assigns an index to a designated time point of the time series information stored in the storage medium.

  When the utterance of the user 12 is detected by the utterance detection unit 22, the speech recognition unit 28 performs speech recognition based on the speech information during the utterance period, and uses the utterance content as language information. Further, the content of the utterance that has become the language information is converted into text information converted into text information by the telop creation unit 30, and this telop is converted into a telop at the time of the utterance corresponding to the telop in the time-series information recording unit 16. Write. When the time-series information in which the telop is written is reproduced, the utterance content is displayed as characters on the image. The display time of the telop can be made to coincide with the actual utterance period, and when the content is long, it can be made longer than the utterance period for easy reading.

  In the above description, the content of the utterance is recorded as character information, that is, a telop is inserted into the image. However, it is also possible to write only the index and omit the telop. In addition, information recorded and recorded by the video camera 20 can be reproduced, and an index or the like can be written based on the reproduction information.

  In addition, the video camera 20 can be provided separately with a camera that captures only the user's mouth and a camera that captures a relatively wide range so that the user's operation can be recorded. Furthermore, recording the display screen of the copy machine 10 can be omitted. In this case, it is also preferable to take a picture of this operation screen with a video camera.

  The information processing apparatus of the above embodiment can be realized by causing a computer to operate with a predetermined program. The computer acquires information from the video camera 20, and the computer functions as an utterance detection unit 22, an index creation unit 24, a voice recognition unit 28, and a telop creation unit 30 according to a predetermined program. The time-series information recording unit 16 can be an external recording device or a recording device inside a computer such as a hard disk. The computer instructs the recording device to write an index according to the detection result of the detection unit 22.

  FIG. 2 shows a schematic configuration of another embodiment of the present invention. This embodiment assumes recording of a presentation, and the same components as those in the embodiment shown in FIG.

  The presenter 40 of the presentation prepares an image according to the contents of the presentation in advance, operates an information device such as the computer 42, and displays the prepared image on the presentation screen 44 sequentially in accordance with the progress of the presentation. To other attendees (hereinafter referred to as the audience) 46. This presentation image is sent to the time-series information recording unit 16 via the scan converter 14, and the video camera records the images and sounds of the attendees of the presentation. The video camera includes a video camera 20 </ b> A that captures the presenter 40 and video cameras 20 </ b> B and 20 </ b> C that capture the audience 46. Image information and audio information captured by the video cameras 20A, 20B, and 20C are stored in a predetermined storage medium by the time-series information recording unit 16. At this time, each of the image information of the video camera and the image information from the scan converter 14 can be recorded so as to correspond to one portion of one display screen divided into, for example, four. In this case, when reproducing the time series information, the four pieces of image information are simultaneously displayed on the display screen. In addition, each image information can be recorded separately so that one or a plurality of image information can be selected and displayed at the time of reproduction.

  The information acquired by each of the video cameras 20A, 20B, and 20C includes an utterance detection unit 22, an index creation unit 24, a voice recognition unit 28, and a telop creation unit 30 that have the same configuration as the embodiment shown in FIG. And sent to the time-series information recording unit 16. In the figure, the above-described configuration corresponding to the video cameras 20B and 20C is omitted. It is also preferable to separately provide a video camera that captures the entire state of the presentation venue.

  Index writing can be performed in parallel with time-series information writing, and once recorded time-series information is reproduced, and index and telop are written later based on the information being reproduced. Also good. The telop can be written corresponding to the image of the person who spoke. In this way, when a necessary image can be selected at the time of reproduction as described above, the telop of the person who speaks in the selected image can be displayed at the time of reproduction. In this manner, writing of indexes and telops for individual time-series information acquired by a plurality of video cameras can be performed independently for each time-series information, or can be written in common for all the time-series information. For example, an index based on an utterance for a certain attendee can be given only to time series information related to that attendee. Also, an index or the like can be written in the time series information of other attendees based on the speech of one attendee. Furthermore, an index or the like based on the utterances of a plurality of attendees can be written in one time-series information, or can be written in time-series information related to a plurality of attendees.

  When there are many attendees at a presentation or meeting, the attendees to be photographed can be limited for indexing. Further, the information on the presented image does not need to be obtained through the scan converter 14, and for example, the posted image can be captured and recorded by a video camera that captures the entire view of the venue. .

  In addition to presentations, where there are one or a small number of presenters and other participants listen to this presentation, so-called conferences where multiple participants share their opinions, and multiple lectures The above embodiment can also be applied to a symposium in which a person expresses an opinion and discusses with other participants based on the opinion.

  The embodiment of FIG. 2 can also be realized by operating a computer with a predetermined program in the same manner as the embodiment of FIG.

It is a block diagram which shows schematic structure of embodiment. It is a block diagram which shows schematic structure of other embodiment.

Explanation of symbols

  10 copy machine (equipment), 12 user (subject), 14 scan converter, 16 time series information recording unit, 18 time series information writing unit, 20 video camera, 22 utterance detection unit, 24 index creation unit, 26 index writing Part, 28 voice recognition part, 30 telop creation part, 32 telop writing part, 40 user (presenter), 42 computer (information equipment), 44 presentation screen, 46 audience.

Claims (9)

Records time-series information including image information obtained by photographing a user who operates the device according to the display on the display screen of the predetermined device, voice information obtained by collecting the user's utterance content, and image information on the display screen of the device. Time-series information recording means,
Utterance detection means for detecting the user's utterance based on the shape of the user's mouth in the image information obtained by photographing the user;
Index giving means for giving an index to a time point on the time series information corresponding to a time point when the utterance detecting means detects an utterance;
An information processing apparatus. The information processing apparatus according to claim 1, further comprising:
Voice recognition means for performing voice recognition on the voice information while the utterance detection means is detecting a user's utterance and converting the voice information into character information;
The time-series information recording unit writes the character information converted by the voice recognition unit at a time point on the time-series information at which an utterance corresponding to the character information is performed.
Information processing device. Time-series information recording means for storing time-series information including image information obtained by imaging attendees of a presentation or meeting, and voice information obtained by collecting the utterance contents of the attendees,
Utterance detection means for detecting the utterance of the attendee based on the mouth shape of the attendee in the image information of the attendee photographed;
Index giving means for giving an index to a time point on the time series information corresponding to a time point when the utterance detecting means detects an utterance;
An information processing apparatus. The information processing apparatus according to claim 3, further comprising:
Voice recognition means for performing voice recognition on the voice information while the utterance detection means is detecting an attendee's utterance and converting the voice information into character information;
The time-series information recording unit writes the character information converted by the voice recognition unit at a time point on the time-series information at which an utterance corresponding to the character information is performed.
Information processing device.   5. The information processing apparatus according to claim 3, wherein the index includes information for identifying an attendee who has made a speech related to the index. A function of acquiring image information of a user who operates the device according to the display on the display screen of the predetermined device, and detecting a user's utterance based on the shape of the user's mouth in the image information;
For the device that writes the time series information including the image information obtained by photographing the user, the voice information collected from the user's utterance content, and the image information of the display screen of the device, the time series information A function for instructing to write an index when the user's utterance is detected;
A computer-readable program that causes a computer to execute. Acquiring image information of a user who operates the device according to the display on the display screen of the predetermined device, and detecting the user's utterance based on the shape of the user's mouth in the image information;
For the device that writes the time series information including the image information obtained by photographing the user, the voice information collected from the user's utterance content, and the image information of the display screen of the device, the time series information Instructing to write an index when the user's utterance is detected;
An information processing instruction method. The ability to capture image information of attendees in a presentation or meeting and detect the attendee ’s utterance based on the mouth shape of the attendee in the image information;
The attendee's utterance of the time-series information is written to a device that writes time-series information including image information obtained by photographing the attendee and audio information obtained by collecting the user's utterance content. A function for instructing to write an index at the time of detection;
A computer-readable program that causes a computer to execute. Obtaining image information of the attendees in the presentation or meeting and detecting the attendee's utterance based on the mouth shape of the attendees in the image information;
The attendee's utterance of the time-series information is written to a device that writes time-series information including image information obtained by photographing the attendee and audio information obtained by collecting the user's utterance content. Instructing to write an index at the time it was detected;
An information processing instruction method.

Images