KR20010089293A - Method and apparatus to prepare listener-interest-filtered works - Google Patents

Abstract

An embodiment of the present invention is a method of presenting an audio or audio-visual work including: (a) a user selecting a speed contour from one or more speed contours stored in a database apparatus; (b) presenting the audio or audio-visual work at a playback device using the user selected speed contour to provide presentation rates; (c) during presenting, the user selecting another speed contour from the one or more speed contours stored in the database apparatus; and (d) continuing to present the audio or audio-visual work at the playback device using the user selected another speed contour to provide presentation rates.

Description

METHOD AND APPARATUS TO PREPARE LISTENER-INTEREST-FILTERED WORKS}

The presently known Time-Scale Modification (TSM) method allows digital record audio to be transformed so that the identified segmentation rate of the oral sentence, ie speech rate, can be converted dynamically during playback. Typical applications of such TSM methods include, but are not limited to, speed reading for craftsmen, speaking books, digital recording lectures, slide shows, multimedia presentations, and foreign language learning. In such an application, the listener can control the speech rate while playing the prerecorded speaker's sound, which is referred to as a "Listener-Directed Time-Scale Modification application (LD-TSM)". This allows the listener to slow down or speed up the segmentation rate and the information transfer rate of the prerecorded speaker's sound.As well known to those skilled in the art, the aforementioned LD-TSM application uses the TSM method. For example, it is possible to accelerate or decelerate voice or audio that must be provided for understanding at increased or decreased playback speeds, so that the listener can easily perceive the material through fast-forwarding, for example.

In a typical LD-TSM system, input from a listener can be specified in a number of ways. For example, input may be specified through the use of a key (button) press, a mouse action, or a voice command, all of which are hereinafter referred to as "key input". As a result, it is easy to understand that the LD-TSM system allows the listener to adjust the information transmission speed of the digital audio media according to the interest and the understanding speed.

As can be easily seen from the above, in order to optimally use the LD-TSM system, it is necessary to determine a method of interworking with the audio media and the listener providing the TSM. In particular, the actual information transmission speed selected by the listener is determined by the speaker's clarity, the listener's interest in a particular field, the familiarity of the listener in a particular field, whether the listener complies with the content, and the listener receives the content of the material. It depends on various factors such as one conventional amount of time.

Conventional methods of determining the listener's interest in parts of speech and / or audio are inherently inaccurate. In particular, these methods include detecting the fast-forward and rewind patterns of cassette tapes made by, for example, button presses. Using such a fast forward or rewind pattern causes various disadvantages. For example, the listener frequently alternates between fast-forward and rewind for a particular piece of audio material because it is not understood or information is not provided during fast-forward or rewind. In particular, when the playback position progresses, it may interrupt playback while progressing through the audio material, or provide a version of the audio material that is incomprehensible (e.g. chipmunk sound for speed increase). . As such, the method of determining the current listener's interest is of little use in determining the optimal information delivery rate.

As can be readily understood from the foregoing, there is a need in the art for a method and apparatus for determining listener interest for portions of voice, audio, and / or audio-visual processing. There is also a need in the prior art for a method and apparatus for providing a listener interest filtering tool (LIF tool) by reproducing voice, audio and / or audio-visual processing in accordance with the determination of listener interest.

The present invention relates to the field of voice, audio and audio-visual works. In particular, the present invention provides a method and apparatus for receiving a listener input related to the playback speed required for portions of voice, audio, and / or audio-visual processing, and a "Spped Contour" representing the listener input. ) Or "Conceptual Speed Association data structure". The listener input functions as a proxy for the listener's interest in voice, audio and / or audio-visual processing and / or the listener's understanding (and / or dictation) (hereinafter referred to as "listener's interest"). do. For example, if a listener wants to enjoy some more of the voice, audio, and / or audio-visual processing, is difficult to understand the part, or wants to transcrib information about the part, he or she would like to slow down. something to do. In particular, the present invention provides a method for reproducing speech, audio and / or sound-visual tools in accordance with velocity contours or concept velocity relationship data structures to create a "listener-interest-filtered work" (LIF tool). A method and apparatus are disclosed. LIF tools are useful for numerous applications, such as education, advertising, news delivery, entertainment, public safety broadcasting, and the like.

1 is a block diagram of a first embodiment of the present invention for generating velocity contours for audio or audio-visual processing.

FIG. 2 is a flow chart of the algorithm used in one embodiment of the speed contour generator shown in FIG.

3 graphically illustrates velocity contours for several different listening sessions of the same audio or audio-visual process.

4 shows a graphical representation of the generated rate contours of a user specified TSM rate or playback rate for several different listening sessions of the same audio or audio-visual process.

5 is a block diagram of a second embodiment of the present invention for generating velocity contours for audio or audio-visual processing, wherein user input and word maps of the audio or audio-visual processing are used to provide velocity contours. do.

6 shows a two-dimensional graph displaying sound waveforms and corresponding text for audio or audio-visual processing.

7 shows a display with text or audio-visual processing texturized.

8 is a block diagram of a third embodiment of the present invention for generating a Concept Velocity Relationship Data Structure (CSA DS) for audio or audio-visual processing.

9 is a flowchart of an algorithm used in one embodiment of the CSADS generator shown in FIG.

10 is a block diagram illustrating a fourth embodiment of the present invention for generating a LIF tool utilizing velocity contours in conjunction with audio or audio-visual processing.

11 is a block diagram of a fifth embodiment of the present invention for generating a LIF tool utilizing CSA data structures in conjunction with audio or audio-visual processing.

12 is a flowchart of an algorithm used in one embodiment of the TSM rate adjuster of FIG. 11 to provide a TSM rate or playback rate.

Summary of the Invention

Embodiments of the present invention readily meet the needs of the prior art described above, and provide an apparatus and method for determining listener interest for portions of voice, audio, and / or audio-visual processing, and a measure of listener interest. Provides a method and apparatus for developing a velocity contour or a conceptual velocity relation data structure. In addition, other embodiments of the present invention utilize a velocity contour or concept velocity relationship data structure to provide a listener interest filtering tool (LIF tool), in accordance with velocity contours or concept velocity relationship data structures, according to speech, audio and / or Or a method and apparatus for reproducing an audio-visual process.

One embodiment of the present invention includes affinity information used to obtain a time scale conversion (TSM) rate, and identifier information used to obtain an identifier of a portion of the audio or audio-visual processing associated with the TSM rate. A device for generating a speed contour, the apparatus comprising: (a) receiving user information and responsive to the identifier, portion, and TSM rate of the portion of the user input device (b) indicating the input of the portion of the audio or audio-visual processing; A timescale conversion system for generating a timescaled portion; (c) a timescale conversion monitor for generating a TSM rate and an identifier of the portion related to the TSM rate in response to the user information, the identifier of the portion, and (d) And a speed contour generator for generating a speed contour in response to the TSM rate and the identifier of the associated portion.

In another embodiment of the present invention, a concept velocity relationship data structure comprising affinity information used to obtain a TSM rate, and concept information used to obtain a concept identifier for a portion of audio or audio-visual processing associated with the TSM rate. A device for generating an apparatus comprising: (a) a timescale in response to an identifier, a portion, and a TSM rate of a portion of a user input device (b) that receives user information and directs input of a portion of audio or audio-visual processing; (C) a concept decoder generating a concept for the part in response to the part identifier and part; (d) a concept identifier relating to the TSM rate and the TSM rate, in response to the user information and concept. (E) generate a concept velocity relationship data structure in response to the time-scaled conversion concept monitor and (e) the TSM rate and the associated concept identifier. Is a device that includes a concept velocity-related data structure generator.

In another embodiment of the present invention, audio is associated with a velocity contour that includes affinity information used to obtain a TSM rate, and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate. Or an apparatus for reproducing audio-visual processing, comprising: (a) an input device indicating input of a portion of audio or audio-visual processing; (b) an identifier, a portion, and a timescaled portion in response to a TSM rate. (C) a playback apparatus for reproducing the time-scaled portion in response to the time-scaled portion, and (d) a time scale for generating the TSM rate in response to the velocity contours and the identifier of the portion. It is a device including a conversion rate determiner.

In another embodiment of the present invention, a concept velocity relationship (CSA) material comprising affinity information used to obtain a TSM rate and concept information used to obtain a concept identifier of a portion of audio or audio-visual processing associated with the TSM rate. A device for reproducing audio or audio-visual processing in association with a structure, the apparatus comprising: (a) an input device indicating input of a portion of the audio or audio-visual processing (b) in response to an identifier, portion, and TSM rate of the portion; And (c) a reproducing apparatus for reproducing the timescaled portion, in response to the timescaled portion, in response to the identifier and portion of the portion, Concept Decoder Generating Concepts (e) A field containing a timescale transformation concept lookup that generates a TSM rate in response to the concept and concept velocity relation data structures. A.

In another embodiment of the present invention, there is provided a method of generating a listener interest filtering tool for audio or audio-visual processing, comprising: (a) generating at least one average velocity contour for at least one audio or audio-visual processing for at least one user category; (B) converting one or more average velocity contours into one or more conceptual velocity relationship data structures; and (c) forming a conceptual velocity relation data structure from which the listener interest is filtered from the one or more conceptual velocity relationship data structures. It is a way. In this embodiment, a listener interest filtered filtered concept velocity relationship data structure is generated using a listener interest filtered concept velocity relationship data structure, or the listener interest filtered concept velocity relationship data structure is used as a listener interest filtered velocity contour. Converting, or generating listener interest filtered audio or audio-visual processing using listener interest filtered rate contours, or using any of the foregoing, listener interest, desired transmission rate, or audio or audio-visual Determining a listener's affinity for the concept or material of the process.

Embodiments of the present invention receive a listener input related to the playback rate required for portions of voice, audio, and / or audio-visual processing, and include a "velocity contour" or "concept velocity relationship data structure" representing the listener input. It relates to a method and apparatus for developing. The listener input acts as a proxy for the listener's interest in voice, audio and / or audio-visual processing and for the listener's understanding (hereinafter referred to as "listener interest"). For example, if the listener wants to enjoy some more of the voice, audio, and / or audio-visual processing, or if it is difficult to understand that part, they will want to slow down. In another embodiment of the present invention, a speech, audio and / or sound-visual process is played in accordance with a velocity contour or a conceptual velocity relationship data structure, thereby creating a new processing tool called a "listener interest filtering tool (LIF tool)." works and a method and apparatus for generating the same). As detailed below, the LIF tool is useful in numerous applications such as education, advertising, news delivery, public safety broadcasting, and the like.

Generation of velocity contours and concept velocity-related data structures

According to the invention, in a first embodiment of the invention a speed contour is generated, which speed contour is optionally stored for later use.

1 shows a block diagram of a first embodiment 1000 of the invention for generating speed contours of audio or audio-visual processing. As shown in FIG. 1, embodiment 1000 includes a user interface (UI) 100 that receives input from a user. The UI 100 provides an output signal indicative of an input from a user. The user input is interpreted by the user input processor / playback controller (UIP / PC) 200 of the embodiment 1000 to indicate the next option selected by the user. (a) Selection of a file to play, the file is a specific audio or audio-visual process (the selected file may be directly input into the embodiment 1000 or may be a file stored in the embodiment 1000). (b) Start playback of the selected file (c) Halt playback of the selected file (d) Pause playback of the selected file (e) A portion of the audio or audio-visual processing being played Specify the parameters Interval_Size, Speed_Change_Resolution, Average_or_Overwrite, and Log_Repeats used in the methods described below for converting the TSM rate, ie, playback speed, or generating speed contours. There are many devices known to those skilled in the art for receiving input from a user. For example, (a) pressing a key, (b) operating a switch on a mouse, (c) moving a slider or position indicator, and (d) a device for detecting a user sound command, and, in response, a keypress, a switch action, It is known to those skilled in the art that there are many commercially available devices, such as a device for sending a digital data indicative of movement of a slider or position indicator, or a sound command or the like to a processing device.

The UIP / PC 200 receives input from the UI 100, (a) converts the user command into a numerical value, (b) interprets the user command to set parameter values, and generates velocity contours. Data from audio or audio-visual processing by controlling the use, conversion, or overriding, and (c) sending requests for stream data to digital storage 75 or other audio or audio-visual data sources. Instructs access and loading of the stream (to perform playback control). In digital storage 75, UIP / PC 200 may request access to a file of digital data representing audio or audio-visual processing stored in a file system on the device. In order to direct access and loading of the data stream from the audio or audio-visual processing, the UIP / PC 200 locates the digital samples representing the user input and audio or audio-visual processing stored in the digital storage 75. Interpret to calculate the playback position relative to the location of the selected file in the particular sample.

Digital storage 75 receives the following inputs: (a) a request for stream data from UIP / PC 200, and optionally (b) a timescaled output from TSM subsystem 200, And optionally (c) a stream of data representing the velocity contours from velocity contour generator 500. Digital storage 75 produces the following output: (a) a data stream representing audio or audio-visual processing; (b) a stream of location information, such as a location in a file of the output data stream. There are many methods known to those skilled in the art utilizing digital storage devices, such as hard disk drives for storing and retrieving general purpose data.

Audio or audio-visual processing is typically stored in digital form on digital storage 75. For example, there are many commercially available devices known to those skilled in the art for use as digital storage devices such as CD-ROMs, digital tapes, magnetic disks and the like. Digital storage 75 receives data requests from UIP / PC 200 according to methods known to those skilled in the art to provide a stream of digital samples indicative of audio and / or audio-visual processing. In an alternative embodiment, audio or audio-visual processing is stored in analog storage in analog form. In an alternative embodiment, a stream of analog signals is input to a device for converting unshown analog samples into digital samples. It is possible to receive an input analog signal such as a voice signal, sample an analog signal at least Nyquist Rate, provide a stream of digital signals, and convert it back into an analog signal without loss of fidelity. There are many commercially known devices known to those skilled in the art. The digital sample is then sent to the TSM subsystem 300.

TSM subsystem 300 receives as input: (a) a stream of samples representing portions of audio or audio-visual processing from digital storage 75 (b) such as, for example, a sample count or time value; Stream location information from the digital storage device 75 used to identify the location of the data stream of the sample to be sent. (C) The required TSM rate, or playback rate, from the TSM monitor 400. The output from the TSM subsystem 300 is applied to the following inputs: (a) Digital Analog Converter / Audio and / or Audio-Visual Playback (DA / APD) 600, and optionally (b) if necessary. Digital storage for storage of time-scaled output, ie LIF tools (75). DA / APD 600 is a device known to those skilled in the art for receiving digital samples and establishing audio or audio-visual processing. According to the present invention, the output of the TSM subsystem 300 is digital indicating that the playback rate provided from the TSM monitor 400 provides audio or audio-visual processing for providing feedback to the user regarding user input TSM rate requirements. A stream of samples. In particular, the user can convert the TSM rate or playback speed by listening to the time-scaled output and providing further input using the UI 100. Specifically, if the user wants to accelerate or decelerate a portion of the audio or audio-visual processing that has just been played, the user can use the UI 100 to provide input to rewind and convert the audio or audio-visual processing to the desired portion. Playback can be resumed at the specified TSM rate or playback speed. In this method, the user determines the desired TSM rate, or playback speed, for each part of the audio or audio-visual process. TSM subsystem 300 transforms an input stream of data according to known TSM methods to provide a stream of samples representing the timescaled signal as output. In one preferred embodiment of the present invention, the TSM method used is the method disclosed in U.S. Patent No. 5,175,769 ('769 patent), which is referred to by reference below, and the inventor of the present invention is also a co-inventor of the' 769 patent. As will be readily appreciated by those skilled in the art, whenever the embodiment 1000 provides playback for audio-visual processing, the TSM subsystem 300 accelerates or decelerates the visual information to match the audio of the audio-visual processing. Let's do it. In order to do this in a preferred embodiment, the video signal is " frame-subsampled " or " frame-replicated " according to one of many methods known in the art, so that the audio-visual Maintain synchronization between the audio and visual parts of the process. Thus, if audio is to be made faster and sampling is requested at a faster rate, then the frame stream is subsampled, i.e. skipping of the frames.

The TSM monitor 400 receives the following as inputs and guides it to the embodiment 1000 generating the speed contours: (a) User input converted to the TSM rate or playback speed required by the UIP / PC 200. (The required TSM rate or playback rate may indicate the TSM rate or playback rate for the portion of the input audio or audio-visual processing being recognized.) (B) Audio or audio-visual from digital storage 75 (C) a group of current stream positional information from 75, such as from digital storage 75, to a digital storage device used to identify a location in the stream of samples being sent. A sample count or time value at the beginning of the samples of (d) parameters Interval_Size and Speed_Change_Resolution from UIP / PC 200.

Velocity contours, for example, in the form of data streams, indicate the TSM rate or playback rate required for audio or audio-visual processing for all or several points of processing. In practice, the time resolution required in embodiment 1000 to generate the TSM rate or playback rate required for audio or audio-visual processing is to determine the digital signal comprising audio or audio-visual processing. Fluctuates late compared to the sampling rate. As a result, in a preferred embodiment of the present invention, the velocity contours have a single TSM value associated with a particular group of samples corresponding to a particular segment of audio or audio-visual processing. As an alternative, one can associate the TSM value with each sample of the input audio-visual process.

In practice, the precision required to reproduce the TSM rate or playback speed is limited. Thus, in a preferred embodiment of the present invention, instead of using a range of continuous TSM rates or playback rates, the TSM rates are quantized to values of quantized levels used to represent fixed intervals and TSM rates. This can be explained as follows.

Two parameters guide the described embodiment of the TSM monitor 400.

1. Interval_Size: This parameter determines the time interval, given as a number of samples of input audio or audio-visual processing, that must elapse between analysis of TSM rate or playback rate change.

2. Speed_Change_Resolution: This parameter indicates the amount of difference between the quantized levels, which is used to indicate the TSM rate or playback speed.

TSM monitor 400 uses the parameter Interval_Size to segment the input digital stream, and arithmetic of a single TSM rate for each segment of the input digital stream, such as the TSM rate at the beginning or end of the segment or across the segments. Determine the mean, etc. Note that the length of each segment is given by the parameter Interval_Size.

The TSM monitor 400 uses the variable Speed_Change_Resolution to determine the appropriate TSM rate and pass it to the TSM subsystem 300 and the speed contour generator 500. The input TSM rate required by the user is converted to one of the quantized levels by methods known to those skilled in the art. This means that the output TSM rate or the input TSM rate at which the refresh rate is required can only change when it changes by an amount that exceeds the difference between the quantization levels, that is, Speed_Change_Resolution. In a practical relationship, the parameter Speed-Change_Resolution filters out small changes in the TSM rate or playback speed that may occur if the user changes the TSM rate or playback speed to a small amount and then immediately returns to the previous value. The parameters Interval_Size and Speed_Change_Resolution in the embodiment 1000 may be set to predetermined parameters according to methods known to those skilled in the art, or user input through the UI 100 according to methods known to those skilled in the art. Can be input and / or changed by receiving. However, how these parameters are set and / or changed is not shown for ease of understanding of the present invention.

TSM monitor 400 produces as output a pair of values for each segment of the input stream specified by Interval_Size. (a) one of a pair of values representing positional information in the input digital stream for the segment; (b) the other of a pair of values representing the TSM rate or playback rate requested by the user for that segment. This pair of values is applied to the input of the velocity contour generator 500 and the other of the pairs of values representing the TSM rate is applied to the input of the TSM subsystem 300.

The velocity contour generator 500 accepts as input: (a) one of a pair of values representing position information in the input digital stream for the segment from the TSM monitor 400 (b) the segment from the TSM monitor 400. The other one of the pairs of values indicative of the TSM rate or the playback rate, and (c) the parameters Average_or_Overwrite and Log_Repeats from the UIP / PC 200. Speed contour generator 500 manages a list of records using a database or scratch-pad memory, each record storing information corresponding to the TSM rate and stream location information for the TSM rate. 2 shows a flowchart of an algorithm used in one embodiment of a speed contour generator 500 for generating a speed contour. The following fields are used for the records used in the embodiment.

1. Rec: A unique number indicating each record and its allocation / occurring order.

2. Loc: A data field containing stream position information for a segment of an input stream.

3. Play_cnt: Data field containing the number of times a segment has been played. Play_cnt is set when the record is created.

4. TSM: data field indicating the TSM rate for the segment

In addition to the data fields described above, two parameters guide velocity contour generator 500 which generates velocity contours.

1. Average_or_Overwrite: This parameter should be used when the user manually moves the rewind or play position (e.g. with a mouse, slider or position indicator) to replay the previously played input audio or audio-visual processing. Indicates whether it is If the value of the parameter is " Average ", the TSM rate or playback rate for the repeated segment is calculated by averaging the specified TSM rate or playback rate respectively when the segment was played. If the parameter value is "Overwrite", only the last TSM rate or playback speed specified for the repeated segment is used for the repeated segment of the speed contour.

2. Log_Repeats: If true, this parameter is a boolean variable that tells the speed contour generator 500 to record each TSM rate when a section of input audio or audio-visual processing is played by the user. . The TSM rate or playback speed is stored each time a segment is played.

The parameters Average_or_Overwrite and Log_Repeats can be set to predetermined parameters for the embodiment 1000 according to methods known to those skilled in the art, or user input via the UI 100 according to methods known to those skilled in the art. Can be input and / or changed by receiving. However, how these parameters are set and / or changed is not shown for ease of understanding of the present invention.

As shown in FIG. 2, the segment location and TSM rate are applied as input to box 1500. In box 1500, a search is performed to locate any record in the database that contains the same segment location value. Proceed to the next box 1510. In box 1510, a determination is made. If a record containing the same segment location information is found, the record is notified and proceeds to box 1520. If no record is found, go to box 1570.

At box 1570, a new record is created in the database, and the internal variable Record_Count is updated to reflect the record count in the database (internal variable Record_Count is initialized to zero at the beginning of each new velocity contour generation). Next, go to box 1580. At box 1580, the data value is stored in the field of the newly generated record, and the process proceeds to box 1550.

In box 1520 a determination is made. If the parameter Log_Repeats is true, go to box 1570; if the parameter Log_Repeats is false, go to box 1530. In box 1530 a decision is made. If the parameter value Average_or_Overwrite is equal to "Average", go to box 1540. If the parameter Average_or_Overwrite is equal to "Overwrite", go to box 1560.

In box 1540, the data in fields TSM and Play_cnt are replaced. As shown in FIG. 2, the previous value of Play_Cnt is used to calculate the arithmetic mean of the TSM rate, and Play_Cnt is incremented. Next, go to box 1550. In box 1560, data of fields TSM and Play_Cnt are replaced. As shown in Figure 2, the current TSM rate overwrites the previous TSM rate and Play_Cnt is incremented. Proceed to the next box 1550.

In box 1550, the newly generated or modified record is stored in a database. It waits until the new data value reaches the speed contour 500, and proceeds to box 1500. Upon completion of the playback of the audio or audio-visual process, a database is searched and the TSM rate or playback speed for each segment of the input signal is extracted and used to construct the speed contour. Note that when no segment is repeated and the process is reproduced as it is, the speed contour is obtained by sorting the database records in ascending order according to the corresponding allocation order stored in the Rec data field. It is also noted that the speed contours can be stored for later use in accordance with one of many methods known to those skilled in the art to store such digital data streams. For example, speed contours may be stored in digital storage 75 or some other storage medium, or may be transmitted to another system via a transmission device, such as a modem.

1 illustrates an embodiment 1000 in which a separate module should be included in the preferred embodiment, while in the preferred embodiment the UI 100, UIP / PC 200, TSM subsystem 300, TSM monitor 400 And speed contour generator 500 may be implemented as a software program or as a module running on a general purpose computer such as a PC. Further, digital storage 75 can be implemented as a disk drive or RAM, and digital-to-analog converter 600 is implemented as a typical accessory to a general purpose computer such as a sound card on a PC. In the foregoing detailed description, it is well known that methods of implementing programs or modules of such software are known to those skilled in the art.

According to one embodiment of the invention, the data given in the velocity contours for a particular user are presented in a graphical form to display similarity or differences by displaying the TSM rate or playback rate selected by the user or user group. Can be. In one embodiment, the TSM rate is displayed on the vertical axis of the two-dimensional graph, and the segment number or time value is displayed on the horizontal axis. 3 shows the speed contours in graphical form for several different listening sessions of the same audio or audio-visual process. By displaying these velocity contours in a graphical form, information about user interest, user understanding, and user confusion can be estimated. For example, all users slow down the TSM rate or playback speed of audio or audio-visual processing in segment 1000 (A in FIG. 3), and in the next segment 2200 (FIG. 3B) Accelerated speed From this, the user is more interested in the data provided in the cycle between segments 1000 and 2200, or the TSM rate or regeneration rate for the preceding segment is too fast for the safe and complete understanding of the task in that cycle. It can be estimated that the complexity has changed. A method for providing a graphical display of stored speed contours in accordance with embodiment 1000 described above, and storing said speed contours for several users and / or several sessions of the same user together with associated authentication information, as known in the art. Methods that allow retrieval of information related to a particular one of stored velocity contours in accordance with the method are known to those skilled in the art.

Alternative embodiments of the present invention have the same components as described above in connection with embodiment 1000 except for speed contour generator 500. In an alternative embodiment of the present invention, velocity contour generator 5000 outputs a "derivative" velocity contour that includes a derivative of the TSM rate or playback rate for each segment of the input audio or audio-visual process. 4 shows a graphical representation of a velocity contour created using a first derivative of the TSM rate or playback rate specified by the user for several different listening sessions of the same audio or audio-visual process. In the two-dimensional diagram of FIG. 4, the first derivative of the TSM rate is displayed on the vertical axis and the time is displayed on the horizontal axis. The same data displayed in Figure 3 is used to generate the derivative velocity contours for each user. As can be seen in Figure 4, the induced speed contours represent changes in the TSM rate or playback speed requested by the user in a somewhat easy-to-observe pronunciation method. It is also readily understood that derivative velocity contours contain less data than velocity contours because there are relatively few TSM rates or regeneration rate changes, and only segments associated with non-zero derivative TSM rates should be stored. It will be apparent to those skilled in the art how to transform the algorithm shown in FIG. 2 to determine the derivative velocity contours using methods known to those skilled in the art or to derive the derivative velocity contours from the velocity contours.

The term average velocity contours is used when a particular user listens to audio or audio-visual passages several times, thereby averaging some velocity contours resulting from the use of an embodiment of the present invention, such as the embodiment 1000 described above. Refers to velocity contours obtained for audio or audio-visual processing. The value of the TSM rate or playback speed for a particular segment of the average speed contour is obtained by calculating the arithmetic mean of the TSM rate or playback speed in each of the several speed contours for that segment of the audio or audio-visual process. According to methods known to those skilled in the art, the speed contours generated in accordance with the embodiment 1000 described above for several users and / or several sessions of the same user with associated authentication information, and stored speed contours It will be apparent to those skilled in the art how to provide retrieval of information related to a particular contour. Also, it will be apparent to those skilled in the art how to calculate the average velocity contour from any number of stored velocity contours. One use of the average speed contour is to create an advertising or informational audio or audio-visual process for which corresponding information, such as a telephone number, for example, will be dictated by the listener. In order to determine the optimal information transmission speed that enables the listener to successfully capture the required information, the average speed contour is generated using the speed contours generated by the representative user among the target listeners. Another use of the average rate contour is to provide information at the maximum rate for audio or audio-visual processing, which allows the maximum rate to allow the listener to understand the information being transmitted. For example, an advertisement that is created in this way may have a faster speaking rate, or a faster information delivery rate, to convey as much information as possible in a given time slot. Listeners using embodiments of the present invention may be able to reduce the TSM rate for segments of audio or audio-visual processing where speaking speeds are too fast for the listener's understanding or perception.

The term Democratic Speed Contour refers to a particular audio or audio-visual process by averaging some average speed contours or some speed contours obtained from other users while listening to a particular audio or audio-visual task. Refers to the velocity contours obtained. The value of the TSM rate or playback rate for a particular segment of the popular speed contour is the arithmetic mean of the TSM rate or playback rate of each of the speed contours (eg from each different listener) for that segment of the audio or audio-visual processing. According to methods known to those skilled in the art, the velocity contours generated according to embodiment 1000 described above for some users and / or for several sessions of the same user with associated authentication information. It will be apparent to those skilled in the art how to store and provide retrieval of information related to a particular contour of a stored velocity contour. One use of popular speed contours is by those who transmit information. In order to determine the optimal rate of information transfer that allows a particular demographic group of listeners to make good use of the information, demographic contours can be generated using velocity contours from members of a particular demographic group of listeners. For example, this embodiment may be used to provide a public contour that takes advantage of the fact that when a listener in one region of the country listens to a speaker having an accent in another region of the country, a slower information transfer rate is required. In another example of popular contouring, information about a particular demographic listener group is obtained, for example, by a questionnaire. Next, the target audience is selected based on the questionnaire's response. For example, one group may be classified into a listener subgroup using a PC and a listener subgroup not using a PC. Next, the optimum speed transfer rate for the computer software product is obtained from the popular speed contours generated by each subgroup. In this way, optimal information rates of audio or audio-visual processing for advertising or information can be obtained for a particular demographic listener group.

5 shows a block diagram of a second embodiment 2000 of the invention for generating velocity contours for audio or audio-visual processing, wherein user input and word maps of audio or audio-visual processing provide velocity contours. It is used to In such embodiments, velocity contours may be generated without the user listening to the audio or audio portion of the audio-visual process. According to the second embodiment of the present invention, the speed contours are not displayed using an editor for displaying and manipulating the speed contours in response to user input, rather than the sampling TSM rate or playback speed described above in connection with the first embodiment of the present invention. Obtained.

As shown in FIG. 5, an embodiment 2000 has a user interface 2100 (UI) for receiving input from a user. There are a number of devices known to those skilled in the art for receiving input from a user. For example, (a) pressing a key, (b) operating a switch on a mouse, (c) moving a slider or position indicator, and (d) a user voice command, and responding to a key press, switch operation, slider or position indicator. It is apparent to those skilled in the art that there are commercially available equipment that sends digital information indicative of movement or sound commands to a central processing unit.

As shown in FIG. 5, an embodiment 2000 includes a user input processor 2200 that receives data or signals from user input or input audio or audio-visual processing stored in the digital storage 2075 from the UI 2100. (UIP) is provided. In response, UIP 2200 generates and outputs data, such as two-dimensional, with (a) time displayed on the horizontal axis and possibly text or phonetic words, and (b) TSM rate displayed on the vertical axis. Create a graph. The graph display 2300 receives as input, data providing a graph screen display image from the UIP 2200. In response, graph display 2300 displays a two-dimensional representation of the input audio or audio-visual process with a text or phonetic label. It will be apparent to one skilled in the art, for example, that text and / or phonetic information can be displayed as an overlay on top of a graphical representation of a sound waveform on a computer screen. Next, in accordance with embodiment 2000, the user highlights an area of text displayed on graph display 2300 using, for example, a cursor under control of UIP 2100, such that the input audio or audio- associated with the highlighted text is selected. Specific portions of visual processing can be identified. Next, using the UI 2100 in a manner known to those skilled in the art, the user selects and / or specifies the TSM rate or playback speed for a particular portion of the input audio or audio-visual processing related to the highlighted text. . As another example of the second embodiment of the present invention, the UIP 2200 has a text editor that displays a texturized portion of the audio portion of the audio processing or the audio-visual processing. In response, using the UI 2100 in a manner known to those skilled in the art, the user selects a text area, and selects and / or specifies a TSM rate or playback speed for the selected text area. Next, a sample or segment of the input audio or audio-visual processing corresponding to the boundary of the selected text area is determined and used to construct the velocity contours. 6 shows in graphical form a two-dimensional graph displaying text and corresponding text of audio or audio-visual processing. As shown in FIG. 6, the user has highlighted the area 6100 of the input audio or audio-visual process containing the telephone number. The user then used the slider bar 6200 to indicate the required TSM rate for the selected area of the input audio or audio-visual processing. Finally, FIG. 6 shows velocity contour 6300 generated based on the TSM rate requested by the user. 7 shows a texturized display of audio or audio-visual processing. As shown in FIG. 7, the user has highlighted a taxonized area 7100 of the input audio or audio-visual process containing the telephone number.

The UIP 2200 builds the velocity contours in the same manner described above with respect to the velocity contours 500 (in conjunction with FIG. 2). Finally, the UIP 2200 stores the speed contours on, for example, digital storage 2075 or other storage media, or transmits the speed contours to another system via a transmission device, such as a modem.

Although FIG. 5 illustrates an embodiment 1000 that should have a separate module, in a preferred embodiment, the UI 2100 and UIP 2200 are implemented as software programs or modules that run on a general purpose computer such as a PC, for example. In addition, the digital storage device 2075 is implemented as a disk drive or RAM. As is evident from the foregoing detailed description, methods of implementing such programs or modules in software will be apparent to those skilled in the art. In addition, according to many methods known to those skilled in the art, audio or audio visual tools may be stored in digital storage 2075 in analog form and converted to digital form.

According to the second embodiment of the invention described above, the velocity contours are naturally temporal. That is, the TSM rate or playback speed is related to every time interval of audio or audio-visual processing. This property of temporal contours requires some kind of preview of audio or audio-visual processing by the listener or editor to determine the speed contours for the processing. To eliminate this, in a third embodiment of the present invention, a conceptual speed association data structure (CSA data structure) is generated for use in generating the LIF tool. A CSA data structure is, for example, a list of list pairs of concept identifiers and a list of velocity value identifiers. The CSA data structure is stored as a list of these sub-list pairs.

Concept identifiers include keywords, word strings, or phrases that represent concepts such as "Stock Market," "Wall Street," and "Financial." This concept identifier is paired with a speed value identifier that indicates the TSM rate, or playback speed, required by the user while listening to audio or audio-visual processing including the concept identifier.

A third embodiment of the present invention uses a detection device that detects concept information of a specific portion of an audio or audio-visual process and concept information to retrieve information from a TSM rate or playback speed, CSA data structure, and retrieved information. Utilize a search device, which allows the to be used to determine the TSM rate or playback speed that should be used for a particular portion. According to one embodiment of the invention, the detection device comprises a speech recognition equipment known to those skilled in the art. According to another embodiment of the present invention, there is included an apparatus for detecting concept information included in closed captioning information, for example, used in many TV broadcasts or utilized in videotape. Detection devices for detecting such closed caption information are known to those skilled in the art.

8 is a block diagram of a third embodiment 4000 of the present invention for generating CSA data structures for audio or audio-visual processing. As shown in FIG. 8, an embodiment 4000 has a user interface (UI) 4100 for receiving input from a user. An embodiment of the UI 4100 is the same as the UI 100 described above with respect to FIG. 1. The UI 4100 provides an output signal indicative of an input from a user. The user input is interpreted by the user input processor / playback controller (UIP / PC) 4200 to indicate the following options selected by the user: (a) File corresponding to the particular audio or audio-visual process to be played. Selection (the selected file may be entered directly into the embodiment 4000, or may be a file that was stored by the embodiment 4000), (b) starts playback of the selected file, (c) plays the selected file (D) pause playback of the selected file, (e) change the TSM rate or playback speed of a portion of the audio or audio-visual processing being played, or (f) generate a CSA data structure. Specify the parameters Refine_or_Average, Theta, and Sigma to be used by the device in the manner described below.

The UIP / PC 4200 receives input from the UI 4100, (a) converts user input to numeric values, (b) interprets user input, sets parameter values, and generates CSA data structures. Control access, use, modification, or overriding, and (c) direct access and loading of data streams from audio or audio-visual processing by sending stream data requests to digital storage 4075 (performing playback control). do). In the case of digital storage 4075, UIP / PC 4200 may request to access a digital data file representing audio or audio-visual processing stored to a file system on the device. In order to direct access and loading of the data stream from audio or audio-visual processing, the UIP / PC 4200 uses a user input and a location of the digital sample representing the audio or audio-visual processing stored on the digital storage 4075. Interpret to calculate the playback position for the selected file in a particular sample.

Digital storage 4075 receives as input (a) a stream data request from UIP / PC 4200, and optionally (b) a TSM output from TSM subsystem 4300, and optionally (c) Data stream representing a CSA data structure from a CSA data structure generator (4500, CSADA generator). Digital storage 4075 produces as output (a) a data stream representing audio or audio-visual processing, and (b) a location information stream of the output data stream, such as a location on a file. Methods of storing and retrieving general purpose data using digital storage devices such as "hard disk drives" are known to those skilled in the art.

Audio or audio-visual processing is typically stored on digital storage 4075 in digital form. An embodiment of digital storage 4075 is the same as digital storage 75 described above with respect to FIG. 1. Digital storage 4075 receives data requests from UIP / PC 4200 in accordance with methods known to those skilled in the art, and provides a stream of digital samples indicative of audio and / or audio-visual processing. In an alternative embodiment, audio or audio-visual processing is stored on analog storage in analog form. In such alternative embodiments, a stream of analog signals is input to an unshown device that converts analog samples into digital samples. Many commercially known to those skilled in the art that receive input analog signals, such as voice signals, and provide a stream of digital signals that can be sampled at least at the Nyquist rate and converted back to analog signals without loss of fidelity. There is a device. The digital sample is then sent to the TSM subsystem 4300.

TSM subsystem 4300 receives as input: (a) a stream of samples representing portions of audio or audio-visual processing from digital storage 4075, (b) a sample count or time value, or the like. Stream location information from the digital storage device 4075 used to identify the location on the data stream of the sample being sent, and (c) the required TSM rate or playback rate from the TSM concept monitor 4400. The output of the TSM subsystem 4300 is applied to the following inputs: (a) digital analog converter / audio and / or audio-visual playback device 4600, DA / APD, and optionally (b) TSM, if required. Digital storage for storage of the output, ie LIF tool (4075). DA / APD 4600 is a device known to those skilled in the art for receiving digital samples and establishing audio or audio-visual processing. In accordance with the present invention, the output of the TSM subsystem 4300 is supplied with its playback rate from the TSM concept monitor 4400 to provide feedback to the user regarding the user's current TSM rate requirements. A stream of digital samples representing visual processing. The user can change the TSM speed or playback speed by listening to the TSM-enabled output and using the UI 4100 to provide further input. Also, if the user wants to accelerate or slow down a portion of the audio or audio-visual processing that has just been played back (if he wants to speed up or slow down other parts with the same concept identifier not yet played), the user can use the UI 4100. An input may be provided to rewind the audio or audio-visual processing to the desired portion, and may be played back at the changed TSM or playback speed (or may specify the required TSM rate or playback speed for other parts). In this way, the user determines the required TSM rate or playback speed for each part of the audio or audio-visual process. Embodiments of the TSM subsystem 4300 and the DA / APD 4600 are the same as the TSM subsystem 300 and the DA / APD 600 described above with respect to FIG. 1. As will be appreciated by those skilled in the art, whenever the embodiment 4000 provides playback for audio-visual processing, the TSM subsystem 4300 may accelerate or decelerate the visual information to provide audio and audio for the audio-visual processing. Match In order to do this in a preferred embodiment, the video signal is " frame-subsampled " or " frame-replicated " according to any one of many methods known to those skilled in the art. To maintain synchronization between the audio and the visual part of the audio-visual process. Thus, if the audio is accelerated and a faster rate of sampling is required, the frame stream is subsampled, i.e. a frame skip is performed.

Concept determiner 4700 accepts different data sets according to specific options as input. According to option 1, the input data is text or text, such as closed caption data or textual annotation, stored with the current segment of input audio or audio-visual processing, for example, that is being fed to the TSM subsystem 4300. Contains a data stream representing the concept. In the case of option 1, concept determiner 4700 forwards the incoming text or data stream representing the concept as output to concept decoder 4800. According to option 2, the input data includes: (a) a stream of samples representing portions of audio or audio-visual processing from digital storage 4075, and (b) digital storage 4075, for example. Current stream position information from digital storage 4075, used to locate the position of the sample stream being transmitted, such as the time value for the start of a group of samples transmitted from or a sample count. For option 2, concept determiner 4700 provides as output the data stream representing the concept included in the current portion of the audio or audio-visual processing being fed to TSM subsystem 4300. The concept and / or textification of the spoken sentence is determined by extracting closed caption information from the audio or audio-visual processing or using a food recognition algorithm to obtain a stream of text from the input audio or audio-visual processing. . Methods of extracting closed caption information and many methods of extracting text using a speech recognition algorithm are known to those skilled in the art.

Caption information decoder 4800 accepts as input a data stream representing concept information from concept determiner 4700. According to the present invention, without limitation, concept information includes: dictated sentences, actual text, keywords, phrases, or other representations of concept information known to those skilled in the art. In response, concept information decoder 4800 generates as output a data stream representing keywords and concepts for the current portion of the input audio or audio-visual processing being sent to TSM subsystem 4300.

Concept information decoder 4800 processes the input to form a concept data representation of the input data stream. For example, the concept information deborder 4800 may simply remove adjectives or articles from input representing text that has been texted to provide an output consisting of only nouns and noun phrases. Alternatively, concept information decoder 4800 may employ natural language processing to extract the content of the concept from the stream of spoken words. Many methods for implementing the concept information decoder 4800 are known to those skilled in the art. For example, using a technique known as clustering, which develops a data set of multidimensional vectors, in which each element of the vector represents a particular property or value associated with the attributes of the entire data set. There are many systems. Clustering allows classification and grouping of concepts based on the N-dimensional Euclidean distance between vectors. An object of a clustered data set can often be obviously not included in any one cluster, in which case the object can be associated with one or more clusters. In such a case, Euclidean distance can be used to indicate the probability that the object is a member of each possible cluster. See, eg, the Ph.D. dissertation, "Semantic Feature Extraction from Technical Texts with Limited Human Intervention" (by Rajeev Agarwal 1995), submitted to Mississippi State University.

TSM concept monitor 440 guides embodiment 4000 to generate a CSA data structure by receiving as input: (a) a user translated by UIP / PC 4200 at the required TSM rate or playback rate. Input (the required TSM rate, or playback rate, may indicate a change in TSM rate or playback rate for a portion of the perceived audio or audio-visual process) (b) currently sent to the TSM subsystem 4300 Data from a concept information decoder 4800 representing the concept of a portion of the input audio or audio-visual processing that is present, and (c) the parameter Speed_Change_Resolution from the UIP / PC 4200.

The TSM concept monitor 4400 processes the TSM rate or playback rate requested from the user, and concept information, and derives a single TSM rate for the concept provided as input. For example, the concept output from the concept information decoder 4800 remains unchanged for a few seconds because an input concept such as "financial market" may represent some words or phrases of audio or audio-visual processing being played. Can be. Because of this, a user may request multiple TSM rates over an interval of input audio or audio-visual processing related to a single concept. In accordance with the present invention, the TSM concept monitor 4400 generates a single TSM rate for one concept, for example by performing an arithmetic average of the TSM rate over the interval of input audio or audio-visual processing related to a single concept. . For example, a weighted average may be used that highlights the most recent TSM rate obtained during a period during which a particular concept is present at the input to the TSM concept monitor 4400. It is to be understood that these are merely examples of the many different methods that can be used.

The TSM concept monitor 4400 uses the parameter Speed_Change_Resolution to determine the appropriate TSM rate and passes it to the TSM subsystem 4300 and the CSADS generator 4500. The TSM rate determined for a particular concept is converted to one of the quantized levels by methods known to those skilled in the art. This means that the output TSM rate or playback rate can only change if the required TSM rate of the input changes by an amount that exceeds the difference between the quantized levels, that is, the Speed-Change_Resolution and the number of possible TSM rates are determined by the data structure. Limited for efficient representation. The parameter Speed_Change_Resolution may be set to predetermined parameters for the embodiment 4000 according to methods known to those skilled in the art, or may receive user input through the UI 4100 according to methods known to those skilled in the art. By input and / or change. However, how these parameters are set and / or changed is not shown for ease of understanding of the present invention.

The TSM concept monitor 4400 produces (a) a single TSM rate value and (b) concept information as output. The TSM rate is applied as input to the TSM subsystem 4300 and the concept velocity relationship data structure generator 4500 (CSADS generator), and the concept information is applied to the input of the CSADS generator 4500. It will be apparent to those skilled in the art that the following merely describes an embodiment for determining a single TSM rate in one concept using the mean for the understanding of the present invention. However, the present invention is also not limited to any one algorithm that determines the TSM rate and associates it with the concept, and embodiments of the present invention are not limited to associating a single TSM rate with one concept. Will be self explanatory. For example, the TSM rate associated with a concept can change, for example, to accelerate during playback to reflect the fact that the user is very familiar with the concept, and not much time is needed to understand the information as the concept is repeated during playback of the process. It may also reflect the fact that it is not.

The CSADS generator 4500 accepts the following inputs from the TSM concept monitor 4400: (a) concept information, (b) the TSM rate or refresh rate for the concept, and (c) the process of generating the CSA data structure. Parameter (Refine_or_Average, Theta, and Sigma) values from UIP / PC 4200 used to control. Many methods for implementing such data structures are known to those skilled in the art.

For example, a CSA data structure may be implemented as a series of related keyword phrases, or concepts, involving appropriate TSM values.

(("stock", "bonds", "stock market", "wall stree", "currency") 0.8)

(("Hollywood", "actor", "movie") 1.5)

Here, the TSM rate of the first concept group is 0.8 and the TSM rate of the second concept group is 1.5. This data structure allows listeners to hear information about stock market and other financial concepts at reduced playback speed (0.8) and to specify that information about Hollywood movies should be provided at an accelerated playback speed (1.5 of normal playback speed). Indicates the need.

CSADS generator 4500 uses a database or scratch-pad memory to maintain a record list, where each record stores information corresponding to the concept and the TSM rate associated with that concept. 9 shows a flowchart of an algorithm used in one embodiment using a CSADS generator 4500 to generate a CSA data structure.

As shown in FIG. 9, the concept information and the TSM rate are applied as input of box 9500. In box 9500, a search is performed to place any record in the database containing the same concept information. Next, go to box 9510. At box 9510, if a concept was found at box 9500, a numerical value is determined that reflects the similarity of the potential matching list for the found concept. Data representing the concept or concept distance between two words can be calculated using any number of methods known to those skilled in the art. For example, in its simplest form, a synonym or list of other reference data may be employed to calculate the distance. Alternatively, Euclidean distance can be used to measure the similarity of multidimensional vector objects in a data set employing a clustering algorithm to classify concepts. In another method, a "head-driven phrase structured grammar" is commonly used to parse the meaning of sentences and word phrases. Next, go to box 9520.

At box 9520, a determination is made whether the record with the closest match is within the range of the given parameter Theta amount. If the closest match is within the given Theta amount range, go to box 9930; otherwise, go to box 9590.

At box 9530, a determination is made whether the parameter Refine_or_Average is equal to "Refine" or "Average". If Refine_or_Average is "Refine", proceed to box 9540. If Refine_or_Average is "Average", go to box 9580.

In box 9580, the stored TSM value for a particular concept is updated by calculating the arithmetic mean of the existing TSM value of the CSA data structure and the currently stored TSM rate. Proceed to next box 9570.

At box 9590, a new record is created in the database. Next, go to box 9600. In box 9600, the value of the CSA data structure is installed as follows: (a) The current concept is stored in the concept field, (b) The current TSM rate is stored in the TSM field. Proceed to next box 9570.

At box 9540, a determination is made to compare the difference between the TSM rate of the closest matching record and the current TSM rate. If there is a difference greater than the parameter Sigma, go to box 9560; otherwise, go to box 9570. In box 9560, the current concept or keyword phrase is narrowed by appending the previous concept to the current concept in order to further characterize and narrow the concept and distinguish it from the existing concept of the CSA data structure. For example, in a preferred embodiment of the present invention, a concept or keyword "bond" may be included in the CSA data structure record corresponding to the financial information. That is, the concept field corresponding to the financial information may include ("money", "stock", "bond"). The input audio or audio-visual processing included the phrase "actor James Bond", and the listener continued to accelerate playback in this phrase, so that the TSM rate differed from the value in the TSM field corresponding to the financial information concept field by more than Sigma. If so, the concept or keyword "bond" may be prefixed to an existing concept or keyword, in this case "James". Next, proceed to box 9500, as shown, and search the database again using this new concept "James Bond". According to this embodiment of the present invention, another entry is generated for the keyword "bond". One entry corresponds to using it in the context of a financial report, and the other entry is used when paired with the name "James." In box 9570, the newly created or updated record is stored in the database.

In another embodiment of the present invention, a CSA data structure may be generated without using the embodiment 4000 described above. Instead, a CSA data structure can be generated, for example, by entering data into a structure using a text editor or by filling in a questionnaire for a concept of interest. This CSA data structure can be used to generate LIF tools from audio or audio-visual processing without the user listening in advance. In the same way, CSA data structures can be constructed using keywords or phrases in the form typically provided in "online" search engines, and used to control data retrieval.

The CSA data structure can also be used to control the playback speed of audio or audio-visual processing retrieved by a search engine, thereby creating a LIF tool from audio or audio-visual processing retrieved by a search engine not heard previously. In one such embodiment, a CSA data structure is obtained using a user specified search category entered into a search engine. For example, user input to a search engine requesting "all boats excluding yachts" would create a LIF tool that reproduces information about boats at normal speed but accelerates or excludes items for yachts. In light of the detailed description, it is apparent to those skilled in the art how to generate CSA data structures using, for example, information transmitted from a search engine.

In another embodiment of the present invention, the CSA data structure may include affinity for a TSM rate entry, eg, a particular concept or keyword. In this embodiment of the invention, the TSM rate of "affinity" (another expression that may be similarly translated) is such that the concept goes beyond a section of audio or audio-visual processing with a TSM rate corresponding to affinity ( skip) Indicates a playback system. According to this embodiment, the user may designate "no interest" in a particular concept or keyword when listening or searching for audio or audio-visual processing. For example, the user may specify the following CSA data structure while listening to nightly newscasts.

(("weather", "partly cloudy", "weather forecast", "temperature", "dew point")

("infinity")

(("stock", "bond", "stock market", "wall street", "currency") 0.8)

(("Hollywood", "actor", "movie") 1.5)

This CSA data structure includes (a) reporting of weather forecasts and temperatures during broadcast, (b) reproducing financial information at 0.8 times normal playback speed, and (c) reproducing information about Hollywood movies and actors at 1.5 times normal playback speed. Increasing TSM rate instructs playback to accelerate.

Embodiments of the present invention are not limited to static CSA data structures, and as described below, a user may provide input during playback to update the TSM rate. For example, the CSA data structure contains an entry,

(("stock", "bond", "stock market", "wall street", "currency") 0.8)

(("Hollywood", "actor", "movie") 1.5)

When the phrase "actor, James Bond" occurs in the input, and the user continues to interfere with the playback speed, embodiment 4000 of the present invention adds a new entry so that the modified data structure is as follows. You can also make changes or updates to.

(("Stock", "bond", "stock market", "wall street", "currency") 0.8)

(("Hollywood", "actor", "movie") 1.5)

(("actor James Bone") 2.0)

In this way, the CSA data structure can be continuously updated to reflect user interest while listening to new material and new concepts using existing CSA data structures.

As can be readily appreciated, the use of the CSA data structure is not limited to the TSM rate, and in practice, as described above, the first derivative of the TSM rate may be used to affect the same result. For example, if the user continues to slow down when hearing the word "free sample", a CSA data structure that stores changes in the TSM rate rather than the TSM rate itself would be equally useful for controlling the rate of playback of previously unlisted data. will be.

FIG. 8 shows an embodiment 4000 that should have a separate module, but in a preferred embodiment the UI 4100, the UIP / PC 4200, the TSM subsystem 4300, the TSM concept monitor 4400, the concept. The determiner 4700, the concept information decoder 4800, and the CSADS generator 4500 may be implemented as software programs or modules that run on a general purpose computer such as a PC, for example. In addition, digital storage 4075 may be implemented as a disk drive or RAM, and D / A converter 4600 may be implemented as a typical accessory of a general purpose computer such as a sound card on a PC. In the light of the detailed description, how to implement such a program or module in software will be apparent to those skilled in the art.

The embodiment 4000 shown in FIG. 8 may be modified to convert a previously generated velocity contour for a particular audio or audio-visual process into a CSA data structure for that process. In this embodiment, the TSM rate is obtained from the velocity contours (to replace the user TSM rate value from the UIP / PC 4200) and provided as input to the TSM concept monitor 4400. From the illumination of the description herein, the method of inputting the velocity contours and obtaining the TSM rate is apparent to those skilled in the art. Equally, the embodiment 1000 shown in FIG. 1 may be modified to convert a previously generated CSA data structure for a particular audio or audio-visual process into a velocity contour for that process. In this variant, the TSM rate is obtained from the CSA data structure (to replace the user TSM rate output from UIP / PC 200) and provided as input to the TSM monitor 400. The TSM rate is obtained from the CSA data structure according to the embodiment 6000 (described below), that is, the TSM rate is output from the TSM concept lookup 6500 of the embodiment 6000.

For ease of understanding, the embodiment described above refers to the TSM rate. However, the present invention is not limited thereto. It will be appreciated that embodiments of the present invention may use anything from which TSM rates can be determined (affinity information above) for use in making or implementing embodiments of the present invention. For example, an indication of user interest or user information retrieval level may be used in place of the TSM rate. Next, a transition is made between the user interest or user information retrieval level and the TSM rate to provide playback. In such an embodiment, a transform function may be used to map the user interest or user information retrieval level to the TSM rate. In some such embodiments, the transform function may be modified, for example, without changing the velocity contours or the CSA data structure.

For ease of understanding, the above embodiments refer to the velocity contours that make the correspondence between the TSM rate and the associated temporal position, and the CSA data structure that makes the correspondence between the TSM rate and the related concepts. However, the present invention is not limited to this. It will be appreciated that embodiments of the present invention refer to velocity contours or CSA data structures that make a correspondence between which enables to determine the TSM rate and which allows one or more portions of the process related to the TSM rate to be identified.

In addition, in embodiments of the present invention, reference is made to the velocity contours or the CSA data structure, wherein the identifier of the TSM rate and the identifier of the part may have a functional dependency to determine the TSM rate that should be used for a particular part identifier. Will be understood. For example, in an embodiment where a concept is used to identify some parts of a process, the TSM rate associated with the particular concept is computed as a function of the number of times the concept has appeared in the process, so that the first replay of the concept uses the slow TSM rate, In subsequent occurrences of the same concept the TSM rate may be increased for faster playback.

Application of Velocity Contour and Concept Velocity Relation Data Structures to Generate Listener Interest Filtering Tools

According to a fourth embodiment of the present invention, a speed contour is used in conjunction with an audio or audio-visual tool to generate a LIF tool to generate a LIF tool, wherein the audio is generated according to the TSM rate or playback speed specified by the speed contour. Or a segment of audio-visual processing is played. In addition, such an embodiment stores the LIF tool for later playback in the same embodiment or on another playback device.

As will be readily appreciated by those skilled in the art, embodiments of the present invention that provide LIF tools for the audio portion of an audio-visual process may also accelerate or decelerate visual information to match the audio of the audio-visual process. To do this in the preferred embodiment, the audio is processed by the TSM method according to methods known to those skilled in the art as described above, and the video signal is " frame-subsampled " or " frame-duplicated " Achieve a TSM rate, and maintain synchronization between the audio and visual portions of the audio-visual process. Thus, if the audio is accelerated and samples are requested to be made at a higher rate, the frame stream is subsampled, i.e., the frames are skipped.

10 shows a block diagram of an embodiment 5000 of a fourth embodiment of the present invention, which utilizes velocity contours in conjunction with audio or audio-visual processing to generate a LIF tool. As shown in FIG. 10, an embodiment 5000 has a user interface (UI) 5100 for receiving input from a user. The UI 5100 is the same as the UI 100 described above with respect to FIG. 1. The UI 5100 provides an output signal indicating an input from a user. The user input is interpreted by the user input processor 5200 / playback controller 5200 (UIP / PC, 5200) to represent the following options selected by the user: (a) Selection of the file to be played, the file being specified Corresponds to audio or audio-visual processing (the selected file may be entered directly in embodiment 5000, or may be a file stored in embodiment 5000). (B) Rate for controlling the TSM rate or playback speed. Select contours, (c) start playback of the selected file, (d) pause playback of the selected file, (e) pause playback of the selected file, and (f) for a portion of the audio or audio-visual Modifies or overrides the TSM rate or playback rate obtained from the velocity contours, or (g) specifies the parameters Offset and Override used by the device in the manner described below.

As shown in FIG. 10, the UIP / PC 5200 receives a user input from the UI 5100, (a) converts the user input into a numerical value, and (b) interprets the user input to set a parameter value. Control the use, transformation, or overriding of velocity contours, and (c) direct and access data streams from audio or audio-visual tools by sending stream data requests to digital storage 5075 (playback). To perform control), and (d) access and load the data stream from the velocity contours by issuing a stream data request to digital storage 5075. In the case of digital storage 5075, the UIP / PC 5200 may request access to a digital data file representing audio or audio-visual processing stored in a file system on the device. In order to direct access and loading of the data stream from audio or audio-visual processing, the UIP / PC 5200 places a user sample and a digital sample representing the audio or audio-visual processing stored on the digital storage 5075. Interpret to compute the playback position for the selected file in a particular sample. In a preferred embodiment, a data request for audio or audio-visual processing and a data request for speed contours may be issued such that data from each same temporal location is provided to the output from digital storage 5075.

Digital storage 5075 receives as input: (a) a stream data request from UIP / PC 5200, and optionally (b) TSM output from TSM subsystem 5300. Digital storage 5075 generates as output: (a) a data stream representing audio or audio-visual processing, (b) a location information stream, such as a location on a file of the data stream being output, and (c) a speed contour. Data stream representing. For example, many methods of storing and retrieving general purpose data using digital storage devices such as hard disk drives are known to those skilled in the art.

Audio or audio-visual processing is typically stored on digital storage 5075 in digital form. The embodiment of the digital storage device 5075 is the same as the digital storage device 75 described above with respect to FIG. Digital storage 5075 is accessed by UIP / PC 5200 in accordance with methods known in the art to provide a stream of digital samples indicative of audio and / or audio-visual processing. In alternative embodiments, audio or audio-visual processing is stored on analog storage in analog form. In such alternative embodiments, a stream of analog signals is input to a device not shown, which converts analog samples into digital samples. There are many commercially available devices that receive an input analog signal, such as a voice signal, and sample the analog signal at a rate of at least Nyquist rate to provide a stream of digital signals that can be converted back to an analog signal without loss of fidelity. The digital sample is then sent to the TSM subsystem 5300.

The TSM rate determiner 5400 receives as input: (a) a speed contour selected by the user applied as input from digital storage 5075, (b) applied as input from UIP / PC 5200. TSM rate specified by the user to be applied, (c) Offset TSM rate specified by the user to be applied as input from UIP / PC 5200, Offset (d) to user, applied as input from UIP / PC 5200 Boolean parameters, overrides, and (e) current stream position information from digital storage 5075 used to identify the location on the stream of the sample being sent, e.g., transmitted from digital storage 5075. Start time value or sample count of the sample group. In response, the TSM rate determiner 5400 generates the TSM rate received by the TSM subsystem 5300 as an output.

The TSM rate determiner 5400 selects the corresponding temporal position nearest the speed contour using the stream position information to determine the relevant TSM rate specified by the speed contour. This approach allows velocity contours generated at different Interval_Size values or different TSM sampling frequencies to be applied to any audio or audio-visual processing, ensuring a one-to-one time correspondence between the data stream position and the TSM rate derived from the velocity contours.

The TSM rate determiner 5400 determines the output TSM rate or refresh rate using one of the following operating modes.

1. Playback mode of speed contour driving: In this mode, the output of the TSM rate determiner 5400 is the TSM rate obtained from the speed contours for the corresponding parts of the input audio or audio-visual processing to be reproduced. This mode outputs the same TSM rate as specified by the velocity contours.

2. Velocity Contour Offset Playback Mode: In this mode, the user specifies via the UI 5100 an offset parameter Offset that is used to adjust the TSM rate specified by the velocity contour. In this mode, the TSM rate output is given by the following formula.

TSM_rate = TSM rate from the speed contour * (1 + Offset)

For example, if the user specifies an offset factor of -0.4, the TSM rate determiner 5400 adds an offset value of -0.4 to the number 1.0 (result 0.6) and scales to each TSM rate specified in the velocity contours. Thus, a uniform reduction (deceleration) of the TSM rate or the reproduction speed will be achieved with respect to the generated output signal. Equally, a positive offset will increase (accelerate) the TSM rate or refresh rate for the output signal generated. An offset value of zero has no effect at the TSM rate. As can be readily appreciated, different offset methods can be employed to achieve nonlinear and linear scaling of the TSM rate.

3. User Override Mode of Speed Contours: In this mode, the user can override the speed contours and manually control the TSM rate or playback speed across portions of the audio or audio-visual processing. When the override is distributed by the user, the TSM rate used to determine the TSM rate or playback speed of the output signal is extracted from the corresponding position of the speed contour.

As shown in FIG. 10, the TSM subsystem 5300 receives as input: (a) a stream of samples representing portions of audio or audio-visual processing from digital storage 5075, (b) Stream location information, such as sample count or time value, from a digital storage device 5075 used to identify the location in the data stream of the sample being sent, and (c) the TSM rate from the TSM rate determiner 5400. As mentioned above, the input may be an analog that is converted into a series of digital samples according to methods and apparatus known to those skilled in the art. The output from the TSM subsystem 5300 is applied as the following inputs: (a) digital to analog converter / audio and / or audio-visual playback device 5600, DA / APD, optionally (b) if required, TSM Digital storage (5075) for storing playback at rate. DA / APD 5600 is a device known in the art for receiving digital samples and providing reproduction of audio or audio-visual processing. The output from the TSM device 4300 is a stream of digital samples comprising a digitized audio or audio-visual stream, which is a TSM version of the input audio or audio-visual processing, and in accordance with the present invention, velocity contours and / or user input. Reflects the TSM rate or playback speed specified by. This output shows the LIF tool.

In some embodiments, the LIF tool is stored for later playback by the same embodiment or by another playback device. In addition, the digital output can be converted to analog form for storage on an analog device. Many devices are known to those skilled in the art for receiving digitized input signals, such as 16-bit pulse code modulation and outputting analog signals thereof. For example, it would be apparent to those skilled in the art that there are commercially available equipment that receives a stream of digitized samples representing a signal and converts these samples into an analog signal without loss of fidelity. The embodiment of the TSM subsystem 5300 and the DA / APD 5600 is the same as the TSM subsystem 300 and the DA / APD 600 described above in FIG. 1. As will be appreciated by those skilled in the art, whenever the embodiment 5000 provides for playback of audio-visual processing, the TSM subsystem 5300 accelerates and visualizes the visual information to match the audio of the audio-visual processing. Slow down. In order to accomplish this in the preferred embodiment, it is "frame-subsampled" or "frame-duplicated" according to one of many methods known in the art to maintain synchronization between the audio and visual parts of the audio-visual process. Thus, if the audio is accelerated and a faster rate of sampling is required, the frame stream is subsampled, ie the frame is skipped.

10 illustrates an embodiment 5000 that must be configured as a separate module, but in a preferred embodiment, the UI 5100, UIP / PC 5200, TSM subsystem 5300, and TSM rate determiner 5400 are shown. It is implemented as a software program or module that runs on a general purpose computer such as a PC. In addition, the digital storage device 5075 is implemented as a disk drive or RAM, and the D / A converter 5600 is implemented as a typical accessory of a general purpose computer such as a sound card on a PC. In light of the detailed description, it will be apparent to those skilled in the art how to implement such a program or module in software.

As can be easily understood, without user input, the time scale of the LIF tool is entirely determined by the velocity contours. Also, the data fetch rate of the input signal is determined by the velocity contours: high speed is required for acceleration and low speed is required for deceleration. Since the speed contour has a temporal correspondence with the input signal, the data patch rate or read speed for the speed contour is the same as the input signal. In many embodiments, it is desirable to reduce the number of devices with varying read speeds. According to the present invention, the variation in the read speed can be eliminated by the following method.

The data contained in the speed contour will be read at the speed specified by the previous value of the speed contour. By performing the time scale transformation of the input velocity contours using the velocity contours themselves, a new velocity contour is obtained. These TSM speed contours will share a temporal correspondence with the output signal generated by applying the original speed contours as input signals. Since the output is generated at a fixed rate regardless of the TSM performed, the TSM converted speed contour values will be accessed at a fixed rate.

According to a fifth embodiment of the present invention, the CSA data structure is utilized in conjunction with audio or audio-visual processing such that portions of the audio or audio-visual processing are reproduced according to the TSM rate or playback speed specified by the CSA data structure. Create a LIF tool. In addition, some such embodiments store LIF tools for later playback by the same embodiment or by other playback devices.

11 shows a block diagram of a fifth embodiment of the present invention for generating a LIF tool utilizing a CSA data structure in conjunction with audio or audio-visual processing. As shown in FIG. 11, an embodiment 6000 includes a user interface (UI) 6100 that receives input from a user. An embodiment of the UI 6100 is the same as the UI 100 described above with respect to FIG. 1. The UI 6100 provides an output signal indicative of an input from a user. The user input is interpreted by the user input processor 6200 / playback controller (UIP / PC 6200), indicating the following options selected by the user: (a) Selecting a file to play, the file being specified audio or audio- Corresponds to the visual processing (the selected file may be entered directly into the embodiment 6000, or may be a file stored in the embodiment 6000). (B) To control the TSM rate or playback speed, the CSA data structure may be used. (C) Start playback of the selected file (d) Hold playback of the selected file (e) Pause playback of the selected file (f) For a portion of the audio or audio-visual processing being played Modify or override the TSM rate or playback rate obtained from the CSA data structure, or (g) the parameters Theta, Offset, Slew-Limit, and Override used by the methods and devices described below. Designate. Embodiments may also receive audio or audio-visual processing, for example, input directly from the TV. In this case, the audio portion is converted into digital form by the above-described method for analog input, and if there is closed caption information, it can also be converted into an appropriate digital form according to one of many methods known in the art.

As shown in FIG. 11, the UIP / PC 6200 receives input from the UI 6100, (a) converts user input into a numerical value, and (b) interprets the user input to set parameter values. Control access, loading, or overriding the TSM rate from the CSA data structure, and (c) sending the stream data request to the digital storage 5075 to facilitate access and loading of the data stream from audio or audio-visual processing. Instruct (perform playback control). For digital storage 6075, UIP / PC 6200 will request access to a file of digital data representing audio or audio-visual processing stored in a file system on the device. In order to direct access and loading of the data stream from the audio or audio-visual processing, the UIP / PC 6200 inputs the location and user input of the digital sample representing the audio or audio-visual processing stored in the digital storage 6075. By interpreting, the playback position for the selected file in a particular sample is calculated.

Digital storage 6075 receives as input: (a) a stream data request from UIP / PC 5200, and optionally (b) a TSM output from TSM subsystem 5300. Digital storage 5075 produces the following: (a) a data stream representing audio or audio-visual processing, (b) a stream of location information, such as a location on a file of the output data stream, and (c) Stream of data representing CSA data structures. For example, many methods of storing and retrieving general purpose data utilizing digital storage devices such as hard disk drives are known in the art.

Audio or audio-visual processing is typically stored in digital form on digital storage 6075. The embodiment of the digital storage 6075 is the same as the digital storage 75 described above with respect to FIG. Digital storage 6075 is accessed by UIP / PC 6200 in accordance with methods known to those skilled in the art to provide a stream of digital samples indicative of audio and / or audio-visual processing. In an alternative embodiment, audio or audio-visual processing is stored on analog storage in analog form. In such alternative embodiments, a stream of analog signals is input to an unshown device that converts analog samples into digital samples. Many commercial devices known to those skilled in the art that receive input analog signals, such as voice signals, and provide a stream of digital signals that can be sampled at least at the Nyquist rate to be converted back to analog signals without loss of fidelity. There is. The digital sample is then sent to the TSM subsystem 6300.

Concept determiner 4700 accepts different data sets according to specific options as input. According to option 1, the input data may contain text or concepts, such as closed caption data or textual annotations, stored with the current segment of input audio or audio-visual processing being fed to the TSM subsystem 6300. Contains the indicated data stream. In the case of option 1, concept determiner 6700 passes an incoming text or a data stream representing the concept as output to concept decoder 6800. According to option 2, the input data includes: (a) a stream of samples representing portions of audio or audio-visual processing from digital storage 6075, and (b) digital storage 6075, for example. Current stream position information from digital storage 6075, used to identify the location of the stream of samples being transmitted, such as the time value of the start of a group of samples transmitted from or a sample count. For option 2, concept determiner 6700 provides as output the stream of data representing the concept included in the current portion of the audio or audio-visual processing being fed to TSM subsystem 6300. The concept of spoken sentences and / or texturized sentences may be extracted using closed caption information from audio or audio-visual processing, or using a speech recognition algorithm to obtain a stream of text from the input audio or audio-visual processing, Is determined. Many methods for extracting closed caption information and many methods for extracting text using a speech recognition algorithm are known to those skilled in the art.

Caption information decoder 6800 accepts as input the data stream representing the concept information from concept determiner 6700. According to the present invention, without limitation, concept information includes: dictated sentences, actual text, keywords, phrases, or other representations of concept information known to those skilled in the art. In response, concept information decoder 6800 generates as output the data stream representing keywords and concepts for the current portion of the input audio or audio-visual processing being sent to TSM subsystem 6300.

Concept information decoder 6800 processes the input to form a concept data representation of the input data stream. For example, concept information decoder 6800 may simply remove adjectives or articles from the texturized input to provide an output consisting of only nouns and noun phrases. Alternatively, concept information decoder 6800 may employ natural language processing to extract concept content from a stream of spoken words. Many methods for implementing the concept information decoder 4800 are known to those skilled in the art.

The TSM concept lookup 6500 accepts as input: (a) a CSA data structure received from digital storage 6075, (b) input audio or audio-visual processing being sent to the TSM subsystem 6300. Data from concept information decoder 6800 that represents the concept for the current portion, and (c) parameter Theta. The TSM concept lookup 6500 maintains a list of records, using a database or scratch-pad memory, where each record stores information about the TSM rate and concept information about the TSM rate. The TSM concept lookup 6500 performs the following steps in accordance with one of many methods known to those skilled in the art. Search the database containing the CSA data structure for the closest matching concept entry. If the difference of the closest matching entry is within the range specified by the parameter Theta, then the TSM rate associated with that entry is provided as output. If a concept entry in the database containing no CSA data structure is not within the range specified by the parameter Theta, then the TSM rate obtained previously is provided as output and received by the TSM rate arbiter 6400.

The TSM rate regulator 6400 receives as input: (a) the TSM rate from the user input processor 6200 specified by the user, (b) the TSM rate from the TSM concept lookup 6500, (c) Parameters Offset, Slew_Limit, Override from UIP / PC 6200, described below. In response, the TSM rate adjuster 6400 generates a single TSM rate that is sent to the TSM subsystem 6300 as an output.

The TSM rate regulator 6400 determines the TSM rate, or playback speed, using one of the following modes of operation.

1. Playback Mode of CSA Data Structure Driven: In this mode, the TSM rate used is the TSM rate provided by the TSM Concept Lookup 6500.

2. CSA Data Structure Offset Playback Mode: In this mode, the user specifies via the UI 6100 an offset parameter, which is an offset parameter used to adjust the TSM rate specified in the CSA data structure. The TSM rate output is given by the formula

TSM_rate = TSM rate from TSM concept lookup * (1 + Offset)

For example, if the user specifies an offset of -0.4, the TSM rate adjuster 6400 adds an offset value of -0.4 to the number 1 (result 0.6) and scales each of the TSM rates specified by the TSM concept lookup. to achieve a uniform reduction (deceleration) in the TSM rate or playback speed for the resulting output signal. Equally, a positive offset will increase (accelerate) the TSM rate or refresh rate for the output signal generated. An offset value of zero has no effect on the TSM rate. As can be readily appreciated, other offset methods can be employed to achieve nonlinear and linear scaling of the TSM rate.

3. User Override Mode for CSA Data Structures: In this mode, the user can override the TSM rate obtained from the TSM Concept Lookup (6500) and set the TSM rate or playback rate for the parts of the audio or audio-visual tool. You can control it manually. When the override is distributed by the user, the TSM rate used to determine the TSM rate or playback rate of the output signal is taken from the TSM concept lookup 6500, which corresponds to the concept information in the current segment of audio or audio-visual processing. Leverage CSA data structure entries.

The TSM rate regulator 6400 limits the rate of change of the TSM rate at its output using a slew-rate parameter specified by the user to smooth the transition between different TSM rates. The TSM coordinator 6400 may scan the front of the input stream to predict an appropriate rate of change for the audio or audio-visual processing being played. In this method, the time-lag associated with the change in the TSM rate can be reduced as follows.

As can be easily understood, the TSM rate or playback speed output from the TSM concept lookup 6500 can vary rapidly. The input parameter Slew_Limit is used to control the rate of change of the playback speed. Slew_Limit filters the large change in TSM rate or playback speed by allowing the playback speed to change gradually by bringing any magnitude of change in the TSM rate below the amount specified in the parameter Slew_Limit. However, if a small Slew_Limit value is selected, it is important that the amount of time required for a change in the new TSM rate or regeneration rate is long. This can have unwanted side effects that slow down the speed response. For example, taking into account when the input is being reproduced at twice the normal rate and the item of interest is matched, the TSM concept lookup 6500 will output half the normal rate or the TSM rate. In this case, the input parameter Slew_Limit imposes such a long conversion time so that the word of interest will not be played back at the rate determined from the CSA data structure entry. One way to avoid these unwanted side effects is that the TSM concept lookup 6500 scans in front of the audio or audio-visual input stream, obtains a future value of the TSM rate or refresh rate, so that upcoming sections of audio or audio-visual processing are present. To be used to determine the target TSM rate. When the target TSM rate for an upcoming segment is different from the TSM rate that Slew_Limit does not allow the TSM rate to adjust quickly enough, the TSM rate adjuster (6400) is adjusted by adjusting the TSM rate for the current segment in the direction of the specified future TSM rate. You may be able to start converting the TSM rate or playback speed faster. Another way to avoid the unwanted side effects of long conversion times due to small Slew_Limit values is to delay the audio or audio-visual input stream by buffering it to a fixed amount equal to the amount that the TSM concept lookup 6500 would have read ahead. This method transitions the TSM rate conversion in the audio or audio visual input stream somewhat earlier, causing the speed change of the output stream to occur fast enough so that the concept is played back from the TSM concept lookup 6500 at a specified rate so that the rate conversion is Slew_Limit. Get an output stream to stick to.

12 shows a flowchart of an algorithm used in one embodiment of a TSM rate adjuster 6400 to provide a TSM rate or playback rate.

As shown in FIG. 12, the following applies as input to box 7105: (a) TSM rate (TSM_USER) specified by the user received from UIP / PC 6200, (b) TSM concept lookup 6500 TSM rate output by TSM_LUS, (c) User specified slew limit parameter received from UIP / PC 6200 (Slew_Limit) (d) Override flag specified by user received from UIP / PC 6200 ( Override), and (e) an offset value specified by the user as received from the UIP / PC 6200.

At box 7105, a determination is made whether the override is true. If so, then go to box 7900, otherwise go to box 7200. At box 7200 a determination is made whether the offset is 0.0. If so, then move to box 7300; otherwise, go to box 7110.

In box 7300, the following variables are computed: Delta = TSM_Prev-TSM_LUS and Sign = sign [TSM_Prev-TSM_LUS], where TSM_Prev is the TSM rate determined previously. Next, go to box 7400.

At box 7400, a determination is made based on the comparison between Delta and Slew_Limit. If Delta is greater than Slew_Limit, go to box 7500, otherwise go to box 7600.

In box 7600, Delta is set equal to Sing * Delta and proceeds to the next box 7700. In box 7500, Delta is set equal to Sign * Slew_Limit and proceeds to the next box 7700. In box 7700, TSM_Prev is set equal to TSM_Prev + Delta, and the flow proceeds to the next box 7800. At box 7800, TSM_Prev is set equal to TSM, and the TSM value is provided as an output.

At box 7900, the TSM is set equal to TSM_User and proceeds to the next box 7800. Finally, in box 7110, the TSM is set to equal TSM_LUS * (1 + Offset).

Combinations of the foregoing modes of operation are also within the scope of the present invention. For example, a user may use a CSA data structure to combine user offsets for closed caption information embedded in audio or audio-visual processing to be played back to determine the TSM rate required for the output signal.

The output from embodiment 6000 is a stream of digital samples that make up a digitized audio or audio-visual stream, which is a TSM transformation of an input audio or audio visual tool, and, according to the present invention, a CSA data structure and / or user Reflects the TSM rate or playback speed specified by the input. This output represents the LIF tool.

In some embodiments, embodiment 6000 also stores the LIF tool for later playback by the same embodiment or by another playback device. In addition, the digital output may be converted into an analog form for storage on analog devices. Many devices that receive digitized input signals, such as 16-bit pulse code modulation, and provide analog signal outputs from those skilled in the art Known. For example, many commercial equipment are known to those skilled in the art that receive a stream of digitized samples representing a signal and convert these samples into analog signals without loss of fidelity. As will be appreciated by those skilled in the art, whenever the embodiment 600 provides playback for audio-visual processing, the TSM subsystem 6300 accelerates or decelerates the visual information to audio-visualize the audio. Match with. In order to accomplish this in a preferred embodiment, the video signal is “frame-subsampled” or “frame-replicated” according to one of many methods known to those skilled in the art, so that the audio and visual portions of the audio-visual process Keep in sync. Thus, if high speed sampling is required by accelerating audio, the frame stream is subsampled, i.e., the frame is skipped.

11 illustrates an embodiment 6000 in which a separate module should be provided, but with a UI 6100, a UIP / PC 6200, a TSM subsystem 6300, a TSM rate regulator 6400, a TSM concept lookup 6500. ), Concept determiner 6700, and concept information decoder 6800 are implemented as software programs or modules executed on a general purpose computer such as a PC. In addition, digital storage 6075 is implemented as a disk drive or RAM, and a digital analog converter is implemented as a typical accessory of a general purpose computer such as a sound card on a PC. How to implement such a program or module in software will be appreciated by those skilled in the art in light of the detailed description.

Examples of the method and apparatus of the present invention will be described below. A first use of the method and apparatus of the present invention relates to education using audio-visual tools. The apparatus of the present invention allows the TSM rate or playback rate of a particular audio-visual process to be controlled on a per-user basis or on a common basis targeting a particular listener group. For example, assume that educational audio-visual processing is used to elaborate on how viewers set up and use an order entry accounting system on a particular operating system to enter and report a particular type of financial transaction. Let's try it. It is also assumed that the target audience for educational audio-visual processing consists of two groups: (a) novice computer users, accountants (b) advanced computer users who are not familiar with standard accounting tasks. During playback of the audio-visual tool, material is given in the following manner. A specific financial transaction is described following the proper manipulation of the software program's user interface, such as "Select pull-down menu and enter NEW", followed by demonstration of the actual process. While this audio-visual processing is playing at normal speed, accounting professionals who are novice computer users are annoyed with the description of financial transactions already familiar, but because such people are unfamiliar with the use of these interfaces, You may think that the demonstration is too fast during the demonstration of how the entry of. Similarly for advanced computer users but novice accountants, the training speed (speaking speed) during the description of a particular financial transaction will be very cumbersome, but such people will be annoyed by the slow method demonstration of the entry process described above. Embodiments of the present invention solve this problem in the following manner. Two speed contours are mounted in the audio-visual processing. One speed contour is for advanced computer users who are novice accountants (FastCompSlowAcc.spdcon), and the other speed contour is for professional accountants who are novice computer users (FastAccSlowComp.spdcon). Speed contour FastCompSlowAcc.spdcon specifies the TSM speed to accelerate the audio-visual segment containing the demonstration and to slow down the playback speed during the accounting transaction description. Speed contour FastAccSlowComp.spdcon specifies the TSM speed to slow down playback during the demonstration and to accelerate it in the audio-visual segment that describes the accounting transaction. By loading the appropriate velocity contours, each target audience can accept information at a rate that is tailored to their particular understanding of that segment of the audio-visual process. As a result, embodiments of the present invention eliminate the need to create multiple versions of the same audio-visual processing for different target audiences.

In the above example, the user of the audio-visual processing is divided into two specific groups. In many cases, however, the creator of the audio-visual process is unaware of the speed at which the audience sees the material provided to the task. In such a case, each user may load a conceptual speed association data structure containing information about keywords and phrases and phrases about the ideal rate of prediction for a particular concept. Concept speed relationship data structures allow users to view information at a presentation speed that is compatible with their own understanding of other data.

A second example of use of the method and apparatus of the present invention relates to entertainment using audio-visual tools. It will be apparent to those skilled in the art that embodiments of the present invention are not limited to pairing of understanding speed and presentation speed during educational audio-visual processing. In fact, embodiments of the present invention also solve the problem of pairing the presentation speed with the level of entertainment or interest of a particular audio-visual process, thus providing greater enjoyment for the listener / viewer. For example, listeners and movie viewers employ the CSA data structure in accordance with the present invention to control the playback speed of audio or audio-visual processing, thereby allowing scenes or sentences of violence and suspense to be reproduced at high speed, thereby easing excessive anxiety. It can also be avoided. Similarly, listeners and moviegoers who are interested in romantic conversation may use the CSA data structure or velocity contours to slow down playback for this portion. As can be readily understood, each user or family utilizes CSA data structures that reflect their interests to function as a "filter" and uses ordinary embodiments of the invention to process movies, television shows, and other entertainment audio or audio-visual You can also generate a LIF tool for. In addition, as will be readily appreciated, a valuable service according to the present invention may be used to change the content of a process by providing a CSA data structure or speed contour for a particular audio or audio-visual process. For example, a movie grade could be changed from "R" to "PG-13" using velocity contours to remove certain passages containing adult language or concepts.

Once the CSA data structure is created, it is then used to guide the playback speed for audio or audio-visual processing that was not previously heard by the listener. In this way, pairs of concepts and TSM rates, obtained by listening to various audio or audio-visual processes, representing interest, speed of understanding, etc., are captured, stored, and subsequently used to play audio for the first time by the listener. Or guide the TSM rate or playback rate for audio-visual processing. Thus, the CSA data structure can be used to automatically generate a speed contour that is tailored to the user's interest in processing that the user has not heard or to control the playback speed. The ability to control playback speed or generate speed contours for these unacknowledged processes allows the delivery of all audio and audio-visual processing to be delivered to the user depending on the level of user interest in the concepts contained in the CSA data structure. An embodiment of the present invention functions as a matching information filter.

A third use of the method and apparatus of the present invention relates to the production and advertisement of content. In this example, a velocity contour can be determined that samples interest by sampling a segment of a particular target audience or market and keeps the listener / viewer's attention. For example, if an advertisement targets a person who owns a particular brand or computer model, the advertising producer shoots the advertisement, adjusts the speed contours, and delivers the information to the target audience at the appropriate presentation speed. Can be obtained. In addition, different speed contours may be developed and broadcasted to other radio or television stations and / or time slots, depending on the familiarity of the broadcast audience with the subject field being demonstrated. Thus, according to the present invention, when a particular commercial is played during a talk show about home-computer maintenance, it is shortened to 20 seconds using the first speed contour, and a listener unfamiliar with computer terminology when the same commercial is played during evening news. It can be extended to 30 seconds using the second velocity contours for their slower comprehension rates.

A fourth example use of the method and apparatus of the present invention is an application of a CSA data structure containing a concept entry for numeric digits paired with a TSM rate that specifies a slow playback speed. In this case, the method of the present invention can be applied to a voice mail system when the listener retrieves his voice mail message. The concept determiner will perform simple speech recognition to determine the presence of numbers in the message. In this way, all phone numbers and numbers are automatically decelerated and will ease the user's texting process. In addition, embodiments of the present invention can also be used to specify playback speeds for concepts such as dates and addresses without limitation.

A fifth use of the method and apparatus of the present invention relates to foreign language teaching and learning. A learner listening to an audio or audio-visual process containing foreign language learning will utilize the embodiments of the present invention to generate speed contours while listening to various phrases. By showing phrases that are requested to be played back more slowly than other parts, or showing repeated passages, the velocity contours will reflect the learner's understanding of the material. Next, speed contours will be used to score learners and to direct future learning. For example, in listening to fast spoken phrases, one may be able to listen to audio or audio visual tools using custom speed contours, which provide additional learning to help develop word parsing abilities. In addition, the CSA data structures generated while listening to materials demonstrated in foreign languages using embodiments of the present invention can be used to analyze which concepts are problematic for a particular learner. In this way, the same audio or audio-visual processing is demonstrated with the class, where each learner can use the present invention to obtain a CSA data structure containing information of the rate of understanding of the concepts of the materials involved in the processing. have. Next, CSA data structures can be provided graphically or ordered by concept, so that the instructor can score individuals and measure the rate of understanding of the learner group or each learner. For example, it may be possible to develop measures related to the familiarity or understanding of the subject field embodied in the concept and speed of reproduction of user requests.

Those skilled in the art will appreciate that the foregoing description has been provided for the purposes of illustration and understanding only. As such, it is not intended to be exhaustive or to limit the invention to the precise form disclosed.

For example, it will be apparent to one skilled in the art that the above-described audio or audio-visual processing can be input from the Internet into embodiments of the present invention. It will also be apparent to one skilled in the art that embodiments of speed contours or CSA data structures can be used as filter information accessed over the Internet. It will also be apparent that embodiments of the present invention may be included in some of the search engines used to access audio or audio-visual processing on the Internet.

As another example, in an embodiment of the present invention, the velocity contour may include a TSM rate entry, such as, for example, an affinity for a particular portion of audio or audio-visual processing. In such an embodiment of the present invention, the TSM rate of "affinity" (or other notation equivalently translated) instructs the playback system to skip an audio or audio-visual section related to the TSM rate of affinity. . Thus, according to such an embodiment, a user may designate "no interest" in a particular portion when searching or listening to audio or audio-visual processing.

As another example, embodiments of the present invention will be apparent to those skilled in the art, including (a) computer readable media encoded with CSA data structures, (b) computer code encoded with speed contours Possible media, (c) computer readable media having audio or audio visual processing encoded with a CSA data structure, and (d) computer readable media having audio or audio visual processing encoded with speed contours. For computer readable media having audio or audio visual processing encoded with a CSA data structure or speed contour, a method of storing audio or audio visual processing with a CSA data structure or speed contour is known to those skilled in the art. There are many ways to be.

Claims (49)

An apparatus for generating a velocity contour comprising affinity information used to obtain a time scale conversion (TSM) rate and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate, the apparatus comprising: A user input device that receives user information and directs input of a portion of audio or audio-visual processing, A TSM system for generating a TSM portion in response to the identifier of the portion, the portion, and the TSM rate; A TSM monitor responsive to the user information, the identifier of the portion, and the portion, generating a portion of the identifier and the TSM rate associated with the TSM rate, and And a speed contour generator for generating said speed contour in response to an identifier of said associated portion and a TSM rate. The method of claim 1, And in response to said speed contour, storing a speed contour. The method of claim 2, The speed contour is stored with user authentication information included in the user information. The method of claim 3, wherein And the speed contour generator generates one speed contour that is a function of a plurality of speed contours generated for a particular user. The method of claim 3, wherein And the speed contour generator generates one speed contour that is a function of a plurality of speed contours generated for another user. The method of claim 1, The TSM monitor generates one or more specific TSM rates in response to predetermined user information, And wherein the TSM system does not generate the TSM converted portion in response to the one or more specific TSM rates. The method of claim 1, And a display device for displaying the speed contour in response to the speed contour. The method of claim 1, And a playback device for reproducing the TSM portion in response to the TSM portion. The method of claim 8, And in response to the TSM portion, a storage device for storing the TSM portion. The method of claim 1, The affinity information comprises a measure of user interest in the portion. The method of claim 1, And said affinity information comprises a derivative of said TSM rate. A method of generating a velocity contour comprising affinity information used to obtain a time scale conversion (TSM) rate and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate, the method comprising: Obtaining user information and directing input of a portion of audio or audio-visual processing, In response to an identifier of the portion, the portion, and a TSM rate. In response to the user information, the identifier of the portion, and the portion, generating a portion of the identifier associated with the TSM rate and the TSM rate, and In response to the identifier of the relevant portion and the TSM rate, generating the speed contour. Generate a conceptual velocity relationship data structure comprising affinity information used to obtain a time scale conversion (TSM) rate and concept information used to obtain a concept identifier of a portion of audio or audio-visual processing associated with the TSM rate. As a device to make A user input device for receiving user information and directing input of a portion of audio or audio-visual processing, A TSM system for generating a TSM portion in response to the identifier of the portion, the portion, and the TSM rate A concept decoder for generating a concept of the portion in response to the identifier of the portion and the portion, A TSM concept monitor, in response to the user information and the concept, generating a concept identifier and a TSM rate associated with the TSM rate, and And a concept speed relationship data structure generator for generating the concept speed related data structure in response to the TSM rate and the associated concept identifier. The method of claim 13, And in response to the concept speed relationship data structure, a storage device for storing the concept speed relationship data structure. The method of claim 14, And said concept speed relationship data structure is stored together with user authentication information contained in user information. The method of claim 15, And said concept speed relationship data structure generator generates one concept speed relationship data structure that is a function of a plurality of concept speed relationship data structures generated for a particular user. The method of claim 15, And the concept speed relationship data structure generator generates one concept speed relationship data structure that is a function of a plurality of concept speed relationship data structures generated for a plurality of different users. The method of claim 13, The TSM concept monitor generates one or more specific TSM rates in response to predetermined user information, And wherein the TSM system does not generate a TSM portion in response to the one or more specific TSM rates. The method of claim 13, And in response to the concept speed relationship data structure, display device for displaying the concept speed relationship data structure. The method of claim 13, And a playback device for reproducing the TSM portion in response to the TSM portion. The method of claim 20, And responsive to the TSM portion, storing the TSM portion. Generate a conceptual velocity relationship data structure comprising affinity information used to obtain a time scale conversion (TSM) rate and concept information used to obtain a concept identifier of a portion of audio or audio-visual processing associated with the TSM rate. As a way to make Obtaining user information and directing input of a portion of audio or audio-visual processing, In response to an identifier of the portion, the portion, and a TSM rate, generating a TSM version of the portion, Generating a concept of the portion in response to the identifier of the portion and the portion, Generating a concept related to the TSM rate and the TSM rate in response to the user information and the concept, and In response to the TSM rate and the related concept, generating a concept velocity relationship data structure. Audio or audio in conjunction with a velocity contour that includes affinity information used to obtain a time scale conversion (TSM) rate and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate A device for reproducing visual processing, An input device for directing input of a portion of audio or audio-visual processing, A TSM system for generating a TSM portion in response to the identifier of the portion, the portion, and the TSM rate; A playback device for reproducing the TSM portion in response to the TSM portion, and And a TSM rate determiner for generating a TSM rate in response to the speed contour and the identifier of the portion. The method of claim 23, wherein The input device further comprises a user input device for receiving user information, And the timescale transformation also generates the TSM rate in response to the user information. The method of claim 24, The user information includes an offset, And the TSM rate determiner uses the offset to generate the TSM rate. The method of claim 24, The user information includes a user specified quantization step at a TSM rate, And wherein said TSM rate determiner generates a TSM rate using said user specified quantization step. In conjunction with a velocity contour, including affinity information used to obtain a time scale conversion (TSM) rate and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate. As a method of playing audio-visual processing, Directing an input of a portion of the audio or audio-visual process, Generating a TSM portion in response to an identifier of the portion, the portion, and a TSM rate, Responsive to the TSM portion, playing the TSM portion, and Generating a TSM rate in response to the portion's identifier and speed contours. Associated with a concept velocity relationship data structure comprising affinity information used to obtain a time scale conversion (TSM) rate and concept information used to obtain a concept identifier of a portion of audio or audio-visual processing associated with the TSM rate. Device for reproducing audio or audio-visual processing, An input device for directing input of a portion of the audio or audio-visual processing, A TSM system for generating a TSM portion in response to the identifier of the portion, the portion, and the TSM rate; A reproducing apparatus reproducing the TSM portion in response to the TSM portion, A concept decoder for generating a concept for the portion in response to the identifier of the portion and the portion, and And responsive to the concept and the conceptual velocity relationship data structure, a TSM concept look-up for generating the TSM rate. The method of claim 28, The input device further comprises a user input device for receiving user information, And wherein said TSM concept lookup is for generating said TSM rate in response to said user information. The method of claim 29, The user information includes an offset, Wherein the TSM concept lookup uses the offset to generate a TSM rate. The method of claim 29, The user information includes a user specified quantization step at a TSM rate, Wherein said TSM concept lookup generates said TSM rate using said user specified quantization step. The method of claim 29, The user information includes a concept and a TSM rate for the concept, And the TSM concept lookup generates a TSM rate using the user-specified concept and the TSM rate. Associated with a concept velocity relationship data structure comprising affinity information used to obtain a time scale conversion (TSM) rate and concept information used to obtain a concept identifier of a portion of audio or audio-visual processing associated with the TSM rate. A method of playing audio or audio-visual processing, Directing an input of a portion of the audio or audio-visual process, In response to an identifier of the portion, the portion, and a TSM rate, generating a TSM portion, Responsive to the TSM portion, playing the TSM portion, In response to the identifier of the portion and the portion, generating a concept of the portion, and And in response to the concept and the concept speed relation data structure, generating the TSM rate. Obtaining a TSM rate, a velocity contour comprising affinity information used to obtain a time scale conversion (TSM) rate and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate An apparatus for converting into a conceptual velocity relationship data structure comprising affinity information used to obtain and concept information used to obtain a concept identifier for a portion of audio or audio-visual processing associated with the TSM rate. An input device, in response to the speed contours, determining a portion of the TSM rate and an identifier of the portion, and directing input of the portion of the audio or audio-visual process; A concept decoder for generating a concept for the portion in response to the identifier of the portion and the portion, A TSM concept monitor correlating the concept identifier with the TSM rate in response to the concept and the TSM rate, and And a concept speed relationship data structure generator for generating the concept speed relationship data structure in response to the TSM rate and the associated concept identifier. A concept velocity relationship data structure comprising affinity information used to obtain a time scale conversion (TSM) rate and concept information used to obtain a concept identifier for a portion of audio or audio-visual processing associated with the TSM; An apparatus for converting into concept contours comprising affinity information used to obtain a TSM rate and concept information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM, comprising: An input device for directing input of a portion of the audio or audio-visual process and determining an identifier of the portion; A concept decoder for generating a concept for the portion in response to the identifier of the portion and the portion, A TSM concept lookup that generates the TSM rate in response to the concept and the conceptual velocity relationship data structure, and And a speed contour generator for generating the speed contour in response to the TSM rate and the identifier of the associated portion. A method for generating a listener interest filtering tool for audio or audio-visual processing, Generating, for one or more user categories, one or more average velocity contours for one or more audio or audio visual processing, Converting the one or more average velocity contours into one or more conceptual velocity relationship data structures, and And forming one listener interest filtered concept velocity relationship data structure from the one or more concept velocity relationship data structures. The method of claim 36, Converting the listener interest filtered conceptual velocity relationship data structure into a listener interest filtered velocity contour. The method of claim 36, Generating a listener interest filtered audio or audio-visual process using the listener interest filtered concept velocity relationship data structure. The method of claim 37, Generating a listener interest filtered audio or audio-visual process using the listener interest filtered rate contour. The method of claim 36, Modifying the listener interest filtering filtered concept velocity relationship data structure. The method of claim 13, The concept decoder comprises an algorithm for recognizing a category of concepts 42. The method of claim 41 wherein One of the categories comprises a number. An apparatus for generating a velocity contour comprising affinity information used to obtain a time scale conversion (TSM) rate and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate, the apparatus comprising: A user input device for receiving user information and directing input of a portion of the audio or audio-visual process; A display device for displaying a representation of an input portion and a TSM rate associated with the portion in response to an identifier of the portion; In response to the user information, an editor for providing an identifier of a user selected portion of the display and a TSM rate specified by the user for the portion selected by the user; And a speed contour generator for generating a speed contour in response to the user specified TSM rate, the associated TSM rate, and the identifier of the user selected portion. The method of claim 43, Wherein the display of the display device includes text material from closed caption material associated with the input portion. The method of claim 43, And the display of the display device includes text material determined by speech recognition of the input portion. Encoded into a CSA data structure containing affinity information used to obtain a time scale conversion (TSM) rate and concept information used to obtain a concept identifier of a portion of audio or audio-visual processing associated with the TSM rate. Computer readable media. The method of claim 46, And the computer readable medium is further encoded with audio or audio-visual processing. Velocity-encoded computer readings comprising affinity information used to obtain a time scale conversion (TSM) rate and identifier information used to obtain an identifier of a portion of audio or audio-visual processing associated with the TSM rate. Media available. 49. The method of claim 48 wherein And the computer readable medium is further encoded with audio or audio-visual processing.