WO2013157190A1 - 音声処理装置、音声処理方法、プログラムおよび集積回路 - Google Patents
音声処理装置、音声処理方法、プログラムおよび集積回路 Download PDFInfo
- Publication number
- WO2013157190A1 WO2013157190A1 PCT/JP2013/001568 JP2013001568W WO2013157190A1 WO 2013157190 A1 WO2013157190 A1 WO 2013157190A1 JP 2013001568 W JP2013001568 W JP 2013001568W WO 2013157190 A1 WO2013157190 A1 WO 2013157190A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- unit
- section
- boundary
- feature
- scene
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title description 2
- 238000004364 calculation method Methods 0.000 claims abstract description 139
- 230000008859 change Effects 0.000 claims abstract description 128
- 230000005236 sound signal Effects 0.000 claims abstract description 96
- 239000013598 vector Substances 0.000 claims description 227
- 238000012545 processing Methods 0.000 claims description 141
- 238000000034 method Methods 0.000 claims description 41
- 230000008569 process Effects 0.000 claims description 28
- 238000001514 detection method Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 33
- 230000006870 function Effects 0.000 description 33
- 238000009826 distribution Methods 0.000 description 18
- 238000000605 extraction Methods 0.000 description 14
- 238000013500 data storage Methods 0.000 description 10
- 230000001052 transient effect Effects 0.000 description 8
- 238000001228 spectrum Methods 0.000 description 6
- 239000000203 mixture Substances 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000001186 cumulative effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000007796 conventional method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 235000012054 meals Nutrition 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 240000006829 Ficus sundaica Species 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
- H04N5/60—Receiver circuitry for the reception of television signals according to analogue transmission standards for the sound signals
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
- G10L25/54—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for retrieval
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
- G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4394—Processing of audio elementary streams involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/147—Scene change detection
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
- G10L25/57—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for processing of video signals
Definitions
- the present invention relates to a technique for detecting a change point of a feature such as frequency from an audio signal.
- a technique for detecting a change point between scenes (hereinafter referred to as a scene change point) using audio information such as sound pressure and frequency in AV content has attracted attention.
- the AV contents photographed by the user are rich in variety of contents according to the user's taste, and as a result, the scene change points to be detected are also diverse. It is difficult to cover the scene change points to be detected using only a specific method for this wide variety of scene change points, and there are scene change points that are difficult to detect even using the above-mentioned conventional method Do.
- an object of the present invention is to provide an audio processing apparatus capable of detecting a scene change point which has been difficult to detect by the conventional method.
- a speech processing apparatus comprises: feature calculation means for calculating a section feature representing a feature of an audio signal for each unit section of a predetermined time length of an input audio signal; Boundary calculation means for calculating boundary information on a boundary between another section and a similar section consisting of a plurality of unit sections similar to the section feature including the unit section, for each of the unit sections; Calculating a priority of each of the boundaries indicated by and determining whether each of the boundaries is a scene change point based on the priority.
- the voice processing apparatus of the present invention it is possible to detect the boundary of similar sections set for each of a plurality of unit sections as a scene change point.
- Block diagram showing an example of the functional configuration of a video viewing device provided with an audio processing device
- Block diagram showing an example of functional configuration of the voice processing device A diagram showing an example of a reference section used to calculate a reference vector Diagram showing reference vectors, frequency vectors and thresholds using the concept of vector space
- Flow chart showing operation of voice processing apparatus Flow chart showing section extension reference index calculation processing
- AV content is composed of sections of various lengths according to the granularity of capturing a scene.
- AV content is obtained by shooting a party, and is composed of the scene shown in FIG.
- FIG.1 (b) assumes that the audio signal in the scene is expressed.
- the party has a scene 10 of toasts and a scene 20 of subsequent meals, and the scene 20 of meals further talks to the scene 21 which is mainly eating and to the main It comprises a scene 22.
- the scene 20 is a transitional scene which changes from the scene 21 which is eating mainly to the scene 22 which is mainly talking.
- the transition point in the transient scene is the boundary of a section (similar section) where voice information is similar to other section, for example, a similar section in the first half of the scene and a similar section in the second half of the scene. It was found that it could be detected as a boundary with
- the similar section in the audio signal can be obtained by comparing the audio information of a reference position (reference position) in the section with the audio information before and after that. Therefore, the similar section in the transient scene can be obtained by designating one point of the transient scene as the reference position.
- a method of selecting the detected change points and suppressing the number of change points to be indexed can be considered.
- Embodiment 1 ⁇ 1-1.
- Overview> 1 shows an outline of a voice processing apparatus according to an embodiment of the present invention.
- the audio processing apparatus first calculates, for unit sections obtained by dividing an audio signal included in a moving image file into predetermined time lengths, feature quantities representing the features of the audio signal in each unit section.
- the speech processing device calculates, for each unit section, the boundary between the section similar to the unit section and another section based on the calculated similarity of the feature amount.
- the speech processing device calculates the boundary degree for each calculated boundary, and detects a scene change point based on the boundary degree.
- the speech processing device outputs the detected scene change point as index information.
- the boundary degree is the number of boundaries indicating the same time.
- the scene desired by the user and the other scenes are calculated under the assumption that boundaries indicating the same time are calculated from unit sections belonging to the same scene desired by the user. It is possible to preferentially detect the change point of. ⁇ 1-2.
- Data> data used in the speech processing apparatus according to the present embodiment will be described.
- ⁇ Video file> The moving image file is composed of an audio signal X (t) and a plurality of image data.
- the audio signal X (t) is time series data of amplitude values, and can be expressed by a waveform as shown in FIG. FIG.
- the feature quantity vector M represents the feature of the audio signal X (t), and in the present embodiment, the mel frequency of the audio signal divided into each first unit interval is assumed as the feature quantity vector M.
- the cepstral coefficient MFCC (Mel-Frequency Cepstrum Coefficients) is used.
- the first unit section is a section of a predetermined length (for example, 10 msec) on the time axis of the audio signal X (t), and a section from time T n to T n + 1 in FIG. It is.
- the feature amount vector M is calculated for each first unit section. Therefore, as shown in FIG. 3, 100 feature quantity vectors M are generated from the audio signal from time 0 sec to time 1 sec.
- FIG. 3 is an example of the feature quantity vector M calculated for each first unit section from time 0 sec to time 1 sec.
- a mixed normal distribution model (GMM, Gaussian Mixture Model) is adopted, and each anchor model Ar is configured of parameters defining a normal distribution.
- the anchor model is created for each of a plurality of types (1024 types in this embodiment) of sound segments, and a feature quantity appearance probability function b Ar (M corresponding to each of the plurality of types of sound segments Is represented by.
- the likelihood vector F is a vector having as a component the likelihood L r for each of a plurality of types of sound segments of the audio signal X (t) calculated based on the anchor model A r .
- FIG. 5 is a diagram showing likelihood vectors F in two first unit intervals.
- FIG. 5A shows, for example, a likelihood vector corresponding to the n-th first unit section (section from time (10 ⁇ n) msec to time (10 ⁇ (n + 1)) msec counting from time 0).
- Fn is shown
- FIG. 5B corresponds to, for example, the m-th first unit interval (interval from time (10 ⁇ m) msec to time (10 ⁇ (m + 1)) msec) counting from time 0
- the likelihood vector Fm (n ⁇ m) the likelihood vector
- the frequency vector NF is a vector that represents a feature of each second unit section of the audio signal, and in particular, is a vector that represents the appearance frequency of each sound segment related to the second unit section of the audio signal.
- the second unit section is a section of a predetermined length (for example, 1 sec) on the time axis of the audio signal X (t), and as shown in FIG. 6, a plurality of continuous first unit sections. It corresponds to the length.
- the frequency vector NF normalizes the normalized cumulative likelihood of the likelihood vector F included in the second unit section, that is, the cumulative value of each component of the likelihood vector F included in the second unit section It is
- normalization means setting the norm of the frequency vector NF to one.
- FIG. 7 is a schematic view showing an example of the frequency vector NF.
- the boundary information is information on the boundary of the similar section calculated in every second unit section of the audio signal and the similar section whose frequency vector is similar.
- the speech processing apparatus calculates, as boundary information, the time at the beginning of the similar section and the time at the end of the similar section.
- FIG. 8 is an example of boundary information calculated in the present embodiment. For example, from the 0-th second unit section (a section between time 0 and 1s), it is shown that boundary information indicating 0s as the start time and 10s as the end time is calculated.
- the boundary degree is the number of boundary information indicating the same time.
- the boundary information indicating 0s at the start time or the end time is the 0th second unit section (a section between time 0 to 1s) and the first second unit section (time 1 to 2s). Since there are three pieces of boundary information calculated from the interval between and the second second unit interval (a interval between time 2 to 3s), the boundary degree of time 0s is 3.
- FIG. 9 is an example of a graph showing time on the horizontal axis and the calculated degree of boundary on the vertical axis.
- FIG. 10 is a block diagram showing an example of a functional configuration of the video viewing device 100.
- the video viewing device 100 includes an input device 101, a content storage device 102, an audio extraction device 103, an audio processing device 104, an index storage device 105, an output device 106, an anchor model creation device 107, an audio data storage device. And an interface device 109.
- the input device 101 is realized by a disk drive device or the like, and when the recording medium 120 is loaded, reads the moving image file from the recording medium 120 and stores the acquired moving image file in the content storage device 102.
- the recording medium 120 is a medium having a function of storing various data, and is, for example, an optical disc, a flexible disc, an SD card, a flash memory, or the like.
- the content storage device 102 is realized by, for example, a hard disk device, and stores the moving image file acquired from the recording medium 120 by the input device 101.
- a moving image file is stored with a unique ID attached to each moving image file.
- the audio extraction device 103 extracts an audio signal from the moving image file stored in the content storage device 102, and inputs the extracted audio signal to the audio processing device 104.
- the speech extraction apparatus 103 performs a decoding process on the encoded audio signal to generate an audio signal X (t) as shown in FIG.
- the voice extraction device 103 is realized by, for example, a processor that executes a program.
- the speech processing unit 104 detects a scene change point based on the audio signal X (t) input from the speech extraction unit 103, and stores index information including the detected scene change point in the index storage unit 105. Details of the functional configuration of the voice processing device 104 will be described later.
- the index storage device 105 is realized by, for example, a hard disk device, and stores index information input from the voice processing device 104.
- the index information includes the ID of the moving image file and the time (the time of the scene change point) in the moving image file.
- the output device 106 acquires index information from the index storage device 105, and outputs video data (a part of a moving image file stored in the content storage device 102) corresponding to the acquired index information to the display device 130.
- the output device 106 adds information on a user interface (UI: User Interface) such as a progress bar with a mark at the time corresponding to the index information, to the video data output to the display device 130, for example.
- UI User Interface
- the voice extraction device 103 is realized by, for example, a processor that executes a program.
- the anchor model creation device 107 creates an anchor model Ar based on the audio signal stored in the voice data storage device 108, and outputs the created anchor model Ar to the voice processing device 104. Details of the functional configuration of the anchor model creation device 107 will be described later.
- the audio signal used when the anchor model creating unit 107 creates anchor models A r from another of the plurality of video files and video files to be subjected to detection of a scene change point is obtained in advance extracted Audio signal.
- the audio data storage device 108 is realized by, for example, a hard disk device, and stores in advance audio data used when the anchor model creation device 107 creates the anchor model Ar .
- the interface device 109 includes an operation unit (not shown) such as a keyboard, receives an input operation from the user, and outputs the operation information of the progress bar to the output device 106.
- the number K of anchor models to be created is an anchor model The creation device 107 is notified.
- the voice processing device 104 includes a memory (not shown) and a processor (not shown), and the processor executes the program written in the memory to realize the configuration shown in FIG.
- FIG. 11 is a block diagram showing an example of a functional configuration of the speech processing apparatus 104.
- the speech processing device 104 includes a feature quantity vector generation unit 201, a likelihood vector generation unit 202, a likelihood vector buffer 203, a frequency vector generation unit 204, a frequency vector buffer 205, and an interval extension reference index calculation unit 206. And a boundary information calculation unit 207, an index generation unit 208, and an anchor model storage unit 209. Each component will be described below.
- the feature quantity vector generation unit 201 generates a feature quantity vector M for each first unit interval based on the audio signal X (t) input from the speech extraction apparatus 103.
- the feature quantity vector generation unit 201 calculates the power spectrum S ( ⁇ ) of the audio signal X (t) in the first unit section (see FIG. 2B).
- the power spectrum S ( ⁇ ) is obtained by converting the audio signal X (t) from the time axis to the frequency axis and squaring each frequency component.
- the feature quantity vector generation unit 201 calculates a mel frequency spectrum S ( ⁇ mel ) obtained by converting the frequency axis of the power spectrum S ( ⁇ ) into a mel frequency axis (see FIG. 2C).
- the feature quantity vector generation unit 201 calculates a mel frequency cepstrum from the mel frequency spectrum S ( ⁇ mel ), and sets a predetermined number (26 in the present embodiment) of components as the feature quantity vector M.
- Anchor model storage unit 209 is constituted by a part of the area of the memory and stores the anchor model A r created by an anchor model creating unit 107.
- the anchor model storage unit 209 before the execution of each processing of the audio processing device 104 stores in advance the anchor model A r.
- Likelihood vector generation unit 202 by using the anchor model A r stored in the feature vector M and the anchor model storage unit 209 is generated by the feature vector generating unit 201, the audio signal X (t), sound The likelihood L r for each segment is calculated, and a likelihood vector F having each likelihood as a component is generated for each first unit section.
- the likelihood vector buffer 203 is configured of a partial area of the memory, and stores the likelihood vector F generated by the likelihood vector generation unit 202.
- the frequency vector generation unit 204 generates a frequency vector NF for each second unit interval based on the likelihood vector F stored in the likelihood vector buffer 203.
- the frequency vector buffer 205 is configured by a partial area of the memory, and stores the frequency vector NF generated by the frequency vector generation unit 204.
- the section extension reference index calculation unit 206 calculates, for each of the second unit sections, a reference section, a reference vector S, and a threshold value Rth, which are reference indices of section extension processing to be described later.
- the section extension reference index calculation unit 206 acquires the frequency vector of the reference section from the frequency vector buffer 205 using the plurality of second unit sections in the vicinity of the second unit section to be processed as the reference section, and acquires the plurality of acquired frequency vectors
- the center of gravity is calculated as a reference vector S.
- FIG. 12 shows an example of calculation of the reference vector S using the frequency vectors (NF1 to NF9) of the reference section in a case where a total of nine sections in the vicinity of the second unit section to be processed are set as the reference section.
- the section extension reference index calculation unit 206 further calculates Euclidean distances between the plurality of frequency vectors NF used when generating the reference vector S and the reference vector S, and Euclidean distances between the reference vector S and the reference vector S.
- the Euclidean distance between the frequency vector NF having the largest value of i and the reference vector S is calculated as the threshold value Rth used to determine whether or not it belongs to the similar section.
- FIG. 13 is a diagram showing the reference vector S, each frequency vector NF, and the threshold value Rth using the concept of vector space.
- white circles respectively indicate frequency vectors NF (corresponding to the frequency vectors NF1 to NF9 in the reference section shown in FIG. 12) used for calculation of the reference vector S, and a hatched area having a circular shape
- a black circle at the center of indicates the reference vector S.
- the length of the arrow from the reference vector S to each frequency vector NF indicates the Euclidean distance between the frequency vector NF and the reference vector S, and the maximum distance is the threshold value Rth.
- the boundary information calculation unit 207 calculates, for the second unit section, a similar section including sections similar to the frequency vector, and specifies the time of the start end and the time of the end of the similar section.
- the boundary information calculation unit 207 uses the frequency vector NF stored in the frequency vector buffer 205, the second unit section to be processed, and the reference index calculated by the section extension reference index calculation unit 206 (reference section, reference vector S And the threshold value Rth), and outputs the identified start time and end time as boundary information to the index generation unit 208.
- the boundary information calculation unit 207 sets the reference interval calculated by the interval extension reference index calculation unit 206 as the initial value of the similar interval.
- the boundary information calculation unit 207 determines whether or not to include the target section in the similar section with the second unit section immediately before the similar section as the target section while going backward in the time axis reverse direction, as shown in FIG. Go. Specifically, the boundary information calculation unit 207 calculates the Euclidean distance between the frequency vector NF and the reference vector S in the target section, and when the calculated Euclidean distance is less than or equal to the threshold Rth, the target section is similar. Include in the section. The boundary information calculation unit 207 repeats the above processing, and specifies the time of the beginning of the similar section when the calculated Euclidean distance first exceeds the threshold value Rth.
- the above process is referred to as an interval extension process because each similar interval is extended one by one.
- the boundary information calculation unit 207 also performs section extension processing in the forward direction of the time axis to specify the end time of the similar section.
- the boundary information calculation unit 207 determines whether or not the target section is included in the similar section at the time of section extension processing, and at the same time, the length of the similar section is shorter than the length le of the similar section set in advance. Also judge. If the calculated Euclidean distance does not exceed the threshold value Rth and the length of the similar section is shorter than the upper limit le of the length of the similar section set in advance, the boundary information calculation unit 207 selects the target section as the similar section. Include in If the length of the similar section is equal to or more than the upper limit le of the length of the similar section, the boundary information calculation unit 207 calculates the boundary information of the similar section at that time. The upper limit le of the length of the similar section uses a value set in advance.
- the boundary information calculation unit 207 calculates boundary information for each second unit section (see FIG. 8).
- FIG. 15 is a block diagram showing an example of a functional configuration of the index generation unit 208.
- the index generation unit 208 includes a voting unit 301, a threshold calculation unit 302, and a boundary determination unit 303. Each component will be described below.
- the voting unit 301 calculates, for each time indicated by the boundary information calculated by the boundary information calculation unit 207, the number of pieces of boundary information indicating the same time as the boundary degree.
- the voting unit 301 votes each of the boundary information input from the boundary information calculation unit 207 at the time indicated by the input boundary information (by adding 1 to the boundary degree KK i corresponding to the time i), Calculate the boundary level.
- the voting unit 301 votes at both the start time and the end time specified by one piece of boundary information.
- the boundary determination unit 303 uses the boundary degree KK i for each time calculated by the voting unit 301 and the threshold value TH calculated by the threshold calculation unit 302 to set a time satisfying the condition of the following equation (4) as a scene change point It is determined that the index information is output to the index storage unit 105 as index information.
- the audio processing device 104 generates index information of a moving image file according to the configuration from ⁇ 3-2-1> to ⁇ 3-2-9>. Description of the configuration of the video viewing device 100 of FIG. 8 will be continued.
- the anchor model creation device 107 is configured of a memory (not shown) and a processor (not shown), and the processor executes the program written in the memory to realize the configuration shown in FIG.
- FIG. 16 is a block diagram showing the functional configuration and peripheral devices of the anchor model creating device.
- anchor model generation device 107 includes feature amount vector generation unit 401, feature amount vector classification unit 402, and anchor model generation unit 403, and based on voice data stored in voice data storage device 108. It has a function of creating an anchor model and storing the created anchor model in the anchor model storage unit 209. Each component will be described below.
- the feature amount vector generation unit 401 generates a feature amount vector M for each of the first unit intervals based on the audio data stored in the audio data storage device 108.
- the feature quantity vector classification unit 402 clusters (classifies) the feature quantity vectors generated by the feature quantity vector generation unit 401.
- the feature quantity vector classification unit 402 classifies a plurality of feature quantity vectors M into K clusters by the K-means method based on the number K of anchor models Ar input from the interface device 109.
- K 1024.
- FIG. 17 is a flowchart showing the operation of the speech processing apparatus 104. As shown in the figure, the audio processing device receives an audio signal (step S1701).
- the speech processing device 104 generates a section feature (a feature amount vector, a likelihood vector and a frequency vector) representing the feature for each second unit section of the audio signal, using the input audio signal (Step S1702).
- the generation of the section feature includes feature quantity vector generation processing by the feature quantity vector generation unit 201, likelihood vector generation processing by the likelihood vector generation unit 202, and frequency vector generation processing by the frequency vector generation unit 204.
- the voice processing apparatus 104 selects one second unit section to be processed, and calculates the section extension reference index executed by the section extension reference index calculating unit 206 in step S1703.
- the boundary information calculation process performed by the boundary information calculation unit 207 is performed.
- the voice processing device 104 performs loop processing until the processing in step S1703 and step S1704 is performed on all the second unit sections.
- the index generation unit 208 of the speech processing apparatus 104 performs index generation processing (step S1705).
- the speech processing device 104 outputs the index information generated by the index generation unit 208 to the index storage device (step S1706).
- FIG. 18 is a flowchart showing details of reference index calculation processing executed by the section extension reference index calculation unit 206 in step S1703 in FIG. As shown in FIG. 18, in the reference index calculation process, the section extension reference index calculation unit 206 receives the second unit section to be processed and the frequency vector held in the frequency vector buffer 205 (step S1801).
- the section extension reference index calculation unit 206 sets a second unit section to be processed and a second unit section of nine sections in total of four sections before and after the second unit section as a reference section (step S1802).
- the section expansion reference index calculation unit 206 calculates the centroid of the frequency vector (NF1 to NF9) in the reference section input by the frequency vector buffer 205, and sets it as a reference vector S (step S1803).
- the section extension reference index calculation unit 206 determines Euclidean distances D (S, NF1),..., D (S, NF9) between the reference vector S and the frequency vectors (NF1 to NF9) in the reference section.
- the calculated and calculated maximum value of the Euclidean distance is set as the threshold value Rth (step S1804).
- section extension reference index calculating unit 206 outputs the calculated reference index to the boundary information calculating unit 207 (step S1805).
- FIG. 19 is a flowchart showing details of the boundary information calculation process performed by the boundary information calculation unit 207 of step S1704 in FIG. As shown in FIG. 19, in the boundary information calculation process, the boundary information calculation unit 207 sets the second unit section to be processed, the reference index calculated by the section extension reference index calculation unit 206, and the length of the similar section set in advance. The upper limit of frequency and the frequency vector held in the frequency vector buffer 205 are input (step S1901).
- the boundary information calculation unit 207 sets the reference interval input from the interval expansion reference index calculation unit 206 as the initial value of the similar interval (step S1902).
- the boundary information calculation unit 207 executes the processing of steps S1903 to S1906 on the initial value of the similar section set in step S1902, and performs section extension processing in the direction opposite to the time axis of the audio signal.
- the boundary information calculation unit 207 sets the second unit section immediately before the similar section on the time axis of the audio signal as the target section (step S1903).
- the boundary information calculation unit 207 calculates the Euclidean distance D (NF, S) between the frequency vector NF of the target section input from the frequency vector buffer 205 and the reference vector S input by the section extension reference index calculation unit 206.
- the calculated Euclidean distance D (NF, S) is compared with the threshold value Rth input from the section extension reference index calculation unit 206 (step S1904).
- step S1904 If the calculated Euclidean distance D (NF, S) is shorter than the threshold value Rth (step S1904: YES), the boundary information calculation unit 207 updates the similar section so as to include the target section (step S1905).
- the boundary information calculation unit 207 compares the length of the similar section with the upper limit le of the length of the similar section (step S1906), and the length of the similar section is shorter than the upper limit le. For example (step S1906: YES), the process returns to step S1803, and if the length of the similar section is equal to or more than the upper limit le (step S1906: NO), the process proceeds to step S1911.
- step S1904 NO
- the boundary information calculation unit 207 ends the section expansion processing in the time axis reverse direction of the audio signal and proceeds to step S1907
- the process advances to step S1910 to perform period extension processing in the time axis forward direction of the audio signal.
- the section extension process in the forward direction of the time axis is the same process as the section extension process in the reverse direction of the time axis except that the second unit section immediately after the similar section in step S1907 is the target section.
- the boundary information calculation unit 207 calculates boundary information including the time of the start and the time of the end of the similar section at the time when the section expansion process in the reverse direction of the time axis and the area expansion process in the forward direction of the time axis are finished (step S1911). .
- the boundary information calculation unit 207 outputs the calculated boundary information to the index generation unit 208 (step S1912).
- FIG. 20 is a flow chart showing the operation of index generation processing executed by the index generation unit 208 in step S1705 in FIG. As shown in FIG. 20, in the index generation process, the index generation unit 208 receives the boundary information calculated by the boundary information calculation unit 207 (step S2001).
- the voting unit 301 votes at the time indicated by the boundary information to calculate the boundary degree (step S2002).
- the threshold calculation unit 302 calculates a threshold using the boundary degree calculated by the voting unit 301 (step S2003).
- the boundary determination unit 303 detects a scene change point using the boundary degree calculated by the voting unit 301 and the threshold value calculated by the threshold calculation unit 302, and generates index information in which the detected scene change point is indexed (see FIG. Step S2004).
- the boundary determination unit 303 outputs the generated index information to the index storage unit 105 (step S2005).
- the speech processing apparatus calculates and calculates a section feature (a feature amount vector, a likelihood vector and a frequency vector) indicating a feature in a unit section of the audio signal for each unit section of a predetermined time length of the audio signal.
- a scene change point is detected from the boundary of similar sections consisting of similar sections of the section feature.
- the audio processing device can detect even a scene change point at which audio information changes gradually.
- the speech processing device calculates the number of pieces of boundary information indicating the same boundary for each boundary as the priority (boundary degree) with respect to the boundary information calculated for each unit section, and only the boundary of the priority exceeding the threshold Is indexed as a scene change point.
- the speech processing apparatus can preferentially detect boundaries calculated from many unit sections (second unit sections) as scene change points desired by the user. Furthermore, in order to sort out the change points to be indexed, the user can easily search for a desired scene.
- Second Embodiment differs from the first embodiment in two points.
- the boundary degree is a calculation method of the boundary degree.
- the number of boundaries indicating the same time is calculated as the boundary degree.
- the maximum value of the boundary change degree in the boundary information indicating the same time is calculated as the boundary degree.
- the boundary change degree indicates how much the section feature (feature amount vector, likelihood vector and frequency vector) changes in the similar section as the boundary information of the similar section by the boundary information calculation unit. Calculated as an indicator.
- index information Another point that is different from the first embodiment is index information.
- the first embodiment only the time of the scene change point is used as the index information.
- classification information in which scene change points are classified by sound environment information is added to index information.
- the sound environment information is information representing the feature of the audio signal at the scene change point, and is calculated by the boundary information calculation unit as the boundary information of the similar section using the section feature in the similar section.
- FIG. 21 is a block diagram showing an example of a functional configuration of the speech processing apparatus 110 according to the second embodiment.
- the speech processing device 110 includes a boundary information calculation unit 211 instead of the boundary information calculation unit 207 in comparison with the speech processing device 104 according to the first embodiment, and an index generation unit 212 instead of the index generation unit 208.
- the boundary information calculation unit 211 In addition to the function of the boundary information calculation unit 207, the boundary information calculation unit 211 further uses, as boundary information, the feature of the audio signal near the second unit section to be processed and the change of the feature of the audio signal at the boundary of the similar section It has a boundary change degree indicating a degree, and a function of calculating sound environment information indicating a representative sound environment in a similar section.
- the boundary information calculation unit 211 sets the reference vector S calculated at the time of extending a section in the section extension process in the reverse direction of the time axis as the start change degree Din (the boundary change at the start of the similar section). And Euclidean distance between the frequency vector NF and the frequency vector NF, which exceeds the threshold Rth. That is, the Euclidean distance between the reference vector S and the frequency vector NF of the second unit section immediately before the similar section is used.
- the boundary information calculation unit 211 uses the second unit section at the start of the similar section instead. As the degree), Euclidean distance between the reference vector S and the frequency vector NF of the second unit section immediately after the similar section is used.
- the boundary information calculation unit 211 uses the reference vector S as sound environment information.
- the boundary information calculation unit 211 calculates, as boundary information, the time at the beginning of the similar section, the change in the start, the time at the end, the change in the end, and the sound environment information.
- FIG. 23 is a block diagram showing an example of a functional configuration of the index generation unit 212.
- the index generation unit 212 includes a boundary degree calculation unit 311 instead of the voting unit 301 in comparison with the index generation unit 208 according to the first embodiment, and the sound environment classification unit 312 further includes a boundary determination unit 303 and index storage. It is inserted between the devices 105.
- the boundary degree calculation unit 311 calculates, as the boundary degree, the maximum value of the boundary change degree of the boundary information for each time indicated by the boundary information calculated by the boundary information calculation unit 211. More specifically, the boundary calculating unit 311 as a boundary of the time T i, the maximum value of the terminal change degree time start change degree and termination at the boundary information of Ti in the boundary information of the start time is Ti calculate.
- the boundary degree calculation unit 311 associates the sound environment information of the boundary information given the maximum value of the boundary change degree with the sound environment information of the boundary (time).
- the sound environment classification unit 312 divides the sound environment information associated with the time determined as a scene change point by the boundary determination unit 303 into a plurality of groups (for example, 3 groups) using, for example, the K-means method. Classify.
- the sound environment classification unit 312 adds the classification information of the classification result to the index information, and outputs the index information with the classification information to the index storage unit 105.
- FIG. 24 shows a specific example of index information to which classification information is added.
- the speech processing apparatus uses, as the boundary degree, the maximum value of the boundary change degree indicating the degree of change of the feature of the audio signal in the similar section. Changes in the characteristics of the audio signal often appear as the subject in the video file moves. That is, the voice processing apparatus according to the present embodiment can preferentially detect a scene in which the subject is moving by using the maximum value of the boundary change degree as the boundary degree.
- the speech processing apparatus assigns classification information in which scene change points are classified according to sound environment information to index information.
- the image viewing apparatus can provide various user interface functions to the user by using the classification information.
- the video viewing device may have a configuration in which the scene change point can be displayed on the progress bar in a form that can distinguish the classification for each classification (for example, color classification for each classification, changing the shape of a mark for each classification).
- the user can grasp the rough scene configuration of the AV content by looking at the progress bar, and can search the target scene more intuitively.
- the video viewing device may be configured to highlight on the progress bar the scene change points of the same classification as the scene change points of the currently viewed scene. This configuration allows the user to quickly skip to a scene similar to the one currently being viewed.
- the speech processing device calculates, as the boundary degree, the number of pieces of boundary information indicating the same boundary or the maximum value of the boundary change degree in the boundary information indicating the same boundary. It is not limited. For example, an accumulated value of boundary change degrees in boundary information indicating the same boundary may be used. According to this configuration, the speech processing device can calculate the boundary of the scene which is calculated from more unit intervals (second unit interval) and in which the change of the feature is large in the same scene with priority. Become.
- the boundary information calculation unit calculates both the start time and the end time of the similar section as the boundary information, but may calculate only the start time. Only the end time may be calculated. In the configuration in which the boundary information calculation unit calculates only the time of the start, there is no need to perform interval extension processing in the forward direction of the time axis, and in the configuration where only the time of the end is calculated, the interval extension process in the reverse direction of the time axis There is no need to do it.
- the threshold calculation unit calculates the threshold using Equation (3), but the calculation method of the threshold is not limited to this. For example, one obtained by changing the coefficient k of the equation (5) shown below between 0 and 3 may be used.
- the scene change point detected using the first threshold TH1 smaller than the second threshold is, for example, a larger scene such as the scene 21 and the scene 22 included in the scene 20 in FIG. It can be estimated as the border of a small scene included in.
- a scene change point detected using the second threshold TH2 larger than the first threshold is larger including a smaller scene, for example, a scene 20 including the scene 21 and the scene 22 in FIG. It can be estimated as a scene boundary.
- the threshold value calculation unit and the boundary determination unit function as a scene structure estimation unit that estimates the hierarchical structure of the scene in the audio signal.
- the boundary determination unit detects the time of the boundary degree exceeding the threshold input from the threshold calculation unit as a scene change point, the present invention is not limited to this.
- the boundary determination unit may detect, for example, a predetermined number N (N is a positive integer) of times in descending order of boundary degree as a scene change point.
- the boundary determination unit detects a predetermined number N (N is a positive integer) of times as the first scene change point in descending order of boundary degree, and the predetermined number M (M is an integer greater than N) in descending order of boundary degree Time may be detected as a second scene change point.
- the first scene change point can be estimated as the boundary of a large scene including a smaller scene, for example, a scene 20 including the scene 21 and the scene 22 in FIG.
- the second scene change point can be estimated as the boundary of a small scene included in a larger scene, such as the scene 21 and the scene 22 included in the scene 20 in FIG.
- the boundary determination unit functions as a scene structure estimation unit that estimates the hierarchical structure of the scene in the audio signal.
- the boundary information calculation unit may calculate similar intervals every N second unit intervals, where N is a positive integer.
- the boundary information calculation unit may obtain a plurality of second unit sections specified by the user from the interface device or the like, and calculate similar sections of the plurality of second unit sections specified by the user.
- the reference section used for the section extension reference index calculation processing performed by the section extension reference index calculating unit is nine sections in total in the vicinity of the second unit section to be processed, It is not something that can be done.
- the reference section may be, for example, a total of N sections (N is an integer of 2 or more) near the second unit section to be processed.
- the similar section calculated in the boundary information calculation unit is a relatively wide section.
- the scene change point detected in the index generation unit indicates the boundary of a large scene including a smaller scene, such as a scene 20 including the scene 21 and the scene 22 in FIG.
- the similar section calculated in the boundary information calculation unit becomes a relatively narrow section.
- the scene change point detected in the index generation unit indicates the boundary of a small scene included in a larger scene, such as scene 21 and scene 22 included in scene 20 in FIG. Be done.
- the section extension reference index calculation unit, the boundary information calculation unit, and the index generation unit first detect a scene change point when N is large, and then detect a scene change point when N is small. It is also good.
- the section expansion reference index calculation unit, the boundary information calculation unit, and the index generation unit can detect the boundary of a large scene in an audio signal and the boundary of a small scene included in the large scene. That is, according to this configuration, the section extension reference index calculation unit, the boundary information calculation unit, and the index generation unit function as a scene structure estimation unit that estimates the hierarchical structure of the scene in the audio signal.
- the reference vector is described as the centroid of the frequency vector of the second unit section included in the reference section, but the reference vector is not limited to this.
- it may be a vector whose component is the median of the components of the frequency vector of the second unit section included in the reference section, or when the number of second unit sections included in the reference section is as large as 100.
- a vector having the mode of the component of the frequency vector as the component may be used as the reference vector.
- the boundary information calculation unit is configured such that the Euclidean distance between the frequency vector in the target section and the reference vector S does not exceed the threshold Rth, and the length of the similar section is preset. If it is determined that the length is shorter than the upper limit le of the length of the similar section, the target section is included in the similar section. This is a measure to prevent the length of the similar section from becoming a certain length or more, but if the length of the similar section is not limited, the comparison process between the length of the similar section and the upper limit le is The target section may be included in the similar section without being performed.
- the upper limit le of the length of the similar section is a value set in advance, it is not limited to this.
- the upper limit le of the length of the similar section may use, for example, a value designated by the user via the interface.
- the order of the section extension processing may be performed before the time axis forward direction before the time axis reverse direction, or the similar section may be performed in the second unit interval unit in the time axis reverse direction and time axis It may be configured to extend alternately in the forward direction.
- extension is performed alternately, it is not necessary to alternate in units of second unit sections, and a method may be employed in which extension is performed alternately every fixed number (for example, five) of second unit sections.
- the boundary information calculation unit converts the target section into the similar section depending on whether the Euclidean distance between the reference vector and the frequency vector of the target section exceeds the threshold value Rth. It was judged whether to include in. However, it is not necessary to use the Euclidean distance if it can be determined whether the reference vector and the frequency vector are similar to each other by a certain amount or more.
- the Kullback-Leibler divergence in both directions of two mixture distributions may be configured to extract similar sections by using as a distance).
- the threshold value Rth may also be calculated using the Kullback-Leibler information amount.
- the Kullback-Leibler information amount is generally known as a measure to determine the difference between two probability distributions in probability theory and information theory, and the KL distance between the frequency vector and the reference vector according to an embodiment of the present invention is It can be calculated as follows.
- one mixed distribution is configured using the frequency vector NF and the probability distribution defined by each anchor model.
- the frequency vector NF ( ⁇ 1 ,..., ⁇ r ,..., ⁇ 1024 ) is a weight for the probability distribution (b A1 ,..., B Ar ,..., B A1024 ) defined by each anchor model.
- the mixed distribution G NF can be given by the following equation (6).
- the Kalbach-Leibler information amount from G NF to G S can be given by the following equation (8).
- X is a set of all arguments of the mixed distributions G NF and G S.
- Whether the target section is included in the similar section may be determined using the KL distance shown in the equation (10) instead of the Euclidean distance shown in the above embodiment.
- the threshold Rth is the frequency at which the KL distance between the plurality of frequency vectors and the reference vector is the largest among the frequency vectors of the plurality of second unit sections included in the reference section instead of the Euclidean distance.
- the KL distance between the vectors may be used.
- correlation operation calculation of cosine similarity or Pearson's correlation coefficient, etc.
- the target section may be included in the similar section.
- the frequency vector in the second unit section is a vector having the normalized cumulative likelihood of each component of the likelihood vector included in each section as a component
- the frequency vector is It may be any vector as long as it can indicate the feature of the audio signal in the section, and in particular, it can be a vector having other than the normalized cumulative likelihood as a component, as long as it can identify the component of the frequently appearing sound.
- each component of the likelihood vector included in the unit section is accumulated, and the accumulated likelihood corresponding to only the top k (a plurality of k, for example, 10) anchor models having high accumulated likelihood is normalized. It may be a vector.
- the frequency vector may be a vector whose component is the likelihood accumulated as it is without normalizing the accumulated value, or a vector whose component is the average value of the respective likelihoods.
- the present invention is not limited to this as long as it represents a feature in the first unit section of the audio signal.
- the frequency characteristic of an audio signal such as a power spectrum may be used, or the time series itself of the amplitude of the audio signal may be used as a feature amount.
- the MFCC 26 dimension is used, but this is because a suitable result is obtained by making it 26 dimensions by experiment, and the feature amount vector of the present invention is not limited to 26 dimensions. .
- anchor models Ar of each of a plurality of types of sound segments classified using clustering from sound data stored in advance in the sound data storage device are created (a so-called anchor model
- an anchor model creation method is not limited to this.
- the user selects sound data corresponding to each of the sound segments, applies a type label to each, and corresponds to sound from sound data having the same type label.
- An anchor model of a segment may be created (a so-called supervised anchor model Ar is created).
- the length of time of each of the first unit section and the second unit section in the above-described embodiment is an example.
- the time length may be a length other than that shown in the above-described embodiment as long as the first unit section ⁇ the second unit section. In this case, if the length of the second unit section is a multiple of the length of the first unit section, the process is simplified.
- the likelihood vector buffer, the frequency vector buffer, and the anchor model storage unit are configured by a part of the memory, but in the case of a storage device that can be read and written by the speech processing device It is not this limitation.
- a storage device that can be read and written by the speech processing device It is not this limitation.
- it may be a hard disk or a flexible disk, or may be an externally connected storage device.
- the anchor model creation device 107 may create a new anchor model.
- the audio processing apparatus has been described as a component of the video viewing apparatus, but it may be a component of the audio editing apparatus.
- the image display apparatus may be an image display apparatus that acquires a moving image file including an audio signal from an external device and outputs image data corresponding to a scene change point of a detection result as a thumbnail image.
- the moving image file is acquired from the recording medium, but the acquiring means is not limited to the method using the recording medium. For example, it may be acquired from wireless or wired broadcasting, from a network, etc. Even if the audio processing device is provided with an audio input device such as a microphone and the scene change point is detected from the audio signal input by the audio input device Good.
- the present invention can also be realized as a video viewing system including the audio processing device of the above embodiment connected to a network and the audio processing device and one or more terminals connected to the network.
- one terminal transmits a moving image file to an audio processing device, and the audio processing device detects a scene change point of the moving image file received and transmits the detected scene change point to the terminal.
- the function of the audio processing device may be divided, and the terminal may include a part of the divided functions.
- a terminal provided with part of the divided functions is referred to as a client, and an apparatus provided with the remaining functions is referred to as a server.
- 25 to 27 are diagrams showing an example of the configuration of a video viewing system in which the functions of the audio processing device are divided.
- the video viewing system comprises a client 2600 and a server 2700.
- the client 2600 includes a content storage device 102, an audio extraction device 103, an audio processing device 2602, and a transmission / reception device 2604.
- the content storage device 102 and the voice extraction device 103 are the same as the content storage device 102 and the voice extraction device 103 in the above embodiment.
- the voice processing device 2602 has a function of part of the voice processing device 104 in the above embodiment, specifically, a function of generating a frequency vector from an audio signal.
- the transmission / reception device 2604 has a function of transmitting the frequency vector generated by the voice processing device 2602 to the server 2700, and a function of receiving index information from the server 2700.
- the server 2700 includes an index storage unit 105, a voice processing unit 2702, and a transmission / reception unit 2704.
- the index storage unit 105 is similar to the index storage unit 105 in the above embodiment.
- the voice processing device 2702 has a function of part of the voice processing device 104 in the above embodiment, specifically, a function of generating index information from a frequency vector.
- the transmission / reception device 2704 has a function of receiving a frequency vector from the client 2600 and a function of transmitting index information stored in the index storage device 105 to the client 2600.
- FIG. 26 is a diagram showing an example of a functional configuration of the audio processing device 2602 included in the client 2600.
- the speech processing device 2602 includes a feature quantity vector generation unit 201, a likelihood vector generation unit 202, a likelihood vector buffer 203, a frequency vector generation unit 204, and an anchor model storage unit 209.
- Each component has the same function as that of the component with the same name and the same symbol in the above embodiment.
- FIG. 27 is a diagram showing an example of a functional configuration of the audio processing device 2702 provided in the server 2700.
- the speech processing device 2702 comprises a frequency vector buffer 205, a segment expansion reference index calculator 206, a boundary information calculator 207 and an index generator 208.
- Each component has the same function as that of the component with the same name and the same symbol in the above embodiment.
- the communication in the video viewing system is limited to the frequency vector and the index information. Therefore, the amount of communication can be reduced as compared with the case of transmitting a moving image file without dividing the audio processing device.
- the server receives from the client a thumbnail image or the like corresponding to the generated index information, and distributes the corresponding thumbnail image together with the generated index information to other terminals in the network. You may provide a function.
- the video viewing system can perform streaming distribution in which only scenes that are of interest to the user are extracted.
- (21) Machine language or high-level language for causing the processor of the speech processing apparatus and various circuits connected to the processor to execute the reference index calculation process, the boundary information calculation process, and the index generation process described in the above embodiment It is also possible to record a control program consisting of program codes on a recording medium or distribute it via various communication paths.
- Such recording media include an IC card, a hard disk, an optical disk, a flexible disk, a ROM, a flash memory and the like.
- the control program distributed and distributed is used by being stored in a memory or the like that can be read by a processor, and the processor executes the control program to realize each function as shown in each embodiment. Will be
- the processor may execute the control program directly, or may compile and execute or execute by the interpreter.
- Each functional component (feature unit section detection unit, section extension reference index calculation unit, interest section extraction unit, etc.) described in the above embodiment may be realized as a circuit that executes the function. It may be realized by executing a program by one or more processors. Further, the audio processing device of the above-described embodiment may be configured as a package of an IC, an LSI, or other integrated circuits. This package is incorporated into various devices and used, and thereby the various devices realize each function as shown in each embodiment.
- Each functional block such as the feature unit section detection unit, the section expansion reference index calculation unit, and the interest section extraction unit is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include some or all. Although an LSI is used here, it may be called an IC, a system LSI, a super LSI, or an ultra LSI depending on the degree of integration. Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible.
- a programmable field programmable gate array FPGA
- a reconfigurable processor that can reconfigure connection and setting of circuit cells in the LSI may be used.
- the speech processing apparatus calculates feature for each unit section, which represents a feature of a unit section of the input audio signal.
- a boundary calculation means for calculating boundary information on the boundary between another similar section and a similar section consisting of a plurality of similar unit sections including the corresponding unit section, and the priority of each of the boundaries indicated by the boundary information is calculated
- determining means for determining whether each of the boundaries is a scene change point based on the priority.
- the speech processing apparatus detects, as a scene change point, one selected from among boundaries of similar sections similar in section feature (feature amount vector, likelihood vector and frequency vector).
- the speech processing apparatus can detect a change point in a transient scene by setting the boundary of the similar section as a scene change point.
- the speech processing apparatus can easily search for a desired scene change point by indexing the selected boundary as a scene change point.
- the determination unit may calculate the number of pieces of boundary information indicating the same boundary as the priority.
- the audio processing device is configured to calculate the user's desired scene and another scene under the assumption that boundaries indicating the same time are calculated from unit sections belonging to the same scene desired by the user. It is possible to preferentially detect the change point of.
- the boundary calculation means further changes, as the boundary information, a degree of change of the feature of the audio signal between the similar section and another section.
- a degree may be calculated, and the determination means may calculate, as the priority, an accumulated value of the degree of change of boundary information indicating the same boundary.
- the audio processing device can preferentially detect both the boundary of the scene having a change in the feature of the audio signal and the boundary calculated from many unit sections.
- the boundary calculation means further changes, as the boundary information, a degree of change of the feature of the audio signal between the similar section and another section.
- a degree may be calculated, and the determination means may calculate, as the priority, the maximum value of the degree of change of boundary information indicating the same boundary.
- the audio processing device can preferentially detect the boundary of the scene having a change in the feature of the audio signal.
- the boundary calculation means further uses, as the boundary information, a section feature of the unit section included in the similar section, the sound environment of the similar section.
- the sound environment information may be calculated, and the scene change point may be classified using the sound environment information, and the classification unit may add classification information of a classification result to the scene change point.
- the video display apparatus or the like that uses the output of the audio processing apparatus can provide various user interface functions to the user based on the classification information.
- the voice processing apparatus further includes threshold calculation means for calculating a threshold based on the priority, and the determination means uses the time of the boundary of priority exceeding the threshold. It may be detected as the scene change point.
- the audio processing device can calculate the threshold adaptively for each audio signal. As a result, it becomes possible to detect scene change points with stable accuracy for all audio signals.
- the boundary calculation means may calculate, as the boundary information, the time of the beginning of the similar section.
- the boundary calculation unit may calculate the time of the end of the similar section as the boundary information.
- the speech processing apparatus when obtaining the similar section for each unit section, the speech processing apparatus only needs to obtain the boundary in either the time axis forward direction or the time axis reverse direction, and the amount of calculation is reduced.
- the unit section is a second unit section, and the second unit section includes a plurality of continuous first unit sections, and further, a plurality of types of sound elements.
- a model storage unit storing in advance a probability model representing the features of each piece; and a likelihood vector having the likelihood for each of the sound segments of the audio signal as a component using the probability model;
- Likelihood vector generation means for generating each unit interval, wherein the interval feature is generated using likelihood vectors of a plurality of first unit intervals included in the second unit interval, and each of the sound segments It may be a frequency vector indicating the appearance frequency of
- the speech processing device generates the likelihood vector and the frequency vector based on the probability model representing the sound segment to generate the sound in each first unit section and each second unit section of the audio signal.
- the degree to which the components of the segment are included can be expressed by the likelihood vector and the frequency vector.
- the speech processing apparatus further includes feature quantity vector generation means for calculating a feature quantity vector indicating a frequency characteristic in the first unit section of the audio signal, wherein the likelihood vector generation means May generate the likelihood vector using the feature quantity vector and the probability model.
- the voice processing apparatus can detect a scene change point using the frequency characteristic of the audio signal.
- the speech processing apparatus is characterized by: feature calculation means for calculating, for each unit section, a section feature representing a feature of a unit section of an input audio signal; The audio signal from among the boundaries indicated by the boundary information calculation means for calculating the boundary information on the boundary between the similar interval consisting of a plurality of similar unit intervals including the unit interval and the other interval, and the boundary indicated by the boundary information And a scene structure estimation unit that detects a boundary of a first scene included in a scene represented by the symbol and a boundary of a second scene included in the first scene.
- the speech processing apparatus estimates the hierarchical structure of scenes in the audio signal, so that the user can easily search for a desired scene based on the estimated hierarchical structure.
- a voice processing apparatus and a voice processing method detect a scene change point from an audio signal such as an AV content including a voice, a sound in the house, a sound at the time of going out, etc. It is possible to easily search for, or to perform emphasis reproduction (trick reproduction or filter processing) of a scene of interest, which is useful for an audio editing apparatus, a moving image editing apparatus, and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Stereophonic System (AREA)
Abstract
Description
AVコンテンツは、シーンを捉える粒度により様々な長さの区間から構成される。例えば、AVコンテンツが、パーティを撮影したものであり、図1(a)に示すシーンで構成されているとする。そして、図1(b)は、そのシーンにおけるオーディオ信号を表現しているとする。図に示すように、パーティには、乾杯のシーン10およびそれに続く食事のシーン20が存在し、食事のシーン20は、さらに、おもに食事をしているシーン21、および、おもに歓談をしているシーン22から構成される。シーン20は、おもに食事をしているシーン21からおもに歓談しているシーン22へ移り変わっていく過渡シーンである。
<実施の形態1>
<1-1.概要>
本発明の一の実施の形態に係る音声処理装置の概要を示す。
<1-2.データ>
ここで、本実施の形態に係る音声処理装置で使用するデータについて説明する。
<動画ファイル>
動画ファイルは、オーディオ信号X(t)と複数の画像データとから構成される。オーディオ信号X(t)は、振幅値の時系列データであり、図2(a)に示すような波形で表現することができる。図2(a)は、横軸に時間、縦軸に振幅をとった、オーディオ信号の波形例である。
<特徴量ベクトル>
特徴量ベクトルMは、オーディオ信号X(t)の特徴を表現するものであり、本実施の形態では、特徴量ベクトルMを示すものとして、第1単位区間毎に分割されたオーディオ信号のメル周波数ケプストラム係数MFCC(Mel-Frequency Cepstrum Coefficients)を用いる。ここで、第1単位区間は、オーディオ信号X(t)の時間軸における所定の長さ(例えば、10msec)の区間であり、図2(a)における時刻TnからTn+1までの区間である。
<アンカーモデル>
アンカーモデルAr(r=1,2,…,K)は、複数種類のサウンド素片となる音声データから生成される特徴量ベクトルを用いて作成される確率モデルであり、各種サウンド素片それぞれの特徴を表現するものである。即ち、アンカーモデルは各種サウンド素片に対応して作成される。本実施の形態では、混合正規分布モデル(GMM, Gaussian Mixture Model)を採用し、各アンカーモデルArは、正規分布を規定するパラメータで構成される。
<尤度ベクトル>
尤度ベクトルFは、上記アンカーモデルArに基づいて算出されるオーディオ信号X(t)の複数種類のサウンド素片それぞれに対する尤度Lrを成分とするベクトルである。
頻度ベクトルNFは、オーディオ信号の第2単位区間毎の特徴を表現するベクトルであり、特に、オーディオ信号の第2単位区間に係る各サウンド素片の出現頻度を表現するベクトルである。ここで、第2単位区間は、オーディオ信号X(t)の時間軸における所定の長さ(例えば、1sec)の区間であり、図6に示すように、複数個の連続する第1単位区間の長さに相当する。
境界情報は、オーディオ信号の第2単位区間毎に計算される当該第2単位区間と頻度ベクトルが類似する類似区間の境界に関する情報である。本実施の形態の音声処理装置は、境界情報として、類似区間の始端の時刻および類似区間の終端の時刻を計算する。図8は、本実施の形態で計算する境界情報の一例である。例えば、0番目の第2単位区間(時刻0~1sの間の区間)からは、始端時刻として0s、終端時刻として10sを示す境界情報が算出されたことを示している。
上述したように、境界度は、同一の時刻を示す境界情報の数である。例えば、図8において、始端時刻または終端時刻で0sを示す境界情報は、0番目の第2単位区間(時刻0~1sの間の区間)、1番目の第2単位区間(時刻1~2sの間の区間)、および、2番目の第2単位区間(時刻2~3sの間の区間)から計算される境界情報の3つであるから、時刻0sの境界度は3となる。図9は、横軸に時刻、縦軸に計算された境界度を示したグラフの一例である。
以下、本実施の形態に係る音声処理装置104を備える映像視聴装置100の機能構成について説明する。
図10は、映像視聴装置100の機能構成例を示すブロック図である。図に示すように、映像視聴装置100は、入力装置101、コンテンツ記憶装置102、音声抽出装置103、音声処理装置104、インデクス記憶装置105、出力装置106、アンカーモデル作成装置107、音声データ蓄積装置108およびインタフェース装置109を備える。
入力装置101は、ディスクドライブ装置などで実現され、記録媒体120が装着されると、記録媒体120から動画ファイルを読み込んで、取得した動画ファイルをコンテンツ記憶装置102に格納する。なお、記録媒体120は、各種データを記憶する機能を有する媒体であり、例えば、光ディスクや、フレキシブルディスク、SDカード、フラッシュメモリなどである。
コンテンツ記憶装置102は、例えば、ハードディスク装置などで実現され、入力装置101が記録媒体120から取得した動画ファイルを格納している。動画ファイルは、動画ファイルそれぞれに固有のIDが付されて格納される。
音声抽出装置103は、コンテンツ記憶装置102に格納されている動画ファイルからオーディオ信号を抽出し、抽出したオーディオ信号を音声処理装置104に入力する。なお、音声抽出装置103は、符号化されているオーディオ信号に対して復号化処理を行って、図2(a)に示すようなオーディオ信号X(t)を生成する。なお、音声抽出装置103は、例えば、プログラムを実行するプロセッサ等により実現される。
音声処理装置104は、音声抽出装置103から入力されるオーディオ信号X(t)に基づいてシーン変化点の検出を行い、検出したシーン変化点からなるインデクス情報をインデクス記憶装置105に格納する。音声処理装置104の機能構成の詳細については後述する。
インデクス記憶装置105は、例えば、ハードディスク装置などから実現され、音声処理装置104から入力されるインデクス情報を記憶する。インデクス情報は、動画ファイルのIDおよび動画ファイルにおける時刻(シーン変化点の時刻)からなる。
出力装置106は、インデクス記憶装置105からインデクス情報を取得し、取得したインデクス情報に対応する映像データ(コンテンツ記憶装置102に格納されている動画ファイルの一部)を表示装置130に出力する。出力装置106は、表示装置130に出力される映像データに、例えば、インデクス情報に対応する時刻に目印を付けたプログレスバーなどのユーザインタフェース(UI:User Interface)に関する情報を付与し、インタフェース装置109に入力されるユーザからの操作に応じて、スキップなどの再生制御を行う。
アンカーモデル作成装置107は、音声データ蓄積装置108に記憶されているオーディオ信号に基づいてアンカーモデルArを作成し、作成したアンカーモデルArを音声処理装置104に出力する。アンカーモデル作成装置107の機能構成の詳細については後述する。
音声データ蓄積装置108は、例えば、ハードディスク装置などで実現され、アンカーモデル作成装置107がアンカーモデルArを作成するときに用いられる音声データをあらかじめ記憶している。
インタフェース装置109は、キーボードなどの操作部(図示せず)を備え、ユーザからの入力操作を受け付けて、プログレスバーの操作情報などを出力装置106に、作成すべきアンカーモデルの個数Kをアンカーモデル作成装置107に通知する。
音声処理装置104は、メモリ(図示せず)およびプロセッサ(図示せず)から構成され、プロセッサがメモリに書き込まれたプログラムを実行することにより、図11に示す構成を実現している。
特徴量ベクトル生成部201は、音声抽出装置103から入力されたオーディオ信号X(t)に基づいて、第1単位区間毎に、特徴量ベクトルMを生成する。
アンカーモデル蓄積部209は、メモリの一部の領域により構成され、アンカーモデル作成装置107により作成されたアンカーモデルArを記憶している。本実施の形態では、アンカーモデル蓄積部209は、音声処理装置104の各処理の実行前に、あらかじめアンカーモデルArを記憶している。
尤度ベクトル生成部202は、特徴量ベクトル生成部201により生成される特徴量ベクトルMおよびアンカーモデル蓄積部209に蓄積されているアンカーモデルArを用いて、オーディオ信号X(t)の、サウンド素片それぞれに対する尤度Lrを算出し、各尤度を成分とする尤度ベクトルFを第1単位区間毎に生成する。
尤度ベクトルバッファ203は、メモリの一部の領域により構成され、尤度ベクトル生成部202により生成される尤度ベクトルFを記憶する。
頻度ベクトル生成部204は、尤度ベクトルバッファ203で記憶されている尤度ベクトルFに基づいて、第2単位区間毎に頻度ベクトルNFを生成する。
頻度ベクトルバッファ205は、メモリの一部の領域により構成され、頻度ベクトル生成部204により生成される頻度ベクトルNFを記憶する。
区間伸長基準指標計算部206は、第2単位区間それぞれに対して、後述する区間伸長処理の基準指標となる基準区間、基準ベクトルS及びしきい値Rthを算出する。
図11に戻り、境界情報計算部207は、第2単位区間に対して、頻度ベクトルの類似する区間からなる類似区間を算出し、類似区間の始端の時刻および終端の時刻を特定する。境界情報計算部207は、頻度ベクトルバッファ205に記憶されている頻度ベクトルNF、処理対象の第2単位区間、および、区間伸長基準指標計算部206により算出された基準指標(基準区間、基準ベクトルSとしきい値Rth)を入力とし、特定した始端の時刻および終端の時刻を境界情報としてインデクス生成部208へ出力する。
インデクス生成部208は、境界情報計算部207で算出された境界情報に基づいて、シーン変化点を検出し、検出したシーン変化点をインデクス化したインデクス情報をインデクス記憶装置105に出力する。図15は、インデクス生成部208の機能構成例を示すブロック図である。図15に示すように、インデクス生成部208は、投票部301、閾値計算部302および境界判定部303を備える。以下、各構成について説明する。
投票部301は、境界情報計算部207で計算された境界情報の示す時刻ごとに、同一時刻を示す境界情報の数を境界度として算出する。投票部301は、境界情報計算部207から入力される境界情報それぞれに対して、入力された境界情報の示す時刻に投票(時刻iに対応する境界度KKiに1を加算)することにより、境界度を算出する。なお、投票部301は、1つの境界情報で指定される始端の時刻および終端の時刻の両方に投票を行う。
閾値計算部302は、投票部301により算出された時刻ごとの境界度の平均μおよび標準偏差σを用いて、閾値Thを算出する。境界情報の示す時刻がTi(i=1,2,3,…,N)、それに対応する境界度がKKi(i=1,2,3,…,N)であった場合、平均値μ、標準偏差σおよび閾値THは、以下の式(1)、(2)および(3)で与えられる。
境界判定部303は、投票部301で算出された時刻ごとの境界度KKiおよび閾値計算部302で算出された閾値THを用いて、以下の式(4)の条件を満たす時刻をシーン変化点と判定し、インデクス情報としてインデクス記憶装置105に出力する。
アンカーモデル作成装置107は、メモリ(図示せず)およびプロセッサ(図示せず)から構成され、プロセッサがメモリに書き込まれたプログラムを実行することにより、図16に示す構成を実現している。
特徴量ベクトル生成部401は、音声データ蓄積装置108に記憶されている音声データに基づいて、第1単位区間毎に、特徴量ベクトルMを生成する。
特徴量ベクトル分類部402は、特徴量ベクトル生成部401により生成された特徴量ベクトルを、クラスタリング(分類)する。
アンカーモデル生成部403は、特徴量ベクトル分類部402により分類されたK個のクラスタそれぞれの平均および分散を算出し、アンカーモデルAr(r=1,2,…,K)としてアンカーモデル蓄積部209に記憶する。
以下、本実施の形態に係る音声処理装置104の動作について図面を参照しながら説明する。
図17は、音声処理装置104の動作を示すフローチャートである。図に示すように、音声処理装置はオーディオ信号を入力とする(ステップS1701)。
図18は、図17におけるステップS1703の区間伸長基準指標計算部206で実行される基準指標計算処理の詳細を示すフローチャートである。図18に示すように、基準指標計算処理において、区間伸長基準指標計算部206は、処理対象の第2単位区間および頻度ベクトルバッファ205で保持する頻度ベクトルを入力とする(ステップS1801)。
図19は、図17におけるステップS1704の境界情報計算部207で実行される境界情報計算処理の詳細を示すフローチャートである。図19に示すように、境界情報計算処理において、境界情報計算部207は、処理対象の第2単位区間、区間伸長基準指標計算部206で算出される基準指標、あらかじめ設定される類似区間の長さの上限および頻度ベクトルバッファ205で保持する頻度ベクトルを入力とする(ステップS1901)。
図20は、図17におけるステップS1705のインデクス生成部208で実行されるインデクス生成処理の動作を示すフローチャートである。図20に示すように、インデクス生成処理において、インデクス生成部208は、境界情報計算部207で計算される境界情報を入力とする(ステップS2001)。
本実施の形態の音声処理装置は、オーディオ信号の所定時間長の単位区間ごとに、オーディオ信号の単位区間における特徴を示す区間特徴(特徴量ベクトル、尤度ベクトルおよび頻度ベクトル)を算出し、算出した区間特徴の類似する区間からなる類似区間の境界からシーン変化点を検出する。
実施の形態2では、実施の形態1と比較して、2つの点で異なっている。
図21は、実施の形態2に係る音声処理装置110の機能構成例を示すブロック図である。音声処理装置110は、実施の形態1の音声処理装置104と比較して、境界情報計算部207の代わりに境界情報計算部211を備え、インデクス生成部208の代わりにインデクス生成部212を備える。
境界情報計算部211は、境界情報計算部207の機能に加え、境界情報として、さらに、処理対象の第2単位区間近傍におけるオーディオ信号の特徴と、類似区間の境界におけるオーディオ信号の特徴の変化の度合いを示す境界変化度、および、類似区間における代表的な音環境を示す音環境情報を計算する機能を有する。
同様に、境界情報計算部211は、終端変化度Dout(類似区間の始端における境界変化度)として、基準ベクトルSと類似区間直後の第2単位区間の頻度ベクトルNFとのユークリッド距離を用いる。
図23は、インデクス生成部212の機能構成例を示すブロック図である。インデクス生成部212は、実施の形態1のインデクス生成部208と比較して、投票部301の代わりに境界度計算部311を備え、さらに、音環境分類部312が、境界判定部303とインデクス記憶装置105の間に挿入される。
境界度計算部311は、境界情報計算部211で計算された境界情報の示す時刻ごとに、境界情報の境界変化度の最大値を境界度として算出する。より具体的には、境界度計算部311は、時刻Tiの境界度として、始端の時刻がTiの境界情報における始端変化度および終端の時刻がTiの境界情報における終端変化度の最大値を算出する。
音環境分類部312は、境界判定部303で、シーン変化点と判定された時刻に対応付けられた音環境情報を、例えば、K-means法を用いて複数のグループ(例えば、3グループ)に分類する。音環境分類部312は、分類結果の分類情報をインデクス情報に付与し、分類情報付のインデクス情報をインデクス記憶装置105に出力する。図24に、分類情報の付与されたインデクス情報の具体例を示す。
本実施の形態の音声処理装置は、境界度として、類似区間内におけるオーディオ信号の特徴の変化の度合いを示す境界変化度の最大値を用いている。オーディオ信号の特徴の変化は、動画ファイルにおける被写体に動きに伴って出現することが多い。つまり、本実施の形態の音声処理装置は、境界度として境界変化度の最大値を用いることで、被写体に動きあるシーンを優先して検出することが可能である。
上記実施の形態に従って、本発明に係る音声処理装置について説明してきたが、本発明はこれに限られるものではない。以下、本発明の思想として含まれる各種変形例について説明する。
<6-2.補足>
以下、更に本発明の一実施形態としての音声処理装置の構成及びその変形例と効果について説明する。
101 入力装置
102 コンテンツ記憶装置
103 音声抽出装置
104 音声処理装置
105 インデクス記憶装置
106 出力装置
107 アンカーモデル作成装置
108 音声データ蓄積装置
109 インタフェース装置
201 特徴量ベクトル生成部
202 尤度ベクトル生成部
203 尤度ベクトルバッファ
204 頻度ベクトル生成部
205 頻度ベクトルバッファ
206 区間伸長基準指標計算部
207,211 境界情報計算部
208,212 インデクス生成部
209 アンカーモデル蓄積部
301 投票部
302 閾値計算部
303 境界判定部
311 境界度計算部
312 音環境分類部
401 特徴量ベクトル生成部
402 特徴量ベクトル分類部
403 アンカーモデル生成部
Claims (17)
- 入力されたオーディオ信号の単位区間の特徴を表現する区間特徴を単位区間ごとに計算する特徴計算手段と、
複数の単位区間それぞれに対して、当該単位区間を含む区間特徴の類似する複数の単位区間からなる類似区間と他の区間との境界に関する境界情報を計算する境界計算手段と、
前記境界情報で示される境界それぞれの優先度を計算し、前記優先度に基づいて、前記境界それぞれがシーン変化点であるか否かを判定する判定手段と
を備えることを特徴とする音声処理装置。 - 前記判定手段は、前記優先度として、同一の境界を示す境界情報の数を計算する
ことを特徴とする請求項1記載の音声処理装置。 - 前記境界計算手段は、前記境界情報として、さらに、前記類似区間と他の区間との間のオーディオ信号の特徴の変化の度合いを示す変化度を計算し、
前記判定手段は、前記優先度として、同一の境界を示す境界情報の前記変化度の累積値を計算する
ことを特徴とする請求項1記載の音声処理装置。 - 前記境界計算手段は、前記境界情報として、さらに、前記類似区間と他の区間との間のオーディオ信号の特徴の変化の度合いを示す変化度を計算し、
前記判定手段は、前記優先度として、同一の境界を示す境界情報の前記変化度の最大値を計算する
ことを特徴とする請求項1記載の音声処理装置。 - 前記境界計算手段は、前記境界情報として、さらに、前記類似区間に含まれる前記単位区間の区間特徴を用いて、当該類似区間の音環境を表現する音環境情報を計算し、
さらに、前記音環境情報を用いて、前記シーン変化点を分類し、分類結果の分類情報を前記シーン変化点に付与する分類手段
を備えることを特徴とする請求項4記載の音声処理装置。 - さらに、前記優先度に基づいて、閾値を計算する閾値計算手段を備え、
前記判定手段は、前記閾値を超える優先度の境界の時刻を前記シーン変化点として検出する
ことを特徴とする請求項1に記載の音声処理装置。 - 前記判定手段は、前記優先度の高い順に、所定数の前記境界を前記シーン変化点として検出する
ことを特徴とする請求項1に記載の音声処理装置。 - 前記判定手段は、前記所定数を前記オーディオ信号の長さに応じて決定する
ことを特徴とする請求項7に記載の音声処理装置。 - 前記境界計算手段は、前記境界情報として、前記類似区間の始端の時刻を計算する
ことを特徴とする請求項1記載の音声処理装置。 - 前記境界計算手段は、前記境界情報として、前記類似区間の終端の時刻を計算する
ことを特徴とする請求項1記載の音声処理装置。 - 前記境界計算手段は、前記オーディオ信号の全単位区間に対して、前記境界情報を計算する
ことを特徴とする請求項1記載の音声処理装置。 - 前記単位区間を第2単位区間とし、前記第2単位区間は、複数の連続する第1単位区間からなり、
さらに、複数種類のサウンド素片それぞれの特徴を表現する確率モデルをあらかじめ記憶しているモデル記憶手段と、
前記確率モデルを用いて、前記オーディオ信号の前記サウンド素片それぞれに対する尤度を成分とする尤度ベクトルを、前記第1単位区間ごとに生成する尤度ベクトル生成手段とを備え、
前記区間特徴は、前記第2単位区間に含まれる複数の第1単位区間の尤度ベクトルを用いて生成され、前記サウンド素片それぞれの出現頻度を示す頻度ベクトルである
ことを特徴とする請求項1記載の音声処理装置。 - さらに、オーディオ信号の前記第1単位区間における周波数特性を示す特徴量ベクトルを算出する特徴量ベクトル生成手段を備え、
前記尤度ベクトル生成手段は、前記特徴量ベクトルおよび前記確率モデルを用いて前記尤度ベクトルを生成する
ことを特徴とする請求項12記載の音声処理装置。 - 入力されたオーディオ信号の単位区間の特徴を表現する区間特徴を単位区間ごとに計算する特徴計算手段と、
複数の単位区間それぞれに対して、当該単位区間を含む区間特徴の類似する複数の単位区間からなる類似区間と他の区間との境界に関する境界情報を計算する境界計算手段と、
前記境界情報の示す境界の中から、前記オーディオ信号によって表現されるシーンに含まれる第1のシーンの境界と、当該第1のシーンに含まれる第2のシーンの境界とを検出するシーン構造推定手段と
を備えることを特徴とする音声処理装置。 - 入力されたオーディオ信号の単位区間の特徴を表現する区間特徴を単位区間ごとに計算する特徴計算ステップと、
複数の単位区間それぞれに対して、当該単位区間を含む区間特徴の類似する複数の単位区間からなる類似区間と他の区間との境界に関する境界情報を計算する境界計算ステップと、
前記境界情報で示される境界それぞれの優先度を計算し、前記優先度に基づいて、前記境界それぞれがシーン変化点であるか否かを判定する判定ステップと
を含むことを特徴とする音声処理方法。 - コンピュータ読取可能な非一時的な記録媒体に記録され、オーディオ信号からシーン変化点を検出するシーン変化点検出処理を実行するプログラムであって、
前記シーン変化点検出処理は、
入力されたオーディオ信号の単位区間の特徴を表現する区間特徴を単位区間ごとに計算する特徴計算ステップと、
複数の単位区間それぞれに対して、当該単位区間を含む区間特徴の類似する複数の単位区間からなる類似区間と他の区間との境界に関する境界情報を計算する境界計算ステップと、
前記境界情報で示される境界それぞれの優先度を計算し、前記優先度に基づいて、前記境界それぞれがシーン変化点であるか否かを判定する判定ステップと
を含むことを特徴とするプログラム。 - 入力されたオーディオ信号の単位区間の特徴を表現する区間特徴を単位区間ごとに計算する特徴計算部と、
複数の単位区間それぞれに対して、当該単位区間を含む区間特徴の類似する複数の単位区間からなる類似区間と他の区間との境界に関する境界情報を計算する境界計算部と、
前記境界情報で示される境界それぞれの優先度を計算し、前記優先度に基づいて、前記境界それぞれがシーン変化点であるか否かを判定する判定部と
を備えることを特徴とする集積回路。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201380001357.5A CN103534755B (zh) | 2012-04-20 | 2013-03-11 | 声音处理装置、声音处理方法、程序及集成电路 |
JP2013548126A JP6039577B2 (ja) | 2012-04-20 | 2013-03-11 | 音声処理装置、音声処理方法、プログラムおよび集積回路 |
US14/113,481 US8930190B2 (en) | 2012-04-20 | 2013-03-11 | Audio processing device, audio processing method, program and integrated circuit |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012-096332 | 2012-04-20 | ||
JP2012096332 | 2012-04-20 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2013157190A1 true WO2013157190A1 (ja) | 2013-10-24 |
Family
ID=49383165
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2013/001568 WO2013157190A1 (ja) | 2012-04-20 | 2013-03-11 | 音声処理装置、音声処理方法、プログラムおよび集積回路 |
Country Status (4)
Country | Link |
---|---|
US (1) | US8930190B2 (ja) |
JP (1) | JP6039577B2 (ja) |
CN (1) | CN103534755B (ja) |
WO (1) | WO2013157190A1 (ja) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6003972B2 (ja) * | 2014-12-22 | 2016-10-05 | カシオ計算機株式会社 | 音声検索装置、音声検索方法及びプログラム |
CN104469487B (zh) * | 2014-12-31 | 2019-02-12 | 优酷网络技术(北京)有限公司 | 一种场景切换点的检测方法及装置 |
US10922551B2 (en) | 2017-10-06 | 2021-02-16 | The Nielsen Company (Us), Llc | Scene frame matching for automatic content recognition |
CN109981970B (zh) * | 2017-12-28 | 2021-07-27 | 深圳市优必选科技有限公司 | 一种确定拍摄场景的方法、装置和机器人 |
ES2901638T3 (es) * | 2018-05-17 | 2022-03-23 | Fraunhofer Ges Forschung | Dispositivo y método para detectar concordancias parciales entre una primera señal variable en el tiempo y una segunda señal variable en el tiempo |
US11475887B2 (en) * | 2018-10-29 | 2022-10-18 | Spotify Ab | Systems and methods for aligning lyrics using a neural network |
CN110136735B (zh) * | 2019-05-13 | 2021-09-28 | 腾讯音乐娱乐科技(深圳)有限公司 | 一种音频修复方法、设备及可读存储介质 |
EP3944100A1 (en) * | 2020-07-20 | 2022-01-26 | Mimi Hearing Technologies GmbH | Method of selecting a suitable content for subjective preference judgement |
US11144764B1 (en) * | 2020-09-30 | 2021-10-12 | Cbs Interactive Inc. | Systems, methods, and storage media for selecting video portions for a video synopsis of streaming video content |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001147697A (ja) * | 1999-11-19 | 2001-05-29 | Matsushita Electric Ind Co Ltd | 音響データ分析方法及びその装置 |
WO2008143345A1 (ja) * | 2007-05-23 | 2008-11-27 | Nec Corporation | コンテンツ分割位置判定装置、コンテンツ視聴制御装置及びプログラム |
WO2011033597A1 (ja) * | 2009-09-19 | 2011-03-24 | 株式会社 東芝 | 信号分類装置 |
WO2011145249A1 (ja) * | 2010-05-17 | 2011-11-24 | パナソニック株式会社 | 音声分類装置、方法、プログラム及び集積回路 |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0520367A (ja) | 1991-07-15 | 1993-01-29 | Hitachi Ltd | インデツクス作成方式 |
EP1073272B1 (en) * | 1999-02-15 | 2011-09-07 | Sony Corporation | Signal processing method and video/audio processing device |
JP2004056739A (ja) | 2002-07-24 | 2004-02-19 | Fuji Photo Film Co Ltd | ディジタルカメラ |
DE60320414T2 (de) * | 2003-11-12 | 2009-05-20 | Sony Deutschland Gmbh | Vorrichtung und Verfahren zur automatischen Extraktion von wichtigen Ereignissen in Audiosignalen |
US8478587B2 (en) * | 2007-03-16 | 2013-07-02 | Panasonic Corporation | Voice analysis device, voice analysis method, voice analysis program, and system integration circuit |
JP5060224B2 (ja) * | 2007-09-12 | 2012-10-31 | 株式会社東芝 | 信号処理装置及びその方法 |
-
2013
- 2013-03-11 WO PCT/JP2013/001568 patent/WO2013157190A1/ja active Application Filing
- 2013-03-11 US US14/113,481 patent/US8930190B2/en active Active
- 2013-03-11 JP JP2013548126A patent/JP6039577B2/ja active Active
- 2013-03-11 CN CN201380001357.5A patent/CN103534755B/zh active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001147697A (ja) * | 1999-11-19 | 2001-05-29 | Matsushita Electric Ind Co Ltd | 音響データ分析方法及びその装置 |
WO2008143345A1 (ja) * | 2007-05-23 | 2008-11-27 | Nec Corporation | コンテンツ分割位置判定装置、コンテンツ視聴制御装置及びプログラム |
WO2011033597A1 (ja) * | 2009-09-19 | 2011-03-24 | 株式会社 東芝 | 信号分類装置 |
WO2011145249A1 (ja) * | 2010-05-17 | 2011-11-24 | パナソニック株式会社 | 音声分類装置、方法、プログラム及び集積回路 |
Also Published As
Publication number | Publication date |
---|---|
CN103534755B (zh) | 2017-03-01 |
US8930190B2 (en) | 2015-01-06 |
US20140043543A1 (en) | 2014-02-13 |
CN103534755A (zh) | 2014-01-22 |
JP6039577B2 (ja) | 2016-12-07 |
JPWO2013157190A1 (ja) | 2015-12-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2013157190A1 (ja) | 音声処理装置、音声処理方法、プログラムおよび集積回路 | |
US8948515B2 (en) | Method and system for classifying one or more images | |
US10679063B2 (en) | Recognizing salient video events through learning-based multimodal analysis of visual features and audio-based analytics | |
US9436875B2 (en) | Method and apparatus for semantic extraction and video remix creation | |
US10134440B2 (en) | Video summarization using audio and visual cues | |
US8583647B2 (en) | Data processing device for automatically classifying a plurality of images into predetermined categories | |
US9189137B2 (en) | Method and system for browsing, searching and sharing of personal video by a non-parametric approach | |
US8867891B2 (en) | Video concept classification using audio-visual grouplets | |
US9143742B1 (en) | Automated aggregation of related media content | |
US20140245463A1 (en) | System and method for accessing multimedia content | |
US8892497B2 (en) | Audio classification by comparison of feature sections and integrated features to known references | |
US20130089304A1 (en) | Video concept classification using video similarity scores | |
JP6945639B2 (ja) | 画像処理方法及び装置 | |
Hoover et al. | Putting a face to the voice: Fusing audio and visual signals across a video to determine speakers | |
JP2011215963A (ja) | 電子機器、画像処理方法及びプログラム | |
JP5723446B2 (ja) | 興味区間特定装置、興味区間特定方法、興味区間特定プログラム、及び、興味区間特定集積回路 | |
Moreira et al. | Multimodal data fusion for sensitive scene localization | |
JP5620474B2 (ja) | アンカーモデル適応装置、集積回路、AV(AudioVideo)デバイス、オンライン自己適応方法、およびそのプログラム | |
JP5658285B2 (ja) | 興味区間抽出装置、興味区間抽出方法 | |
JP5908924B2 (ja) | 音声処理装置、方法、プログラムおよび集積回路 | |
KR101804679B1 (ko) | 스토리에 기초하는 멀티미디어 콘텐츠 개발 장치 및 방법 | |
Hoover et al. | Using audio-visual information to understand speaker activity: Tracking active speakers on and off screen | |
CN115810209A (zh) | 一种基于多模态特征融合网络的说话人识别方法和装置 | |
Cricri et al. | Multi-sensor fusion for sport genre classification of user generated mobile videos | |
JP5254900B2 (ja) | 映像再構成方法、映像再構成装置および映像再構成プログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 2013548126 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14113481 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13778914 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 13778914 Country of ref document: EP Kind code of ref document: A1 |