CN102708907B - Linear multimedia positioning method based on picture segmentation and synchronous labeling - Google Patents

Abstract

The invention relates to a linear multimedia positioning method based on picture segmentation and synchronous labeling, which comprises the following steps of: quickly determining the play order based on sections by using a photocopies picture; performing real-time linear positioning based on synchronous labeling on the picture by multimedia; realizing multimedia skipping by clicking the picture; and storing and reading linear multimedia positioning information. The invention provides a method for quickly determining the multimedia play order on the picture and a method using synchronous labeling and linear interpolation, the position of multimedia on the picture can be obtained at any time, and the multimedia positioning information is stored and is used for later repeated loading.

Description

A kind of linear multimedia localization method based on picture segmentation and sync tag

Technical field

The present invention relates to a kind of method determining multimedia (audio frequency, video) playing sequence on picture (music score, numbered musical notation) fast.

Background technology

The position indicating multimedia to user conveniently, lyrics technology is a solution of present stage maturation.For the lrc form lyrics, lrc file be the time editing that the lyrics occurred by song lyrics by editing machine to together, then synchronously successively the lyrics are shown when played songs.Standard format is: [minute: second. millisecond] lyrics.Software such as the lrc lyrics editing device 2010 of some specialties can the assist edit lyrics.

Although lyrics technology comparative maturity, there is the weakness that editor is inconvenient, instruction content is single.Be subject to the restriction in lyrics source in addition.

This external Apple has a software being ianalyse can editing electronic music score under Mac OS operating system, the position displayed the play on music score while can realizing playing music.But this software exists weakness with strong points, namely each multimedia needs music speciality personage to make music score for it, could realize the position that music score displays the play.In addition this software is not easy to find rhythmic stress; Only support staff, do not support numbered musical notation; Namely can not put and namely broadcast; And this software application method is complicated, layman is difficult to use.

Summary of the invention

The technical problem to be solved in the present invention is: when play multimedia (audio frequency, video), the real-time position of playing corresponding in the upper display of picture (music score, numbered musical notation), and when clicking picture, multimedia is jumped to relevant position.

The technical solution used in the present invention is: a kind of linear multimedia localization method based on picture segmentation and sync tag, picture (music score, numbered musical notation) is determined multimedia (audio frequency, video) playing sequence fast, and use the way of sync tag and linear interpolation, obtain the position of any time multimedia on picture, and achieve and multimedia locating information is preserved, for being repeatedly loaded into use later, that includes following four steps:

(1) use photocopy picture and determine playing sequence fast based on trifle; Use multiple photocopy pictures, fast music score is divided into page, row, trifle; Every page of multirow, often capable many trifles, and determine position and the size of page, row, trifle, and a little redirect mark is set, obtain next trifle that trifle is play each time, have thus determined playing sequence;

(2) multimedia carries out real time linear location based on sync tag on picture; Based on the playing sequence determined in (1), the time of each trifle of song is fixing, according to the way of linear interpolation, calculates the position of multimedia any time on music score, in music score, add sync tag simultaneously, accurate adjustment is carried out to the inaccurate place of linear interpolation;

(3) multimedia redirect is realized by clicking picture; Add sync tag in step (2) after, user clicks i-th page, and j is capable, during the side-play amount x position of k trifle, calculates multimedia time t, and realizes multimedia redirect;

(4) preservation of linear multimedia locating information and reading; The information of page, row, trifle, redirect, sync tag is preserved, and repeatedly can be loaded into use.

Based on trifle, described utilization photocopy picture also determines that playing sequence comprises the position and size determining page, row, trifle fast fast, arrange redirect, finally calculate next trifle when any trifle is play each time.

Multimedia is corresponding multipage picture (music score), and a pictures has one to multiple row, has one or more trifle in a line.First each pictures is sequenced in order, each pictures is corresponding one page.Then by the place having music score to mark in the every one page mode rectangle marked pulled, as shown in Fig. 1 (a).Rectangle is corresponding multiple row, has multiple trifle in every a line.Devise in the present invention and a kind ofly determine row, the position of trifle and the method for size fast: for each rectangle, input line number and columns, as shown in Fig. 1 (b).Result roughly can be generated, as shown in Fig. 1 (c).Eventually pass trickle adjustment, final result can be generated, as shown in Figure 1 (d) shows.Trickle adjustment operation comprises: the position of adjustment row, size, and the position of adjustment bar line, inserts and delete trifle etc.These all realize fast by mouse drag operation.

Suppose that a multimedia has NP page, one page has multirow, and a line has multiple trifle.P _i(0≤i≤NP-1) is i-th page, NR(P _i) be the line number of i-th page, R _{i, j}(0≤i≤NP-1,0≤j≤NR(P _i)-1) represent the jth row of i-th page.Similar NB(R _{i, j}) be the little joint number of the jth row of i-th page, B _{i, j, k}(0≤i≤NP-1,0≤j≤NR(P _i)-1,0≤k≤NB(R _i, _j)-1) represent i-th page, j is capable, a kth trifle.For arbitrary trifle B _{i, j, k}, define its from left to right next trifle NextPhBar(B physically from top to bottom _{i, j, k}) as follows

$\mathrm{NextPhBar}\left(B_{i,j,k}\right)=\left\{\begin{array}{c}{B}_{i,j,k+1}(k\≠\mathrm{NB}\left(R_{i,j}\right)-1)\\ B_{i,j+\mathrm{1,0}}(k=\mathrm{NB}\left(R_{i,j}\right)-1,j\≠\mathrm{NR}\left(P_t\right)-1)\\ B_{i+\mathrm{1,0,0}}(k=\mathrm{NB}\left(R_{i,j}\right)-1,j=\mathrm{NR}\left(P_i\right)-1)\end{array}\right.---\left(1\right)$

When playing, in most of the cases from left to right next trifle physically i.e. next trifle in logic from top to bottom.Repeatedly mark running into, during redirect symbol, can arrange redirect, a redirect have recorded for a trifle B _{i, j, k}, to time herein, next trifle is BarJump(B to play (counting from 0, lower same) for the w time _{i, j, k}, w).

According to physical location and redirect, next logic trifle NextLgBar(B that can play for the w time in the hope of arbitrarily small joint _{i, j, k}, w), and be the NextBTime(B of next trifle _{i, j, k}, w) secondary broadcasting.Further, this trifle has play PreBar(B before playing for the w time _{i, j, k}, w) individual trifle.

Suppose that the broadcasting time of each trifle is BT(B).LB(i) be the trifle play according to the multimedia playing sequence of reality i-th time, TLB(i) for playing i-th trifle according to the multimedia playing sequence of reality time this trifle broadcasting time.NLB is the little joint number of whole broadcasting.The false code calculating NextLgBar, NextBTime and PreBar is as follows

The step of above false code is as follows:

1. be BT by the broadcasting time of each trifle at the beginning, the temporary variable of the variable NLB trifle that empties that to be 0, TB be, representative, when the trifle of pre-treatment, is initialized as first trifle.

2. travel through each trifle from the beginning to the end: first calculate next trifle, if do not have redirect during this broadcasting of current trifle TB, next trifle is next trifle physically, otherwise next trifle is redirect trifle;

3. the broadcasting time of next trifle is recorded in NextBTime;

4. calculate the information such as PreBar, LB, TLB, and little joint number is altogether added 1, represent process and complete a trifle.

5. last is next trifle by current trifle assignment, and judges whether to terminate, if do not terminate to jump to step 2, otherwise terminates

Just the little joint number NLB of all broadcastings can be obtained fast by above calculating, and the trifle LB(i play for i-th time), any trifle B _ijkbroadcasting time BT(B _ijk).And a certain trifle B can be obtained fast _ijkwhen playing for the w time, next trifle is NextLgBar (B _ijk, w), and next trifle is NextBTime (B _ijk, w) secondary broadcasting.Achieve next trifle obtaining trifle fast and play each time like this.

Described multimedia carries out real time linear location based on sync tag on picture, namely based on the position of calculating multimedia any time on music score of sync tag.

Sync tag is the label manually marked on music score, and for arbitrary label M, the information of its record has a positional information and a temporal information.Positional information have recorded i-th page, and j is capable, and k trifle is played to the position of side-play amount x for the w time.This positional information is played to the little joint number L of process herein from starting point by being converted into.If the length of a trifle B is LX(B) formula that then calculates L is as follows

L(M)＝PreBar(B _i，j，k，w)+x/LX(B _i，j，k) (2)

L (M) represents label M and is played to little joint number herein from starting point, can be decimal.Temporal information T(M) represent the time of this point in multimedia.

If multimedia altogether duration is TT, wherein, corresponding NM+1 label, sorts successively according to the time and is respectively M ₀m _nM.T (M is necessarily met for arbitrary i>j, M _i) >T (M _j) L (M _i) >L (M _j).Even if manually do not mark, in a multimedia, be at least also present in two sync tag M ₀and M _nM.Represent respectively and start label and end-tag, i.e. L (M ₀)=0, T (M ₀)=0; L (M _nM)=NLB, T (M _nM)=TT.

For random time 0<t<TT, MT1 (t)=M between corresponding label area _k, MT2 (t)=M _k+1.Between the label area that time t is corresponding, [MT1 (t), MT2 (t)] is defined as T (M _k) <t<T (M _k+1).Provide below random time t corresponding from count, the formula of the little joint number LT of process:

LT(t)=L(MT1(t))+(L(MT2(t))-L(MT1(t)))×(t-T(MT1(t)))/(T(MT2(t))-T(MT1(t)))

The integral part of LT (t) is ILT, and fraction part is RLT, then the trifle that time t is corresponding is LB (ILB), and the side-play amount in trifle is LX(LB (ILB)) × RLT, the TLB(ILB for this trifle) secondary broadcasting.So just obtain the indicating positions of random time t multimedia in picture.Current broadcasting speed is

Speed(t)=LX（LB(ILB)）×(L(MT2(t))-L(MT1(t)))/(T(MT2(t))-T(MT1(t))) (3)

If the unit of LX is pixel, the unit of T is second, and the unit of Speed is pixel/second, and the speed of namely advancing on picture at t cursor is that Speed (t) pixel is per second.

User can utilize play multimedia, observe the method for progress bar position, the sync tag of real-time adjustment, the upper trifle of clicking on music score of user is to add label, if a trifle needs to play n time, then occur n dot (as shown in Figure 2) from top to bottom, these dots represent sync tag when being played to this place for the 0th ~ n-1 time successively.

When clicking broadcasting at the beginning, two labels are effective end to end to only have acquiescence, and all progresses position according to the method for linear interpolation.When multimedia actual play is to sync tag place, user clicks corresponding sync tag, adds a sync tag in a program, by above-mentioned algorithm, recalculates the position of current indicator, real-time the feeding back on music score of calibration information.

User can be real-time interpolation delete sync tag, repeatedly contrast music change sync tag, with reach calibration each moment pointer position way.

Describedly realize multimedia redirect by clicking picture: after adding sync tag, user clicks i-th page, and j is capable, during the side-play amount x position of k trifle, if it is w second that mouse presses the time of upspringing, determines the broadcasting time w of this trifle.Calculate multimedia time t by the method, and realize multimedia redirect.

This time click corresponding position, the little joint number s formula being played to process herein from starting point is as follows:

s＝PreBar(B _i，j，k，w)+x/LX(B _i，j，k) (4)

For random length 0<s<NLB, ML1 (s)=M between corresponding label area _k, ML2 (s)=M _k+1.Between the label area that length s is corresponding, [ML1 (s), ML2 (s)] is defined as L (M _k) <t<L (M _k+1).The formula of the time t that s is corresponding is as follows:

t=T（ML1(s)）+(s-L(ML1(s)))(T(MT2(s))-T(MT1(s)))/(L(MT2(s))-L(MT1(s)))(5)

Above two formula and user click i-th page, and j is capable, during the side-play amount x position of k trifle, calculate the formula of multimedia time t.Then multimedia is fast-forward to the t time, realizing that institute puts is the effect of gained.

The preservation of described linear multimedia locating information and the technology of reading, namely music score is divided into page, row, trifle by above-mentioned by one, every page of multirow, often capable many trifles, and determine position and the size of page, row, trifle, arrange redirect mark, the process records such as mark sync tag get off, and preserve to be loaded into the technology used later.

The information of record has:

Page information: record successively 0 ... the number of PN-1 page

Row information: the information recording every a line successively, a line i.e. a rectangle, has coordinate x, y of the upper left angle point of corresponding rectangle in preservation, width, highly totally four value.And the numbering of page that this row belongs to.

Trifle information a: trifle is also a rectangle, have recorded the position of corresponding bar line before trifle in the present invention, and the numbering of this trifle corresponding row.

Jump information: have recorded two trifle b1, the numbering of b2 and a number of times t, illustrate b1 to play for the t time terminate after jump to b2.

Sync tag information: the numbering that have recorded corresponding trifle, the offset pixels in trifle, broadcasting time, corresponding time and whether effective.

XML file is saved as by above information, as shown in Figure 3, the information such as whole picture segmentation, sync tag can be carried out record, next time is loaded into this xml when using, recalculate NextLgBar, the information such as NextBTime and PreBar, and use the formula calculating LT (t) Speed (t), again next trifle that trifle is play each time can be obtained, and the corresponding position of multimedia on picture of any time t and the speed of indicator, then realize clicking picture and make multimedia redirect.

Accompanying drawing explanation

Fig. 1 (a) is the schematic diagram with rectangle marked music score position in the present invention;

Fig. 1 (b) is the schematic diagram of input trifle line number columns;

Fig. 1 (c) is the little shackle mark generated according to line number columns;

Fig. 1 (d) is through trickle adjustment, the final little shackle mark result of generation;

Fig. 2 is the example play after adding sync tag;

Fig. 3 saving result examination example;

The color melody of Fig. 4 seven interactive music teaching special software operational effect figure.

Embodiment

Seven color melody interactive music teaching special software V1.0 is a system for music education, and this system achieves science and technology, and music, education combines.Achieve the method in the present invention within the system.Fig. 4 is seven color melody interactive music teaching special software operational effect figure.

Seven color melody interactive music teaching special software V1.0 adopts C/C++ to develop as development language, employs QT technology in R&D process.

Qt is a cross-platform C++ graphical user interface application program development framework developed by Trolltech for 1991.Be the programming language tools under Nokia now.

The interactive music teaching special software of seven color melody achieves and uses photocopy picture and determine playing sequence fast based on trifle.First be loaded into plurality of pictures in systems in which as music score, each figure click the place rectangle marked that " picture square frame " will have music score to mark in every one page, as shown in Fig. 1 (a).Then click " fractionation square frame " and input line number and columns, square frame is split and embarks on journey and trifle.Then user can use the function such as " fractionation trifle ", " deletion trifle ", " mobile bar line " to carry out trickle adjustment to row and trifle, and these all realize fast by mouse drag operation.Then, user double-clicks a certain trifle and arranges redirect, and " generation playing sequence " system of finally clicking adopts the utilization photocopy picture in this method and determines playing sequence fast based on trifle, determines playing sequence.

The interactive music teaching special software of seven color melody achieves the multimedia real time linear location algorithm on music score based on sync tag, user clicks " interpolation sync tag " form with dot on picture can add sync tag, only have two sync tags given tacit consent at the beginning, click after playing, indicator moves according to the way of linear interpolation.User listens to the music while click sync tag, and the time corresponding to sync tag calibrates, and repeatedly obtains last result after calibration.Achieve the multimedia real time linear location algorithm on music score based on sync tag.

In addition in the present system, after repeatedly obtaining last result after calibration, user can click i-th page, j is capable, and during the side-play amount x position of k trifle, native system achieves the method realizing multimedia redirect in this patent by clicking picture, click location is converted into temporal information, realizes multimedia redirect.

Then native system achieves preservation and the reading of linear multimedia locating information, use xml technology, the editor of user can be saved as the file of xml form, as shown in Figure 3, and the xml of preservation can be loaded into use again, achieve and once edit many people use.

The multimedia location technology module of last native system also achieves multi-person synergy editor, Network Synchronization is play, arranged the functions such as repeat playing.Music score can be split as multiple module when music score is larger by multi-person synergy editting function, and many people carry out audition calibration sync tag, finally multiple sync tag result are gathered the calibration realizing whole song.Network Synchronization play-back technology is developed based on the socket of QT, be divided into teacher computer and machine for student in the present system, can by the broadcasting of network control machine for student on teacher computer, serve as a teacher on machine and click broadcasting, time-out, stopping etc., the position at the indicator place on each machine for student is consistent, achieves the synchronous function play like this.Arrange repeat playing to utilize " method realizing multimedia redirect by clicking picture ", achieve and mark between duplicate block with clicking the mode pulling picture, teacher can pass through this functional label emphasis, repeatedly carries out audition.

Seven color melody interactive music teaching special software V1.0 is by achieving all method in the present invention with upper type.

Claims (2)

1., based on a linear multimedia localization method for picture segmentation and sync tag, it is characterized in that comprising following four steps:

(1) use photocopy picture and determine playing sequence fast based on trifle; Use multiple photocopy pictures, fast music score is divided into page, row, trifle; Every page of multirow, often capable many trifles, and determine position and the size of page, row, trifle, and a little redirect mark is set, obtain next trifle that trifle is play each time, have thus determined playing sequence;

(2) multimedia carries out real time linear location based on sync tag on picture; Based on the playing sequence determined in step (1), the time of each trifle of song is fixing, according to the way of linear interpolation, calculates the position of multimedia any time on music score, in music score, add sync tag simultaneously, accurate adjustment is carried out to the inaccurate place of linear interpolation;

(3) multimedia redirect is realized by clicking picture; Add sync tag in step (2) after, user clicks i-th page, and j is capable, during the side-play amount x position of k trifle, calculates multimedia time t, and realizes multimedia redirect;

(4) preservation of linear multimedia locating information and reading; The information of page, row, trifle, redirect mark, sync tag is preserved, and repeatedly can be loaded into use;

Described step (1) determines that playing sequence specifically comprises:

(1.1) input the line number of every one page and little joint number according to actual conditions, and carry out insertion and the deletion of trifle, determine position and the size of page, row, trifle fast;

(1.2) from left to right next trifle physically i.e. next trifle in logic from top to bottom; When running into redirect mark, arrange redirect, a redirect have recorded for a trifle B, next trifle when being played to herein for the w time;

(1.3) calculate the broadcasting time of next trifle that trifle is play each time: according to trifle physical location and redirect, calculate the broadcasting time of next trifle when any i-th page of jth row kth trifle is play for the w time and next trifle.

2. the linear multimedia localization method based on picture segmentation and sync tag according to claim 1, is characterized in that: step (2) specifically comprises:

(2.1) according to the method for linear interpolation, the position of multimedia any time on picture is calculated; The duration of each trifle is fixing, calculates total little joint number, then calculates which trifle of current time correspondence, then finds the position of this trifle in picture, finally add that linear deflection measures the position on picture;

(2.2) add sync tag, accurate adjustment is carried out to the inaccurate place of linear interpolation; Each sync tag accurately specifies i-th page, and j is capable, the time that the label of the side-play amount x position of k trifle is corresponding; In continuous print two sync tags, utilize the method in step (2.1) to determine the position of any time on music score further;

(2.3) sync tag utilizing feedback information to adjust in real time; Music score is clicked the multiple sync tag of mark, and user can calibrate sync tag in real time according to playing progress bar on the music heard and music score, real-time the feeding back on music score of calibration information.