JP5014184B2 - Data editing apparatus, data editing program, and reproducing apparatus - Google Patents

Description

The present invention relates to a data editing apparatus , a data editing program, and a reproducing apparatus that can edit linked display data for displaying text in a predetermined form in accordance with the reproduction of sound.

  Karaoke devices for singing songs while watching the lyrics displayed on the screen according to the music are widely used. Many karaoke devices display lyrics on the screen as music is played, and smoothly change the display color of the lyrics to be sung at an appropriate timing. For example, the display color of the lyrics is changed from gray to white. Such karaoke equipment uses linked display data for changing the display color of lyrics in accordance with music reproduction, in addition to music data and lyrics data. The interlocking display data includes lyric data and time stamp information assigned to each segment obtained by subdividing the lyrics. The time stamp information includes a start time for changing the display color of the lyrics for each section and a time length or an end time until the display color of the lyrics for the section is changed. Note that the start time and end time of each section are represented by the elapsed time from the start of music playback.

  The method for generating linked display data is roughly divided into a method in which a person actually listens to a song including a song and accompaniment, a score data or voice data of a song including a song and an accompaniment, and lyrics data of the song And a computer-generated method using According to the former method, since the operator manually generates the interlocking display data while listening to the music, it takes cost and time. For this reason, the latter method was developed (see Patent Document 1 and Patent Document 2).

  According to the method disclosed in Patent Document 1 (Japanese Patent No. 3787545), a computer displays a score image based on musical score information on a screen and a text string of lyrics based on character information on the screen. , Set the layout on the screen so that the position of each character in the lyrics corresponds to the note on the score, and treat the horizontal axis of the score as the time coordinate to each of the lyrics arranged corresponding to the note on the score Singing time information is added to the text information. The interlocking display data generated by the method is mainly information in which time information is added to each character of the lyrics.

  According to the method disclosed in Patent Document 2 (Japanese Patent Application Laid-Open No. 2001-175267), a computer extracts song voice data from voice data of music and converts it into character data. Collate, correct the character data, and output the first collation result. Further, the computer converts the lyrics data into speech, collates with the song speech data extracted from the speech data, corrects the song speech data, and outputs a second collation result. The computer creates display lyrics data in which display time information is assigned to the display lyrics from the first matching result and the second matching result. The interlocking display data generated by the method is information in which time information is assigned to each section that is part of the lyrics.

  By using the methods disclosed in Patent Document 1 and Patent Document 2 described above, if necessary data is prepared, the computer automatically generates the interlocking display data. Therefore, the cost is lower than the manual generation method. And save time.

Japanese Patent No. 3787545 JP 2001-175267 A Japanese Patent Laid-Open No. 7-046462

  Conventionally, a form in which karaoke equipment displays lyrics on a television screen has been common. For this reason, linked display data for displaying lyrics in a layout suitable for the area on the television screen where the lyrics are displayed has been generated. However, recently, a form in which lyrics are displayed on a screen of a mobile phone, a personal digital assistant (PDA) or the like to reproduce accompaniment has been realized. The screen ratio and screen size differ between the screen of a mobile phone or PDA and the television screen. For this reason, when lyrics are displayed on the screen of the mobile phone or PDA based on the interlocking display data generated for the television screen display, there may be a problem that a part of the lyrics is not displayed. Therefore, it has been necessary to separately generate interlocking display data corresponding to the screen of the mobile phone or PDA.

  Moreover, the technology using the interlocking display data can be applied not only to karaoke but also to teaching materials for learning a foreign language. For example, the linked display data can be used for teaching materials such as displaying a sentence on a screen in accordance with the reproduction of a voice uttered in a foreign language. Note that with such teaching materials, unlike karaoke, it is not necessary to smoothly change the display color of the characters in accordance with the playback of the voice. If the text corresponding to the playback position of the voice is displayed for each line or page, good. As described above, there is a difference that the suitable display form of the text is different between the teaching material for learning a foreign language and karaoke.

  As described above, when displaying text such as lyrics and sentences in a desired format in accordance with the playback of audio such as music and foreign languages, linked display suitable for the text display area and display format on the screen Data needs to be used. However, it is very laborious and costly and time-consuming to individually generate interlocking display data suitable for each of a plurality of display areas or display forms.

  In Patent Document 3 (Japanese Patent Laid-Open No. 7-046462), a video clip is stored as digital data in a memory, the selected clip is displayed on a stretched track on the screen, and the displayed cursor and icon are operated. And a video editing method and apparatus for previewing a clip by combining an edited video program. However, even in this method and apparatus, it is possible to display a voice clip on the screen, relate text such as lyrics and sentences with the voice while referring to the clip, and generate linked display data. It is very laborious and costly and time-consuming to individually generate interlocking display data suitable for each of a plurality of display areas or display forms.

An object of the present invention is to provide a data editing apparatus and a data editing program capable of easily editing linked display data for displaying a text in a predetermined form in accordance with the reproduction of sound according to the display area or display form of the text. And providing a playback device .

The present invention is a data editing apparatus for editing linked display data for displaying a text describing the contents of a voice in a predetermined form in accordance with the playback of the voice, and processes the linked display data. comprising a data processing unit, and a display control unit for generating display data for displaying the linked display data in a predetermined display form, the display control unit, a display area for displaying at least a portion of the text The interlocking display data includes time stamp information indicated by a predetermined elapsed playback time of the sound, attached to each of the text and a character string composed of one or more characters. have a, the data processing unit, in response to an instruction about editing of the interlocking display data, to provide a data editing apparatus and changes the time stamp information .

  In the data editing apparatus, the time stamp information is unique time information that does not change depending on a display form of the text or a set value of the display area.

  In the data editing apparatus, the data editing apparatus includes a data processing unit that processes the interlocking display data, and the data processing unit changes the time stamp information.

  In the data editing apparatus, the data processing unit divides the text included in the interlocking display data into blocks that are sets of character strings composed of a plurality of one or more characters.

Above data editing apparatus, the interlocking display data, a section of each block divided by the data processing unit have a block information indicating the elapsed playback time of the voice, the block information, the time stamp information The start time to start displaying the first character string included in each block and the end time to finish displaying the last character string included in each block or the last character string included in each block are displayed. And the length of time to finish .

  In the data editing apparatus, the text display form is a first form in which one character string is included in one block, a second form in which a character string included in one line of the text is included in one block, and One or more of the third forms in which a character string included in one page corresponding to a text display area is included in one block are included.

  In the data editing apparatus, when one of the text display forms is set, the block information is updated corresponding to the set block.

  In the data editing apparatus, the time stamp information includes a start time at which the display of the character string is started, and an end time at which the display of the character string is completed or a time length until the display of the character string is completed.

  In the data editing apparatus, the block information includes a start time for starting to display the first character string included in each block, an end time for displaying the last character string included in each block, or the last time included in each block. And the length of time until the character string is completely displayed.

  In the data editing apparatus, when the size of the text display area is set, the block information is updated according to the set size.

  In the data editing apparatus, the text display area is set by the number of lines and the maximum number of bytes per line.

  In the data editing apparatus, the text display area is set by the number of lines and the maximum number of characters per line.

  In the data editing apparatus, the editing screen displayed in the predetermined display form based on the display data generated by the display control unit includes a first area displaying at least a part of the text, and the plurality of blocks And a second area for displaying the character strings of the text divided into two on the time axis of the elapsed playback time for each block based on the block information.

  In the data editing apparatus, when the first form or the second form is set as the display form of the text, a part of the text is displayed for each line in the first region, When the third form is set as the text display form, a part of the text is displayed for each page in the first area.

  In the data editing apparatus, the editing screen displays information indicating a display area of the text or information indicating a display form of the text.

  In the data editing apparatus, when the display control unit cannot display all the character strings included in one block in the second area, the display control unit displays the display data so as to display a predetermined image or character string in the block. When the predetermined image or character string or the block is selected, display data for displaying all the character strings on different screens is generated.

  In the data editing apparatus, when the first area or the second area on the editing screen displayed in the predetermined display form based on the display data generated by the display control unit is selected, each area is selected. The audio is reproduced based on the elapsed reproduction time of the interlocking display data included in.

The data editing apparatus includes a determination unit that determines whether an instruction regarding editing of the interlocking display data is prohibited, and the instruction regarding editing of the prohibited interlocking display data is on a time axis. This is an instruction to change the order of blocks, or an instruction to overlap a plurality of blocks adjacent on the time axis .

Above data editing apparatus, an instruction about editing of the interlocking display data, when an instruction to overlay a plurality of blocks adjacent on the time axis, the data processing unit, the end time of the temporally preceding block or temporally processing the interlocking display data while keeping the same time the start time of the time length and the temporally subsequent block of the previous block.

  In the data editing device, when the text is paused while being displayed in the predetermined form in accordance with the playback of the voice based on the interlocking display data, the data processing unit is The start time or end time or time length of the block including the character string displayed in the predetermined form in the playback elapsed time is changed to the playback elapsed time.

  The data editing apparatus includes a data generation unit that generates data for interlocking display of the partial range using the partial data of the audio reproduction time and the text data.

  In the data editing apparatus, the display control unit generates display data based on the interlocking display data to which the reliability is given for each character string so that the display color or pattern of each character string varies depending on the reliability.

  In the data editing device, the data processing unit separates the linked display data within a specified range using a plurality of linked display data generated based on one audio data and corresponding text data. Replace the data with the same range of the linked display data.

  In the data editing apparatus, the linked display data includes line information indicating the line number of the text displayed in the first area, which is attached to each character string including one or more characters. The display control unit generates display data for displaying the line number indicated by the line information corresponding to the character string displayed in the second area in the second area.

  In the data editing apparatus, when the row number displayed in the second area is displayed by a block, and the width of each block is changed by an operation by the user of the data editing apparatus, the data processing unit The line information included in the interlocking display data is processed.

  In the data editing apparatus, the second area displays the same number of time axes as the number of lines of the first area, and each time axis is assigned a different line number.

  In the data editing apparatus, when the temporal position of the character string of the text is changed, the display control unit changes the value of the time axis affected by the change.

  The data editing apparatus includes a determination unit that determines whether an instruction relating to editing of the linked display data is prohibited, and the prohibited editing instruction of the linked display data is on a time axis. This is an instruction to change the order of blocks, or an instruction to overlap a plurality of blocks adjacent on the time axis.

  In the above data editing apparatus, the display control unit calculates the page number derived from the line number indicated by the line information corresponding to the character string displayed in the second area and the number of lines included in the first area. Display data to be displayed in the second area is generated.

  In the data editing apparatus, the display control unit compares the size of the display area with the size of each character string composed of one or more characters of the linked display data, and the first area and Display data to be displayed in the second area is generated.

  In the data editing apparatus, the interlocking display data holds a plurality of block information corresponding to the size of the display area of the text and the display form in accordance with each setting.

In the data editing apparatus, the data processing unit converts a value obtained by subtracting the start time from the end time included in the time stamp information or the time length included in the time stamp information into a character string of 0. A value obtained by editing the attached time stamp information and subtracting the start time from the end time included in the time stamp information of the character string , or the time length included in the time stamp information is a predetermined value or more. The interlocking display data is processed so that the time becomes.

In the data editing apparatus, a character string in which a value obtained by subtracting the start time from the end time included in the time stamp information or the time length included in the time stamp information is a punctuation mark or a special character. .

  The present invention provides a data editing program for causing a computer to function as each unit included in the data editing apparatus.

  The present invention provides a playback device that displays text based on linked display data edited by the data editing device or the data editing program at a predetermined timing in accordance with the playback of audio.

  The present invention displays text in accordance with voice reproduction based on the interlocking display data edited by the data editing apparatus or the data editing program, and acquires a user's selection location using the display portion. Thus, a playback device that starts playback from the character string is provided.

According to the data editing device , the data editing program, and the playback device according to the present invention, the interlocking display data can be easily edited according to the display area or the display form.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. A data editing apparatus according to an embodiment described below is used for editing linked display data generated in advance for displaying text in a predetermined form in accordance with the reproduction of sound. Note that “speech” means karaoke accompaniment , back chorus , singing voice, foreign language audio teaching materials, and the like. The text is a Japanese or foreign sentence, lyric, etc. including punctuation marks and special characters. In addition, displaying text in a predetermined form means, for example, changing the transparency of the text display color from 50% to 0%, changing the text display color to another easy-to-see color, It is to display from a state that is not displayed.

  A monitor is connected to the data editing apparatus. The data editing apparatus operates to display on the monitor a screen on which the user can visually and intuitively grasp the interlocking display data being edited.

  When the interlocking display data is generated, the text is divided into character strings composed of one or more characters. This character string may be a morpheme (minimum unit having meaning in the language), a single character, or a phoneme (minimum unit of sound possessed by the language). In the following description, morphemes are used as examples. The generated interlocking display data includes text data and time stamp information that is unique time information assigned to each morpheme obtained by subdividing the text. The time stamp information includes a start time for displaying each morpheme in a predetermined form, and a time length or an end time until the morpheme is displayed in a predetermined form. These start time and end time are indicated by a predetermined playback elapsed time of the sound.

  The linked display data includes a start time for displaying the first morpheme of each group composed of a plurality of morphemes in a predetermined form, and a time until all the morphemes in the group are displayed in a predetermined form. Also included is block information including an end time for displaying the last or the last morpheme of the group in a predetermined form. Details of the block information will be described later.

(First embodiment)
FIG. 1 is a diagram illustrating an example of a screen during editing of interlocking display data displayed on a monitor by the data editing apparatus according to the first embodiment. A screen 10 for editing the interlocking display data shown in FIG. 1 (hereinafter referred to as an “edit screen”) 10 includes information 11 indicating a text display area, information 13 indicating a text display form, and actual text. A text part 15 indicating the display and a timeline part 17 for displaying the text morphemes arranged in time series in a state surrounded by blocks are displayed. The text portion 15 is based on text data included in the interlocking display data. The timeline unit 17 is based on time stamp information and block information included in the interlocking display data. Furthermore, in the screen example shown in FIG. 1, “6 rows and 30 bytes” is selected as the display area, and “karaoke mode” is selected as the display form. (Note that one Japanese character is 2 bytes.)

  The operator of the data editing apparatus of this embodiment sets the number of lines and the maximum number of bytes per line, such as “10 lines 30 bytes” or “3 lines 20 bytes” as the display area. In the text portion 15 of the editing screen, text that fits within the number of lines and the maximum number of bytes set as the display area is displayed. Note that the display area may be set by the maximum number of characters per line and the number of lines instead of the maximum number of bytes.

  Further, the operator selects one of “karaoke mode”, “scroll mode”, and “page switching mode” as the display form. When “Karaoke Mode” is selected, one morpheme is displayed in a state surrounded by one block on the timeline portion 17 of the editing screen. For this reason, the block information included in the interlocking display data in the karaoke mode is the same as the time information indicated by the time stamp information of the corresponding morpheme.

  The text portion 15 of the editing screen shows the actual display of the text at the specified playback elapsed time. That is, the text portion 15 displays text before and after the morpheme corresponding to the designated reproduction elapsed time and the state in which the display color of the morpheme corresponding to the designated reproduction elapsed time is changed. As described above, when the voice is reproduced using the interlocking display data in which the karaoke mode is selected as the display form, the morpheme corresponding to the elapsed reproduction time of the text displayed on the text portion 15 is synchronized with the reproduction of the voice. The display color of changes.

  When “scroll mode” is selected, the morpheme included in one line of text displayed in the text portion 15 is displayed in the timeline portion 17 of the editing screen in a state surrounded by one block. Therefore, the block information included in the interlocking display data in the scroll mode includes the start time indicated by the time stamp information of the first morpheme in each row and the end time indicated by the time stamp information of the last morpheme in that row. .

  The text portion 15 of the editing screen shows the actual display of the text at the specified playback elapsed time. In other words, the text portion 15 displays the text before and after the line including the line corresponding to the designated reproduction elapsed time and the state in which the display color of the line corresponding to the designated reproduction elapsed time is changed. The number of characters per line and the number of lines per page are based on the set display area. When the scroll mode is selected, the text displayed in the text portion 15 is scrolled in the line direction in accordance with the sound reproduction. As described above, when the sound is reproduced using the interlocking display data whose scroll mode is selected as the display mode, the line corresponding to the elapsed reproduction time of the text displayed in the text portion 15 is synchronized with the sound reproduction. The display color of changes.

  When “page switching mode” is selected, the morpheme included in one page is displayed in a state surrounded by one block on the timeline portion 17 of the editing screen. Therefore, the block information included in the linked display data in the page switching mode includes the start time indicated by the time stamp information of the first morpheme of each page, and the end time indicated by the time stamp information of the last morpheme of the page. including.

  The text portion 15 of the editing screen shows the actual display of the text at the specified playback elapsed time. That is, the text of the page corresponding to the designated elapsed playback time is displayed in the text portion 15. The number of lines per page and the number of characters per line are based on the set display area. As described above, when the voice is reproduced using the interlocking display data selected as the display mode in the page switching mode, the text displayed in the text portion 15 is switched for each page in accordance with the reproduction of the voice.

  As described above, in the present embodiment, the number of lines and the number of characters that can be displayed in the text portion 15 of the editing screen are changed according to the set display area, and the text portion 15 is changed according to the set display form. The way of displaying the text displayed on the screen in accordance with the sound reproduction and the way of dividing the text by a plurality of blocks are changed.

  In each block displayed on the timeline portion 17 in the edit screen, as shown in FIG. 1, text corresponding to the mode of display mode (morpheme, text included in one line, text included in one page) ) Is displayed. However, if the amount of text that should fit in one block does not fit, an image such as an icon is displayed in the block instead of the text. In this case, when the operator of the data editing apparatus positions the pointer on the display screen that can be operated by the mouse on the icon, the data editing apparatus displays the text corresponding to the block of the icon in another window or a balloon.

  FIG. 2 is a block diagram illustrating the data editing apparatus according to the first embodiment. As shown in FIG. 2, the data editing apparatus 100 according to the first embodiment includes an input unit 101, a data storage unit 103, a setting storage unit 105, a reception unit 107, a data processing unit 109, and a display control unit. 111. The receiving unit 107, the data processing unit 109, and the display control unit 111 operate by executing a program. As described above, the monitor 151 is connected to the data editing apparatus 100. A screen based on display data generated by the display control unit 111 is displayed on the monitor 151.

  The input unit 101 is an input interface for storing audio data and interlocking display data corresponding to the audio data in the data storage unit 103. The data storage unit 103 stores audio data and linked display data input via the input unit 101. The setting storage unit 105 stores information related to the display area according to the number of lines and the maximum number of bytes per line, and information related to the display mode according to each mode of the karaoke mode, the scroll mode, and the page switching mode.

  The receiving unit 107 receives an operation for setting a display area or a display form and editing linked display data by an operator of the data editing apparatus 100. The operator can edit with the GUI while referring to the editing screen displayed on the monitor 151. The display area is set when the operator clicks on the information 11 indicating the display area in the editing screen and inputs or changes the number of lines and the number of bytes in a newly displayed window. The display form is set when the operator clicks on the information 13 indicating the display form on the editing screen and selects one mode. Further, the time stamp information of the interlocking display data is obtained by the operator selecting a block displayed on the timeline unit 17 in the editing screen with the pointer and moving it in the time direction (horizontal direction in FIG. 1) Edited by changing the length of the direction. The receiving unit 107 analyzes the received operation and sends an instruction indicated by the operation to the data processing unit 109.

  The data processing unit 109 processes the interlocking display data read from the data storage unit 103 based on the instruction sent from the receiving unit 107. When the display form is set, the data processing unit 109 reads information on the set display form from the setting storage unit 105, and according to the set display form, one morpheme, a morpheme included in one line, Or, it is divided into a plurality of blocks including morphemes included in one page. That is, the data processing unit 109 creates or updates block information including the start time and end time of each block according to the set display form. However, when “scroll mode” or “page switching mode” is selected as the display form, the morphemes included in one line or one page (one block) differ depending on the display area. For this reason, the data processing unit 109 reads out information related to the set display area from the setting storage unit 105, and the morpheme included in one line or one page is displayed for each line or page according to the set display area. Decide and create or update block information.

  Also, after the processing according to the display form and display area by the data processing unit 109, when a time stamp information change instruction is sent from the receiving unit 107, the data processing unit 109 changes the time stamp information according to the instruction. To do. For example, when “karaoke mode” is selected as the display form and an instruction to shift the start timing of the morpheme corresponding to the specified block is received, the data processing unit 109 starts the start time corresponding to the morpheme in the time stamp information To change.

  Further, the data processing unit 109 has a cache memory (not shown), and temporarily stores the processed interlocking display data in the cache memory. When instructed to write the processed interlocking display data to the data storage unit 103, the data processing unit 109 stores the interlocking display unit data stored in the cache memory in the data storage unit 103.

  The display control unit 111 monitors the interlocking display data (text, time stamp information and block information) processed by the data processing unit 109, and each information indicating the display area and the display form in the form of the editing screen described above. Display data to be displayed in 151 is generated. Further, when the receiving unit 107 receives an instruction to change the display area from the operator, the display control unit 111 reads information on the set display area from the setting storage unit 105, and the text unit 15 on the editing screen displays the desired text area 15. Change the display data so that it becomes the display area.

  Furthermore, when the amount of text that should be contained in one block does not fit, the display control unit 111 changes the display data so that an icon is displayed in the block instead of the text. When the editing screen based on the display data is displayed on the monitor, when the pointer is moved to the position of the icon, the display control unit 111 displays the data corresponding to the block of the icon so that the text is displayed in another window or a balloon. To change.

  In the present embodiment, the data processing unit 109 retains the change contents of the time stamp information performed during editing of the interlocking display data even when the display form is changed. For example, when the page switching mode is selected as the display mode, if the start time of the top morpheme of a page is changed, the karaoke mode is changed even if the display mode is changed from the page switching mode to the karaoke mode after this. The start time of this morpheme in is the changed time.

  This is because time stamp information is uniquely assigned to each morpheme regardless of the display form, and the block information uses time stamp information of the morpheme.

  However, for example, after changing the start time of the morpheme when the karaoke mode is selected, the display form is changed to the scroll mode or the page switching mode, and the morpheme whose start time is changed is Since the block displayed on the timeline portion 17 in the edit screen is displayed based on the start time and end time when it is located in the middle other than the last, the display changes before and after the start time change of the morpheme There seems to be no.

(Second Embodiment)
In the second embodiment, an operation of changing the order of blocks on the time axis and an operation of overlapping a plurality of adjacent blocks on the time axis are prohibited. That is, the operator of the data editing apparatus cannot change the order of the blocks on the time axis or superimpose a plurality of adjacent blocks on the time axis when moving the blocks with reference to the editing screen. In order to realize this function, the data editing apparatus according to the present embodiment includes a determination unit 113 between the reception unit 107 and the data processing unit 109 in addition to the components included in the data editing apparatus 100 according to the first embodiment. Prepare.

  FIG. 3 is a block diagram illustrating a data editing apparatus according to the second embodiment. The determination unit 113 determines whether the operation instruction analyzed by the reception unit 107 is a prohibited operation. The determination unit 113 sends only instructions related to operations that are not prohibited to the data processing unit 109 and does not send instructions related to prohibited operations. For example, when a block selected by the operator is moved, the determination unit 113 sends an instruction related to this operation to the data processing unit 109 until this block and an adjacent block come into contact with each other. However, when an operation of overlapping at least a part of these two blocks is performed, the determination unit 113 does not send an instruction relating to this operation to the data processing unit 109.

  In this embodiment, when an operation of overlapping two blocks displayed on the timeline unit 17 of the editing screen is performed, the determination unit 113 touches these two blocks in place of an instruction related to the operation. An instruction to move collectively in a state is sent to the data processing unit 109. FIG. 4 shows a part of the screen displayed on the timeline unit 17 when one of adjacent morpheme blocks is moved across a blank time zone with the karaoke mode selected as the display mode. FIG. As shown in FIG. 4, when the later block 21 is moved forward in time, the later block 21 is moved until the start time of the later block 21 matches the end time of the preceding block 23. . However, when trying to move the later block 21 further forward, the preceding block 23 also moves while these two blocks 21 and 23 are in contact.

  In this way, the data processing unit 109 that has received an instruction to move the blocks together while in contact with each other starts the start of each block while maintaining the end time of the previous block and the start time of the subsequent block at the same time. Time stamp information and block information included in the interlocking display data are processed so as to change the time and the end time. In the present embodiment, two adjacent blocks have been described, but the same applies to three or more blocks adjacent in time series. Moreover, although FIG. 4 demonstrated the case where karaoke mode was selected as a display form, it is the same also about scroll mode and page switching mode.

(Third embodiment)
In the third embodiment, when the operator pauses the display of the text in accordance with the playback of the voice, the start time or the end time of the morpheme displayed in a predetermined form during the playback elapsed time when the pause is performed Is changed to the playback elapsed time. Note that the data editing apparatus according to the present embodiment has a function of reproducing sound and displaying text in a predetermined form based on data for linked display (hereinafter, these are collectively referred to as “reproduction”). In order to realize this function, the data editing apparatus according to the present embodiment further includes a reproducing unit 115 in addition to the components included in the data editing apparatus 100 according to the first embodiment.

  FIG. 5 is a block diagram illustrating a data editing apparatus according to the third embodiment. The reproduction unit 115 reads out and reproduces audio data and interlocking display data from the data storage unit 103. The text is displayed on the monitor 151 via the display control unit 111, and the sound is output from a speaker 153 provided outside. The receiving unit 307 of this embodiment accepts an operation for reproduction by an operator. The receiving unit 307 analyzes this operation and sends a playback instruction to the playback unit 115. When the playback unit 115 receives a playback instruction from the reception unit 307, the playback unit 115 starts playback processing. The receiving unit 307 also accepts an operation for pausing playback. When the playback unit 115 receives a pause instruction from the reception unit 307, the playback unit 115 pauses playback.

  The receiving unit 307 of the present embodiment also accepts an operation for changing the start time or end time of the morpheme displayed at the playback elapsed time when paused to the playback elapsed time. The receiving unit 307 sends an instruction indicated by this operation to the data processing unit 109. The data processing unit 109 processes the interlocking display data according to this instruction. FIG. 6 shows that the start time (a) and the end time (b) of the morpheme displayed at the playback elapsed time when the karaoke mode is selected as the display mode are changed to the playback elapsed time. It is a figure which shows a mode when doing. As shown in FIGS. 6A and 6B, the data processing unit 109 changes the disclosure time or the end time of the morpheme 31 when paused to the playback elapsed time at the pause time. Note that the data processing unit 109 does not change the end time of the morpheme whose start time has been changed. Similarly, the data processing unit 109 does not change the start time of the morpheme whose end time has been changed.

  Although FIG. 6 illustrates an example in a state where the karaoke mode is selected, the same applies to the scroll mode and the page switching mode. However, in the case of the scroll mode and the page switching mode, the start time or end time of the block including the morpheme when temporarily stopped is changed.

  In this embodiment, when the operator selects a block displayed in a predetermined form, the reception unit 307 notifies the reproduction unit 115 of the morpheme included in the block, and the reproduction unit 115 The start time and end time of the morpheme may be acquired, and the sound corresponding to the morpheme included in the block may be played back.

(Fourth embodiment)
In the fourth embodiment, the interlocking display data corresponding to a partial range of the audio reproduction time is regenerated. Note that the range of a part of the audio playback time is specified by the operator. In order to realize this function, the data editing apparatus according to the present embodiment includes a data generation unit 117 in addition to the components included in the data editing apparatus 100 according to the first embodiment.

  FIG. 7 is a block diagram showing a data editing apparatus according to the fourth embodiment. The data generation unit 117 generates the interlocking display data in the range specified by the operator using the voice data and text data in this range. As a method for generating linked display data, even if the related technology (Japanese Patent Laid-Open No. 2001-175267) is used, the corresponding data is searched and acquired from linked display data prepared in advance on the Web or the like. Also good. The receiving unit 407 of the present embodiment accepts an operation for designating a range by an operator and an operation for instructing generation of linked display data in this range. The receiving unit 407 analyzes these operations and sends an instruction to generate data for interlocking display within a specified range to the data generating unit 117. Upon receiving this generation instruction, the data generation unit 117 reads out voice data and text data from the data storage unit 103 and starts generation processing. The data generation unit 117 overwrites the generated linked display data on the corresponding portion of the original linked display data stored in the data storage unit 103.

(Fifth embodiment)
In the fifth embodiment, the interlocking display data stored in the data storage unit 103 is assigned a reliability for each morpheme. The method for deriving the reliability of each morpheme is not limited to one. For example, the reliability may be derived based on the average and standard deviation of the time stamp information of each morpheme using a plurality of interlocking display data manually generated by different persons. Note that the data editing apparatus of the fifth embodiment has the same configuration as that of the data editing apparatus 100 of the first embodiment, and is not shown.

  When the display control unit of the present embodiment generates display data for displaying the time stamp information included in the interlocking display data on the timeline unit 17 of the edit screen, the display color of the morpheme displayed in the block or Display data is generated so that the pattern varies depending on the reliability. In addition, even when the value of the sum of the reliability of all the morphemes is lower than a predetermined threshold, the name of the associated display data or the voice data corresponding to the associated display data is displayed on the monitor 151. good.

(Sixth embodiment)
In the sixth embodiment, a plurality of interlocking display data generated based on one voice data and corresponding text data is stored in the data storage unit 103, and as shown in FIG. The interlocking display data 41 in the range specified by is replaced with data 43 in the same range of another interlocking display data.

  In order to realize this function, the receiving unit 107 according to the present embodiment displays an operation for designating a partial range of the audio playback time by the operator and the associated display data for the designated range in another linked display. Accepts an operation to replace the data with business data. The receiving unit 107 analyzes these operations, and sends to the data processing unit 109 an instruction to replace the linked display data designated in the designated range with data of another linked display data. The data processing unit 109 reads the interlocking display data from the data storage unit 103 and processes the interlocking display data according to the instruction sent from the receiving unit 107. Note that the data editing apparatus of the sixth embodiment has the same configuration as the data editing apparatus of the first embodiment, and is not shown.

  When the reliability described in the fourth embodiment is attached to each linked display data, the operator may determine the replacement range and the linked linked display data with reference to the reliability. .

(Seventh embodiment)
In the seventh embodiment, in addition to the configuration of the above-described embodiment, the linked display data is not only time stamp information but also a text portion 15 based on a set display area as information assigned to each morpheme. Includes information indicating the line number of the text displayed in (hereinafter referred to as “line information”). FIG. 9 is a diagram illustrating an example of a screen during editing of the interlocking display data displayed on the monitor by the data editing apparatus according to the seventh embodiment. As shown in FIG. 9, the display control unit 111 displays time information on the timeline unit 27 of the editing screen 20 of the present embodiment based on the line information and display area assigned to each morpheme included in the interlocking display data. In parallel with the line 271, the row block 273 and the page block 275 are controlled to be displayed on the monitor 151. The row block 273 indicates on which line each morpheme is displayed, and the page block 275 indicates on which page each morpheme is displayed. Note that the lines and pages in the present embodiment are not the actual lines and pages displayed in the text part 15 but the lines and pages in the entire text that can be displayed in the text part 15.

The page number indicated by the page block 275 can be derived from the line number assigned to the morpheme and the number of lines indicated by the display area. In the example shown in FIG. 9, since “2 rows and 16 bytes” is selected as the display area, the text portion 15 displays 2 rows per page. Therefore, the morpheme assigned as the first or second line is displayed on the first page, and the morpheme assigned as the third or fourth line is displayed on the second page.
The plurality of page numbers derived from the number of lines and the number of bytes indicated by each display area may be held in the interlocking display data.

  The operator of the data editing apparatus according to the present embodiment can slide either the left or right end of the row block 273 in the timeline unit 27 to the left or right using the pointer in the display screen that can be operated with the mouse. . By sliding the end of the row block 273, the data processing unit 109 changes the row information assigned to the morpheme related to the end.

  For example, on the editing screen 20 shown in FIG. 9, when the operator slides the right end of the row block indicating the second row (the left end of the row block indicating the third row) to the left as shown by the dotted line, The last morpheme “no” line information is changed from the second line to the third line. In this example, the right end of the row block indicating the second row (the left end of the row block indicating the third row) and the right end of the page block indicating the first page (the left end of the page block indicating the second page) are the same time. Since it should be positioned above, when the above slide is performed, the right end of the page block indicating the first page (the left end of the page block indicating the second page) is also slid.

  In addition, since the line information does not change in the middle of the morpheme, when the operator slides the end of the line block to the middle of the morpheme, the start time and end time of the morpheme are compared with the time of the slide position. Then, the end of the row block may be automatically slid to a close position, and the end of the row block may be changed to the value after the slide for the corresponding morpheme row information. The page information is the same as that when the row information is operated.

  According to the present embodiment, the operator of the data editing apparatus can edit data while visually recognizing the row and page on which each morpheme is displayed by the display of the timeline unit 27. In particular, the line feed position on the text portion 15 can be visually recognized.

  In this embodiment, the example in which the information indicating the row number is already included in the interlocking display data has been described. However, the display control unit 111 sets the number of bytes of each morpheme of the interlocking display data and the display area 11. By comparing the number of bytes and the number of lines, the necessary number of lines and the number of pages are calculated, and the calculation results of the number of lines and the number of pages are stored in the interlocking display data. It may be displayed. A plurality of pieces of line information included in the linked display data may be held based on the number of bytes and the number of lines set in the display area 11. In this embodiment, the row block 273 and the page block 275 are arranged vertically in the timeline unit 27. However, the corresponding morpheme is surrounded by a frame for each range indicating a row and a page, and the size of the frame. You may let the operator edit by changing

(Eighth embodiment)
In the eighth embodiment, the interlocking display data is the same as in the seventh embodiment, but the editing screen displayed on the monitor 151 is different. FIG. 10 is a diagram illustrating an example of a screen during editing of the interlocking display data displayed on the monitor by the data editing apparatus of the eighth embodiment. As shown in FIG. 10, the display control unit 111 has the same number of lines set in the display area 11 in the timeline unit 37 of the editing screen 30 of the present embodiment, that is, the same number of lines displayed in the text unit 15. Control is performed so as to display the timeline and the morpheme assigned to each timeline on the monitor 151. In addition, the corresponding line number 373 is shown in each timeline. Further, the corresponding page number 375 is also displayed on each timeline.

  In the example shown in FIG. 10, since “3 rows and 30 bytes” is selected as the display area, three lines per page are displayed in the text portion 15. Therefore, the morpheme assigned as the first, second, or third line is displayed on the first page, and the morpheme assigned as the fourth, fifth, or sixth line is displayed on the second page. .

The operator of the data editing apparatus according to the present embodiment can correct the time information of the block by moving the block in the timeline unit 37 left and right using the pointer in the display screen that can be operated with the mouse. . Further, the block can be moved to another line, and by moving the block to another line, the data processing unit 109 changes the line information assigned to the morpheme included in the block. Further, based on the time stamp information assigned to the morpheme included in the moved block, the display control unit 111 changes each timeline of the destination row and the source row.
When moving the block to another line, the time information of the block is not changed.

  In the editing screen 30 shown in FIG. 10, for example, when the operator designates a block including the last morpheme “ha” on the second line and moves it to the third line, the block moves to the top of the third line. Moreover, the timeline of the 2nd line and the 3rd line is changed by the said movement. The second line timeline was changed from 5 to 9 seconds to 5 to 8 seconds, and the third line timeline was 9 to 12 seconds and 8 to 12 seconds. Changed to That is, the timeline in the second row is changed to a time excluding the time indicated by the time stamp information of the morpheme included in the moved block, and the timeline in the third row is the time indicated by the time stamp information of the morpheme. Will be changed to the time you added.

  Also, in the present embodiment, when the operator designates a block including the morpheme “Tori” located second from the end of the second row and moves it to the third row on the editing screen 30 shown in FIG. Along with the block, the block including the last morpheme “ha” in the second row is also moved to the beginning of the third row. Note that the temporal relationship between the two moved blocks does not change. Also during the movement, the timelines of the second and third lines are changed.

  In the above description, the block movement from the young line number to the old line number has been described. In the opposite case, the moved block moves to the end of the destination line.

  According to this embodiment, the operator of the data editing apparatus can edit data while visually recognizing the row and page on which each morpheme is displayed by the display of the timeline unit 37. In particular, the line feed position on the text portion 15 can be visually recognized.

(Ninth embodiment)
FIG. 11 is a diagram illustrating an example of a screen during editing of the interlocking display data displayed on the monitor by the data editing apparatus according to the ninth embodiment. As illustrated in FIG. 11, the display control unit 111 displays a waveform indicating sound including accompaniment sounds and the like on the editing screen 40 of the ninth embodiment in parallel with the timeline of the timeline unit 17 on the same time scale. 18 or a waveform 19 showing only the singing voice is displayed. The waveform data is stored in the data storage unit 103. According to the present embodiment, the block can be moved by visual judgment based on changes in the waveforms 18 and 19.

(Tenth embodiment)
As described above, the text may include punctuation marks and special characters. However, punctuation marks? ,! Special characters such as are one morpheme but have no sound. For this reason, when voice recognition technology is used when generating time stamp information assigned to the morpheme, a predetermined playback elapsed time of the voice is set as the start time, but 0 is set as the time length. The end time may be set to the same time as the start time. For this reason, the morpheme is represented as a line having no width at all in the timeline portion 17 in the editing screen 10. That is, the morpheme is not displayed as an operable morpheme in the timeline unit 17.

  In the tenth embodiment, the data processing unit 109 changes the time length included as time stamp information of such a morpheme to 0.5 seconds, for example, and the time difference between the start time and the end time is 0.5 seconds, for example. Change the time stamp information. FIG. 12 is a diagram illustrating an example of a timeline unit displayed on a monitor before and after editing by the data editing apparatus of the tenth embodiment. For example, as shown in FIG. 12A, the text includes a morpheme “?” After the morpheme “Yaru”, but the morpheme “?” Has no sound. Is not displayed. However, in this embodiment, the time stamp information of the morpheme is changed so that the morpheme has a time length of about 0.5 seconds. That is, as the time stamp information of the morpheme “?” Before the change, the start time is set to 11.7 seconds and the time length is set to 0 seconds (end time is 11.7 seconds). It is changed to 5 seconds (end time is 12.2 seconds). Therefore, as shown in FIG. 12B, the morpheme “?” Is displayed behind the morpheme “do”.

In addition, when another morpheme is set before and after such a morpheme and the time width is given to the morpheme, it may overlap with an adjacent morpheme. In such a case, the data processing unit 109 shortens the time length of the adjacent morpheme and sets the time length of the morpheme to be displayed as described above, so that the display control unit 111 wants to display the morpheme. The morpheme can be displayed on the timeline unit 17.
In the present embodiment, the method of displaying the morpheme that is not displayed alone is described. However, the morpheme adjacent to the morpheme may be displayed.
For example, by making the end time of the morpheme immediately before the morpheme the end time of the morpheme, it can be included in the immediately preceding morpheme. Further, by making the start time of the morpheme immediately after the morpheme the start time of the morpheme, it can be included in the immediately preceding morpheme.

(Eleventh embodiment)
FIG. 13 is a diagram illustrating an example of a screen during editing of the interlocking display data displayed on the monitor by the data editing apparatus according to the seventh embodiment. In the eleventh embodiment, as shown in FIG. 13, when the morpheme displayed in the text part 15 of the editing screen 10 is specified by the pointer in the display screen that can be operated by the mouse, the display control unit 111 The morpheme specified by the pointer is controlled to be displayed in a different color in the timeline unit 17. In addition, after recognizing the block including the morpheme specified by the pointer according to the display form, all the morphemes included in the recognized block may be displayed in the timeline unit 17 in color.

Furthermore, in the present embodiment, after recognizing a block including the morpheme specified by the pointer according to the display form, the sound from the first morpheme included in the recognized block to the last morpheme is reproduced, Output from a speaker (not shown).
The display control unit 111 calculates the position in the editing screen for each morpheme when displaying each morpheme in the text part 15 and the timeline part 17 based on the number of lines and the number of bytes in the display area 11. The relationship between the position and the morpheme may be stored in the linked display data.

  The data editing apparatus according to the present invention easily edits linked display data for displaying text such as lyrics in a predetermined form in accordance with the reproduction of sound such as accompaniment according to the display area or display form of the text. It is useful as a device or program that can be used.

The figure which shows an example of the screen which is editing the data for interlocking | linkage display displayed on a monitor by the data editing apparatus of 1st Embodiment 1 is a block diagram showing a data editing apparatus according to a first embodiment. The block diagram which shows the data editing apparatus of 2nd Embodiment The figure which shows a part of screen displayed on a timeline part when moving one of the adjacent morpheme blocks across a blank time zone in a state where karaoke mode is selected as the display mode The block diagram which shows the data editing apparatus of 3rd Embodiment Block when changing the start time (a) and end time (b) of the morpheme displayed in the playback elapsed time when paused with the karaoke mode selected as the display mode to the playback elapsed time Figure showing The block diagram which shows the data editing apparatus of 4th Embodiment The figure which shows a mode that the data for interlocking display of some ranges are replaced with the data of the same range of the data for another interlocking display The figure which shows an example of the screen which is editing the data for interlocking | linkage display displayed on a monitor by the data editing apparatus of 7th Embodiment The figure which shows an example of the screen which is editing the data for interlocking | linkage display displayed on a monitor by the data editing apparatus of 8th Embodiment The figure which shows an example of the screen during editing the data for interlocking | linkage display displayed on a monitor by the data editing apparatus of 9th Embodiment The figure which shows an example of the timeline part displayed on the monitor before and behind the edit by the data editing apparatus of 10th Embodiment. The figure which shows an example of the screen which is editing the data for interlocking | linkage display displayed on a monitor by the data editing apparatus of 11th Embodiment

DESCRIPTION OF SYMBOLS 101 Input part 103 Data storage part 105 Setting storage part 107,307,407 Reception part 109 Data processing part 111 Display control part 113 Determination part 115 Reproduction part 117 Data generation part

Claims (31)

A data editing device for editing linked display data for displaying a text describing the contents of a sound in a predetermined form in accordance with the reproduction of the sound,
A data processing unit for processing the linked display data;
A display control unit for generating display data for displaying the interlocking display data in a predetermined display form ;
With
The display control unit sets a size of a display area for displaying at least a part of the text;
The interlocking display data, possess the text, attached to each of the configured string one or more characters, and a time stamp information indicated by the predetermined elapsed playback time for the audio,
The time stamp information is unique time information that does not change depending on the display form of the text or the setting value of the display area,
The data editing device , wherein the data processing unit changes the time stamp information . The data editing device according to claim 1 ,
The time stamp information includes a start time at which a character string starts to be displayed and an end time at which the character string has been displayed or a time length until the character string has been displayed. The data editing device according to claim 1 or 2 ,
The data editing apparatus, wherein the data processing unit divides the text included in the interlocking display data into a plurality of blocks that are sets of character strings including one or more characters. The data editing device according to claim 3 , wherein
The interlocking display data, a section of each block divided by the data processing unit have a block information indicating the elapsed playback time for the audio,
The block information is a part of the time stamp information, the start time to start displaying the first character string included in each block, and the end time to finish displaying the last character string included in each block or each block And a time length until the last character string included in is displayed . The data editing apparatus according to claim 4 , wherein
The editing screen displayed in the predetermined display form based on the display data generated by the display control unit,
A first region displaying at least a portion of the text;
A second region for displaying the character string of the text divided into the plurality of blocks side by side on the time axis of the elapsed playback time for each block on the basis of the block information. Data editing device. The data editing device according to claim 5 ,
The display form of the text is:
A first form in which one character string is included in one block;
Any of the second form in which the character string included in one line of the text is included in one block and the third form in which the character string included in one page corresponding to the display area of the text is included in one block A data editing apparatus comprising one or more of them. The data editing device according to claim 6 ,
When one of the display forms of the text is set, the block information is updated corresponding to the set block. The data editing device according to claim 6 or 7 , wherein
When the first form or the second form is set as the display form of the text, a part of the text is displayed for each line in the first area,
When the third form is set as the text display form, a part of the text is displayed for each page in the first area. The data editing device according to any one of claims 5 to 8 ,
The data editing apparatus, wherein the editing screen displays information indicating a display area of the text or information indicating a display form of the text. The data editing device according to any one of claims 5 to 9 ,
When the display control unit cannot display all the character strings included in one block in the second area, the display control unit processes the display data to display a predetermined image or character string in the block, and A data editing apparatus for generating display data for displaying all the character strings on different screens when an image, a character string, or the block is selected. The data editing device according to any one of claims 5 to 10 ,
When the first area or the second area on the editing screen displayed in the predetermined display form based on the display data generated by the display control unit is selected, for the interlocking display included in each area A data editing apparatus for performing audio reproduction based on an elapsed reproduction time of data. The data editing device according to claim 6 ,
The interlocking display data has line information indicating the line number of the text displayed in the first area, attached to each of a character string composed of one or more characters,
The display control unit generates display data for displaying a line number indicated by line information corresponding to a character string displayed in the second area in the second area. . The data editing apparatus according to claim 12 , wherein
The row number displayed in the second area is displayed by a block, and when the width of each block is changed by an operation by the user of the data editing device, the data processing unit displays the linked display data. A data editing apparatus, wherein the line information included is processed. The data editing device according to claim 12 or 13 ,
In the second area, the same number of time axes as the number of lines of the first area are displayed, and different time numbers are assigned to the respective time axes. The data editing device according to claim 14 , wherein
When the temporal position of the text string is changed, the display control unit changes the value of the time axis affected by the change. The data editing device according to any one of claims 12 to 15 ,
The display control unit displays, in the second area, a page number derived from the line number indicated by the line information corresponding to the character string displayed in the second area and the number of lines in the first area. A data editing apparatus for generating display data for performing the operation. The data editing device according to claim 5 ,
The display control unit compares the size of the display area with the size of each character string composed of one or more characters of the interlocking display data, and stores the size in the first area and the second area. A data editing apparatus for generating display data for display. The data editing device according to any one of claims 4 to 17 ,
A determination unit that determines whether an instruction relating to editing of the linked display data is prohibited ;
The instruction relating to the editing of the prohibited linked display data is an instruction to change the order of blocks on the time axis, or an instruction to overlap a plurality of adjacent blocks on the time axis. apparatus. The data editing apparatus according to claim 18 , wherein
When the instruction relating to the editing of the interlocking display data is an instruction to overlap a plurality of blocks adjacent on the time axis,
Wherein the data processing unit, temporally preceding the end time or temporally preceding the time length and the temporally subsequent the interlocking display data while the start time was maintained at the same time the block of the block of the block A data editing device characterized by processing the data. The data editing device according to any one of claims 4 to 19 ,
The data editing apparatus according to claim 1, wherein when the size of the display area of the text is set, the block information is updated according to the size after the setting. The data editing device according to any one of claims 4 to 20,
When the text is paused while the text is being displayed in the predetermined form in accordance with the playback of the voice based on the interlocking display data, the data processing unit is configured to perform the predetermined elapsed time when the text is paused. of the start time or the end time or the time length of the block including the character string is displayed in a form, the data editing apparatus and changes to the playback elapsed time. The data editing device according to any one of claims 2 to 21 ,
The data processing unit includes time stamp information in which a value obtained by subtracting the start time from the end time included in the time stamp information or the time length included in the time stamp information is attached to a character string of 0. The data editing apparatus is characterized in that the linked display data is processed so that the time length included in the time stamp information of the character string is equal to or longer than a predetermined value. The data editing apparatus according to claim 22 , wherein
A character string in which a value obtained by subtracting the start time from the end time included in the time stamp information or the time length included in the time stamp information is a punctuation mark or a special character. Editing device. The data editing device according to any one of claims 1 to 23 , wherein:
The text editing area is set by the number of lines and the maximum number of bytes per line. The data editing device according to any one of claims 1 to 23 , wherein:
The text editing area is set by the number of lines and the maximum number of characters per line. A data editing apparatus according to any one of claims 1 to 25 , wherein
A data editing apparatus, comprising: a data generation unit configured to generate data for interlocking display of the partial range using the partial data of the audio reproduction time and the text data. A data editing apparatus according to any one of claims 1 to 26 ,
The display control unit generates display data based on the interlocking display data to which the reliability is given for each character string so that the display color or pattern of each character string varies depending on the reliability. apparatus. A data editing apparatus according to any one of claims 1 to 27 , wherein
The data processing unit uses a plurality of linked display data generated based on one audio data and text data corresponding thereto, and converts the linked display data in a specified range into another linked display data. A data editing device characterized by replacing data with the same range. A data editing program for causing a computer to function as each unit included in the data editing apparatus according to any one of claims 1 to 28 . A text based on the interlocking display data edited by the data editing device according to any one of claims 1 to 28 or the data editing program according to claim 29 is displayed at a predetermined timing in accordance with the reproduction of the sound. A reproducing apparatus. A text is displayed in accordance with the reproduction of voice based on the data for interlocking display edited by the data editing apparatus according to any one of claims 1 to 28 or the data editing program according to claim 29 , and the display is performed. A playback apparatus characterized in that playback is started from a character string by acquiring a user's selection location using a portion.

Images