JP6083654B2 - Data processing apparatus, data structure, data processing method, and program - Google Patents

Description

  The present invention relates to a data processing apparatus for processing data having display data.

  As a conventional technique, a character recognition unit that obtains a character code corresponding to each character image by performing character recognition processing on a plurality of character images in the document image, the document image, and the character recognition unit A plurality of character codes obtained, a plurality of types of character data to be used in common by a plurality of different character codes when a character corresponding to the plurality of character codes is drawn, and used for drawing the character code There has been known an image processing apparatus or the like having a generation means for generating an electronic document storing data indicating the type of character data (for example, see Patent Document 1).

Japanese Unexamined Patent Publication No. 2009-009527 (first page, FIG. 1 etc.)

  However, the conventional technology has a problem that it is not easy to appropriately process the display data associated with the sound data using the sound data.

  For example, in the conventional technique, when some processing is performed on display data using audio data associated with the display data, the display data and the audio corresponding to the display data are displayed. It is necessary to keep the correspondence with the data before and after the processing, and there is a problem that it is not easy to handle display data and audio data. Also, if the correspondence relationship is not known, the correspondence relationship between the display data and the audio data is not understood, and the display data is processed using the audio data associated with the display data. There was a problem that it was impossible.

  The present invention has been made to solve the above-described problems, and is a data process capable of appropriately processing, using audio data, display data associated with audio data. An object is to provide a device or the like.

  A data processing apparatus according to the present invention includes a data-with-speech storage unit that stores data with sound, which is data having display data that is display data and sound data that is sound data integrated, and data with sound. A data processing apparatus includes a processing unit that performs predetermined processing on data using audio data included in the data with audio, and an output unit that outputs a processing result of the processing unit.

  With such a configuration, the data with sound in which the sound data and the display data are integrated allows the correspondence between the sound data and the display data to be maintained. For the display data associated with the sound data, Appropriate processing can be performed using audio data.

  The data processing apparatus of the present invention further includes a voice receiving unit that receives voice in the data processing apparatus, and the processing unit is designated in advance for the data with voice using the voice received by the voice receiving unit. This is a data processing device that performs the above processing.

  With this configuration, appropriate processing can be performed using the received voice.

  In the data processing device according to the present invention, in the data processing device, the processing unit has a voice that matches the voice received by the voice receiving unit from the voice data included in the voice-added data stored in the voice-added data storage unit. The voice data including voice data including the voice data including the matching voice is retrieved, and the output unit is a data processing device that outputs the data with voice acquired by the processing unit.

  With this configuration, it is possible to appropriately search for data having display data associated with audio data using the received audio.

  In the data processing device of the present invention, in the data processing device, a display data storage unit that stores display data, a display unit that displays display data, and an input voice that receives sound for the display data displayed by the display unit A voice-attached data accumulating unit for accumulating in the voice-added data storage unit data with voice, which is a combination of the reception unit, the voice data of the voice received by the input voice reception unit, and the display data displayed by the display unit; Is a data processing apparatus.

  With this configuration, it is possible to create data with sound having sound data indicating the received sound.

  In the data processing device of the present invention, in the data processing device, the display data storage unit that stores display data having characters, and the display data stored in the display data storage unit use characters of the display data. Audio data having an audio data generation unit that generates audio data, display data stored in the display data storage unit, and audio data generated by the audio data generation unit for the display data, The data processing apparatus further includes a voice-attached data storage unit that stores data in the attached data storage unit.

  With this configuration, data with sound can be automatically created from display data.

  In the data processing device according to the present invention, in the data processing device, the voice data generation unit may generate voice data for a character string within a predetermined number of characters from the beginning for each character group including characters included in the display data. Is a data processing device for generating

  With this configuration, voice-attached data in which voice data is associated with each character group can be automatically created from display data.

  In the data processing device of the present invention, in the data processing device, an additional audio receiving unit that receives audio for one or more data with audio stored in the audio-added data storage unit, and audio received by the additional audio receiving unit Is a data processing device provided with a voice adding unit for adding the voice data to the voice data constituting the voice-attached data that has received the voice.

  With this configuration, audio data can be added.

  The data processing apparatus according to the present invention is a data processing apparatus according to the data processing apparatus, wherein the voice adding unit further adds an attribute value related to voice reception to the data with voice that has received voice.

  With this configuration, an attribute value related to reception of audio data can be added to the data with audio.

  According to the data processing apparatus and the like according to the present invention, it is possible to appropriately process the display data associated with the sound data using the sound data.

The block diagram of the data processor in an embodiment of the invention A flowchart for explaining the operation of the data processing apparatus The figure which shows an example of the external appearance of the data processing apparatus The figure which shows the data management table with sound of the same data processor The figure which shows the display data management table of the same data processor The figure which shows the example of a display of the same data processor Schematic diagram showing an example of the appearance of a computer that implements the data processing apparatus The figure which shows the internal structure of the computer which implement | achieves the data processing apparatus

  Hereinafter, embodiments of a data processing apparatus and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again.

(Embodiment)
FIG. 1 is a block diagram of a data processing apparatus 1 in the present embodiment.

  The data processing apparatus 1 includes an audio data storage unit 101, an audio reception unit 102, a processing unit 103, an output unit 104, a display data storage unit 105, a display unit 106, an input audio reception unit 107, an audio data generation unit 108, an audio A data storage unit 109, an additional voice receiving unit 110, and a voice adding unit 111 are provided.

  The data with sound storage unit 101 stores one or more data with sound. The data with sound is data having display data and sound data integrated. The data having the display data and the audio data integrated with each other is, for example, data having the display data and the audio data as one file. This may be considered that display data and audio data are stored in one file. The one data with sound may be one page such as an electronic book. Further, the one data with sound may be one object such as an object-oriented database. The audio data included in the data with audio is, for example, audio data for specifying display data. The audio data included in the data with audio is audio data for searching for display data. It does not matter how display data and audio data are stored in one file. For example, the audio data may be stored in a header, a footer, or the like included in the display data, and thus the display data in which the audio data is stored in the header may be considered as data with audio. One voice-attached data may include a plurality of text data. One piece of data with sound may have a plurality of image data.

  Display data is data for display. The display data is, for example, data that can be visualized. The display data is data that can be displayed on a monitor, for example. The display data is, for example, text data, image data, or a combination of these. The text data is, for example, data having one or more character codes indicating characters. The image data may be bitmap data or vector data. Bitmap data is also called raster data, for example. The image data may be image data indicating one or more characters, for example.

  The audio data is audio data. The voice data is data indicating a voice waveform, for example. For example, the voice data is data indicating a change in voltage indicating a voice waveform. The voice data may be data obtained by sampling a voice waveform. The audio data may be uncompressed data or compressed data. Further, this compression may be lossy compression or lossless compression. Moreover, the codec used at the time of compression is not ask | required.

  The data with sound may further have a feature amount of the sound data stored in the data with sound. The feature amount of the audio data will be described later. In addition, the voice-attached data is information obtained by breaking down the voice data stored in the voice-added data into phonemes, information further broken down into elements (hereinafter referred to as phonemes) than the phonemes, and encoded these. You may have information etc.

  The correspondence between display data and audio data in one display data may be, for example, one to one, one to many, or many to one. In the data with sound, the reproduction position of the sound data, the position (for example, coordinates) in the display data, one of a plurality of text data and image data constituting the display data, and one or more character strings constituting the text data Etc., or one or more character groups constituting the text data may be associated with each other.

  For example, when one piece of data with sound has one or two or more pages, one piece of sound data may be associated with one page of data with sound. The fact that the data with sound has a page may mean that it has the concept of a page. The page with data with sound may be considered as a page of display data included in the data with sound. Normally, one or more display data is arranged on one page of audio-attached data. However, one display data, for example, one text data may be arranged across two or more pages. When the data with sound includes a page, the display data may be considered as a display element arranged on the page, a so-called display object or the like.

  The audio data storage unit 101 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium. The same applies to other storage units.

  The voice reception unit 102 receives voice. The voice receiving unit 102 receives voice input via a microphone (not shown) or the like, for example. The voice received by the voice receiving unit 102 is, for example, a voice signal. For example, the voice receiving unit 102 acquires voice data indicating the received voice. For example, the voice reception unit 102 obtains voice data by sampling the received voice. The voice received by the voice receiving unit 102 may be considered as voice data. For example, the voice reception unit 102 may receive voice data as voice from another device, another configuration in the data processing device 1, or the like.

  The voice receiving unit 102 may or may not include an input unit such as a microphone for receiving voice. The voice reception unit 102 can be realized by a device driver of input means, control software for a menu screen, or the like.

  The processing unit 103 performs predetermined processing on the data with sound using the sound data included in the data with sound. The process designated in advance may be considered as a predetermined process. The process designated in advance by the processing unit 103 using the audio data may be any process. Further, the processing unit 103 does not matter how the audio data is used in the process designated in advance.

  For example, the processing unit 103 performs a process designated in advance on the data with voice using the voice received by the voice receiving unit 102. For example, the processing unit 103 uses the voice received by the voice receiving unit 102 to perform a search process on the data with voice.

  For example, the processing unit 103 searches the audio data included in the audio-attached data stored in the audio-added data storage unit 101 for audio data including the audio that matches the audio received by the audio accepting unit 102, and matches the audio The data with sound having the sound data including is acquired. Here, the matching voice may be considered as a voice that satisfies a predesignated condition for determining that they match, or may be considered as a voice that matches the voice received by the voice reception unit 102. The sound that matches is, for example, a sound whose matching level exceeds a predetermined threshold value.

  It does not matter how the processing unit 103 searches for voice data including voice that matches the voice received by the voice receiving unit 102. For example, the processing unit 103 may search for voice data including a voice that matches the voice received by the voice receiving unit 102 using the feature amount acquired for the voice and the voice data.

  For example, the processing unit 103 acquires one or more feature amounts specified in advance for the voice received by the voice receiving unit 102 and the voice data included in the voice-attached data, and the voice receiving unit 102 receives the feature amount. When the feature quantity acquired from the voice and the feature quantity acquired from the voice data included in the voice-attached data are collated, and the portion having the feature quantity whose fitness is equal to or greater than the threshold is included in the voice data included in the voice-attached data The data with sound is acquired as data with sound having sound data including sound that matches the sound received by the sound receiving unit 102.

  As described above, when the data with sound has the feature amount of the sound data included in the sound-added data as described above, the processing unit 103 calculates the feature amount using the sound data included in the sound-added data. Instead of acquiring (for example, extracting), a feature amount of audio data included in audio-attached data may be read. In addition, the feature amount of the voice or voice data may be acquired by the processing unit 103, or may be acquired by the voice receiving unit 102, the voice-attached data storage unit 109, or the like.

  The feature amount acquired with respect to speech is, for example, a sequence of feature vectors extracted from speech signals cut out every short time. The feature quantity acquired here is, for example, MFCC obtained by discrete cosine transform of a filter bank output of 24 channels using a triangular filter, and has 12 dimensions each of its static parameter, delta parameter, and delta delta parameter, It may also have normalized power and delta power and delta delta power (39 dimensions total). Alternatively, the feature quantity may be 25 dimensions including 12 dimensions of MFCC, 12 dimensions of ΔMFCC, and 1 dimension of Δlogarithmic power. As described above, various feature amounts can be used. The feature amount is also called a feature parameter or feature vector.

  Since the process of acquiring feature quantities for voice and voice data is a known technique, detailed description thereof is omitted here. Moreover, since the process which collates the feature-value acquired about audio | voice and audio | voice data, and acquires the value etc. which show a fitness is a well-known technique, detailed description is abbreviate | omitted here.

  Further, the processing unit 103 may search for voice data including voice that matches the voice received by the voice receiving unit 102, using information obtained by decomposing voice or voice data into phonemes or phonemes.

  For example, the processing unit 103 decomposes the voice received by the voice receiving unit 102 and the voice data included in the voice-attached data into phonemes and phoneme pieces and encodes them. And the information which decomposed | disassembled and encoded the audio | voice which the audio | voice reception part 102 received, and the information which decomposed | disassembled and encoded the audio | voice data which audio | voice data which audio | voice data have were collated, and a fitness is between the encoded information. If the audio data included in the audio-attached data includes a portion that is equal to or greater than the threshold, the audio-attached data is acquired as audio-attached data having audio data that includes audio that matches the audio received by the audio receiving unit 102.

  Note that, as described above, when the data with sound has phonemes or phonemes of the sound data included in the data with sound as described above, the processing unit 103 decomposes the sound data included in the data with sound. Instead of acquiring a phoneme or a phoneme piece, information such as a phoneme or a phoneme piece included in voice-attached data may be read. When the data with speech already has information obtained by encoding phonemes, phoneme pieces, etc., these encoded information may be read out. . Also, the processing for decomposing speech and speech data to obtain phonemes and phonemes may be executed by the processing unit 103, or may be executed by the speech receiving unit 102, the speech-added data storage unit 109, and the like. good.

  In addition, since the process which acquires a phoneme and a phoneme piece, and the process which searches using the data which encoded the phoneme and the phoneme piece are well-known techniques, detailed description is abbreviate | omitted here.

  Note that the processing unit 103 matches the voice received by the voice receiving unit 102 from the voice data associated with one or more pages when the voice-attached data is composed of a plurality of pages as described above. Voice data including voice may be searched, and one or more pages associated with voice data containing matching voice may be detected. Such processing can be realized by the same processing as the search processing described above except that the search target is a page associated with the voice data in the voice-attached data instead of the voice-attached data as a search target. is there.

  In addition, when the data with sound is composed of a plurality of display data as described above, the processing unit 103 converts the sound received by the sound receiving unit 102 from the sound data associated with one or more display data. It is also possible to search for audio data including the matching sound and detect display data associated with the sound data including the matching sound. Such processing can be realized by the same processing as the above search processing except that the display data associated with the voice data in the data with voice is searched instead of the data with voice as a search target. It is.

  Note that the processing unit 103 may perform a process other than the search process as described above as a process designated in advance using the audio data included in the data with audio. For example, as described above, the processing unit 103 displays the sound data stored in one sound-attached data and the display position of the sound data (for example, a time code indicating the reproduction time) within the same sound-added data. When the data is associated with text data or a part thereof, the audio data is reproduced by the output unit 104 described later, and the text data associated with the reproduced audio data (or the reproduction position) A part of the text data may be displayed on the output unit 104 or the like in a display mode different from other display data. Displaying in a different display mode means, for example, displaying text data characters, background color, or background pattern in a display color or background pattern different from other portions.

  The output unit 104 outputs the processing result of the processing unit 103. For example, the output unit 104 outputs the data with sound acquired by the processing unit 103 performing a previously designated process such as the search described above. Outputting data with sound means, for example, transmitting data with sound (specifically, a file with sound data) or storing it in a storage unit (not shown). In addition, outputting the data with sound includes, for example, displaying the display data of the data with sound, outputting the sound indicated by the sound data (for example, reproducing), or both. Also good. Further, outputting the data with sound may be considered to include outputting an identifier such as a file name of the data with sound.

  For example, the output unit 104 outputs the data with sound detected by the processing unit 103 by the search process described above. For example, the detected data with sound is transmitted or accumulated. In addition, the display data of the detected data with sound is displayed and the sound indicated by the sound data is output. Moreover, the display data of the page of the data with sound detected by the processing unit 103 by the search process may be displayed, or the sound indicated by the sound data of this page may be output. In addition, display data associated with voice data including voice that matches the voice received by the voice receiving unit 102 detected by the processing unit 103 through the search process may be displayed.

  Further, the output unit 104 outputs audio data with audio data according to the processing of the processing unit 103, and the output audio data and display data associated with the reproduction position are different from others. You may make it display in a display mode.

  Note that the output unit 104 may appropriately output the sound-added data stored in the sound-added data storage unit 101 in accordance with an instruction from the user or the like.

  Output here means display on a display, projection using a projector, sound output, transmission to an external device, storage in a recording medium, delivery of processing results to other processing devices or other programs, etc. It is a concept that includes

  The output unit 104 may or may not include an output device such as a display or a speaker. The output unit 104 can be implemented by output device driver software, or output device driver software and an output device.

  The display data storage unit 105 stores one or more display data. The display data storage unit 105 stores, for example, a display data file composed of one or more display data. The display data storage unit 105 stores display data having characters, for example. The display data having characters is, for example, text data or image data having a character image. The display data stored in the display data storage unit 105 is, for example, display data used for creating data with sound. Since the display data is the same as the display data included in the data with audio, description thereof is omitted here.

  The display unit 106 displays the display data stored in the display data storage unit 105. For example, the display unit 106 displays a file composed of one or more display data. When the display data has a plurality of pages, the display unit 106 may display one or more pages of a file composed of the display data.

  The display unit 106 may be considered as including a display device or not. The display unit 106 can be realized by display device driver software, or display device driver software and a display device.

  When the output unit 104 described above has a configuration for performing display or the like, the configuration for performing the display or the like of the output unit 104 may be used as the display unit 106. Further, a display device or the like used for display by the output unit 104 may be used by the display unit 106 for display.

  The input voice receiving unit 107 receives voice for display data displayed on the display unit 106. For example, the input voice receiving unit 107 receives voice for the display data displayed on the display unit 106. For example, when the display unit 106 displays a plurality of display data, the input voice reception unit 107 receives a voice for the focused display data. The focused display data is, for example, display data displayed on the forefront or display data designated by the user via an operation receiving unit (not shown). The plurality of display data may be a plurality of files having display data. Further, the focused display data may be a focused file.

  Note that the input voice receiving unit 107 may receive voice for a part specified by the user in one display data. For example, when the display data displayed by the display unit 106 is text data, voice may be received for a character string specified by the user of this data.

  The input voice receiving unit 107 receives voice input via a voice input device such as a microphone (not shown), for example. For example, when the user performs an operation for starting reception of voice to an operation reception unit (not shown), the input voice reception unit 107 may receive voice.

  The voice received by the input voice receiving unit 107 and the configuration and processing of the input voice receiving unit 107 are the same as the voice received by the voice receiving unit 102 and the configuration and processing of the voice receiving unit 102 described above. Detailed description is omitted.

  In the present embodiment, the voice receiving unit 102 described above may also be used as the input voice receiving unit 107.

  The voice data generation unit 108 generates voice data for the display data stored in the display data storage unit 105 using characters of the display data. When the display data includes characters, the audio data generation unit 108 generates audio data using the characters. For example, when the display data is text data, the characters included in the display data are characters constituting at least a part of the text data. Further, when the display data is image data, the characters included in the display data are characters indicated by at least a part of the image data. For example, the characters indicated by the bitmap image or the vector image indicating the outline of the characters. The character to be displayed.

  For example, when the display data is text data, the voice data generation unit 108 converts at least a part of the text data into voice data and generates voice data. Since the process of converting text data into audio data is a known technique, a detailed description thereof is omitted here.

  For example, when the display data is image data including a character image, the audio data generation unit 108 performs character recognition processing such as so-called OCR (Optical Character Recognition) on the character to acquire the text data. The voice data is generated by converting at least a part of the text data to voice data.

  The voice data generation unit 108 generates voice data for a character string within a predetermined number of characters from the beginning for each character group composed of characters included in the display data. For example, when the display data is text data, the character group composed of characters included in the display data is text data in sentences, paragraphs, pages, or text data in frames. Alternatively, the entire text data may be considered as a character group. When the display data is image data, for example, it is the same unit for characters constituting text data obtained by performing character recognition processing or the like from the image data.

  Note that the voice data generation unit 108 does not have to generate voice data for all character groups included in the display data. For example, the audio data generation unit 108 acquires a character string of the first designated number of characters from one or more pages of text data that is display data, and uses each of the acquired character strings for each page. Acquire corresponding audio data.

  In addition, for example, the audio data generation unit 108 acquires a character string within the first specified number of characters from each paragraph of text data that is display data, and corresponds to each paragraph using the acquired character string. Get audio data. The number of characters specified in advance is usually preferably 2 or more. In order to exclude conjuncts that frequently appear in text data such as “and” or “but”, however, three to six or more characters are required. Preferably it is the number of characters. The character string within the number of characters specified in advance is, for example, a character string within the number of characters specified in advance, from the beginning of each character group until the first or second punctuation or punctuation is entered. Preferably there is.

  There is no limitation on the timing or trigger at which the audio data generation unit 108 generates the audio data for the display data. For example, audio data may be generated when display data is accumulated in the display data storage unit 105, or may be generated when a user instructs generation.

  The data-with-sound storage unit 109 accumulates in the data-with-speech storage unit 101 data with sound, which is a combination of the sound data of the sound received by the input sound receiving unit 107 and the display data displayed by the display unit 106. To do. Specifically, a file with audio data having audio data and display data integrated is generated and stored. The voice-attached data storage unit 109 generates, for example, a voice-attached data file by adding the voice data to the display data, and stores the file with the voice-added data storage unit 101. The display data displayed on the display unit 106 here is, for example, the above-described focused display data. When the input voice receiving unit 107 receives voice for a part designated by the user in one display data, the voice-attached data storage unit 109 associates the voice data with the designated part of the display data, for example. Accumulate data with audio. In addition, when the input voice receiving unit 107 receives voice for one page in one display data, the voice-attached data storage unit 109 associates the voice data with this one page of display data, for example. Accumulate data with audio.

  Further, the data-with-sound storage unit 109 integrates sound-added data having the display data stored in the display data storage unit 105 and the sound data generated by the sound data generation unit 108 with respect to the display data. Accumulate in the data storage unit 101. As described above, when the voice data generation unit 108 acquires voice data using the first character string specified in advance for each character group specified in advance for text data as display data, the data with sound storage is performed. The unit 109 generates voice-attached data having the voice data acquired for each character group in association with the acquired character group, or in correspondence with the character string specified in advance at the beginning of the acquired character group. The data may be stored in the voice-attached data storage unit 101.

  The additional voice receiving unit 110 receives voice for one or more data with voice stored in the voice-added data storage unit 101. The sound received here is, for example, a sound used for adding sound data to the sound data included in the data with sound. For example, the additional voice receiving unit 110 receives voice for the data with voice specified by the user. For example, the additional voice receiving unit 110 may receive voice for the data with voice displayed by the output unit 104.

  The additional sound receiving unit 110 may receive sound for a part of display data that is already associated with sound data of the data with sound. For example, in a case where audio data is associated with one page of display data in the data with audio, audio may be received for this one page. For example, when one page of the display data in the data with audio associated with the audio data is displayed, the audio may be received for this page by receiving the audio.

  Note that the additional voice receiving unit 110 may receive voice for a part specified by the user in the display data included in the data with one voice. For example, when the display data of the data with voice displayed by the output unit 104 is text data, voice may be received for a character string designated by the user of this data.

  The additional sound receiving unit 110 receives sound input via an input device such as a microphone (not shown), for example. For example, when the user performs an operation for starting reception of voice to an operation reception unit (not shown), the input voice reception unit 107 may receive voice.

  The voice received by the additional voice receiving unit 110, the configuration and processing of the additional voice receiving unit 110, the voice received by the voice receiving unit 102 and the input voice receiving unit 107 described above, and the voice receiving unit 102 and the input voice receiving unit 107 The configuration and processing are the same, and detailed description thereof is omitted here.

  In the present embodiment, the voice receiving unit 102 and the input voice receiving unit 107 described above may also be used as the additional voice receiving unit 110.

  The audio adding unit 111 adds the audio data of the audio received by the additional audio receiving unit 110 to the audio data that constitutes the data with audio that has received the audio. For example, the voice adding unit 111 adds the received voice data to the end of the voice data included in the voice-attached data. This may be considered as adding audio data corresponding to the received audio as a continuation of the audio data already included in the audio-attached data.

  For example, when the sound adding unit 111 receives sound for sound data associated with a part of display data included in the data with sound, the sound adding unit 111 performs processing for the sound data associated with a part of the display data. Then, the voice data indicated by the received voice is added. For example, the audio data indicated by the received audio is added as a continuation of the end of the audio data associated with a part of the display data.

  For example, the voice adding unit 111 may further add an attribute value related to voice reception to the data with voice that the additional voice reception unit 110 has received voice. The attribute values related to voice reception include the date and time when the voice is received, the device identifier of the data processing device 1 that has received the voice, the user identifier of the user who operates the data processing device 1, and the like. The device identifier is a code such as a serial number assigned to the device, or address information such as a MAC address or an IP address. The user identifier is a user name, a code assigned to the user such as a user account, a user mail address, or the like. The date is acquired from, for example, a clock (not shown) or a network server (not shown), and is added to the date. In addition, device identifiers, user identifiers, and the like stored in advance in a storage unit (not shown) may be read as appropriate.

  As with the voice adding unit 111, when the data-with-sound storage unit 109 creates and stores data with sound, the device identifier, user identifier, and date of the data processing device 1 that creates the data with sound are set as the sound. You may make it accumulate | store in data with an audio | voice by matching with data.

  Next, an example of the operation of the data processing apparatus 1 will be described using the flowchart of FIG.

  (Step S <b> 101) The data processing apparatus 1 determines whether or not to perform a predesignated process using audio data on the audio data stored in the audio data storage unit 101. For example, when a reception unit (not shown) receives an instruction to perform processing, it determines to perform processing. For example, it is assumed here that the process designated in advance using voice data is a process of searching for voice-attached data associated with voice data including voice that matches the voice received by the voice receiving unit 102. When the process is performed, the process proceeds to step S102, and when the process is not performed, the process proceeds to step S105.

  (Step S102) The voice receiving unit 102 determines whether or not a voice is received. If accepted, the process proceeds to step S103, and if not accepted, the process returns to step S102.

  (Step S <b> 103) The processing unit 103 performs a process designated in advance on the data with voice using the voice received by the voice receiving unit 102. For example, data with sound having sound data including the same sound as the sound received by the sound receiving unit 102 is searched from the data with sound stored in the data with sound storing unit 101.

  (Step S104) The output unit 104 outputs the processing result in step S103. For example, the detected data with sound is displayed on a monitor (not shown). Then, the process returns to step S101.

  (Step S <b> 105) The display unit 106 determines whether to display the display data stored in the display data storage unit 105. For example, the display unit 106 displays the instructed display data when a reception unit (not shown) receives an instruction to display one or more display data stored in the display data storage unit 105 from a user or the like. Decide that. When displaying, it progresses to step S106, and when not displaying, it progresses to step S111.

  (Step S <b> 106) The display unit 106 displays one or more display data stored in the display data storage unit 105. For example, the display unit 106 displays the display data specified in step S106.

  (Step S107) The input voice receiving unit 107 determines whether or not to generate data with voice for the displayed display data. For example, when a reception unit (not shown) receives an instruction to generate data with sound for display data (for example, display data displayed on the front surface) currently focused by a user or the like, an instruction is given. It decides to generate data with sound for the displayed data. When it produces | generates, it progresses to step S108, and when not producing | generating, it progresses to step S110.

  (Step S108) The input voice receiving unit 107 determines whether or not a voice is received. If accepted, the process proceeds to step S109, and if not accepted, the process returns to step S108.

  (Step S109) The data-with-sound storage unit 109 generates data with sound including the one display data displayed at step S107 and the sound data indicating the sound received at step S108, and a data-with-sound storage unit 101. Then, the process returns to step S101.

  (Step S110) The display unit 106 determines whether or not to end the display. For example, it is determined whether a reception unit (not shown) has received an instruction to end the display. If accepted, display is terminated and the process returns to step S101. If not accepted, the process returns to step S107.

  (Step S <b> 111) The audio data generation unit 108 determines whether to generate audio data from the display data stored in the display data storage unit 105. For example, when a reception unit (not shown) receives an instruction to generate audio data for one or more display data stored in the display data storage unit 105, the reception unit determines to generate audio data. If sound data is to be generated, the process proceeds to step S112, and if not, the process proceeds to step S114.

  (Step S <b> 112) The audio data generation unit 108 generates audio data for the display data stored in the display data storage unit 105. For example, audio data is generated for each of the one or more display data specified in step S111.

  (Step S113) In step S112, the voice-attached data storage unit 109 generates voice data generated for the display data and voice-added data including the display data, and stores the data in the voice-added data storage unit 101. Then, the process returns to step S101.

  (Step S114) The additional voice receiving unit 110 determines whether or not to add voice data to the voice-added data stored in the voice-added data storage unit 101. For example, when a reception unit (not shown) receives an instruction to add audio data to one of the audio data stored in the audio data storage unit 101, the audio data is added. decide. If additional audio data is to be added, the process proceeds to step S115, and otherwise, the process returns to step S101.

  (Step S115) The additional voice receiving unit 110 determines whether a voice has been received. If accepted, the process proceeds to step S116. If not accepted, the process returns to step S115.

  (Step S116) The voice adding unit 111 adds the voice data indicating the voice received in step S115 to the voice data included in the voice-attached data. For example, the audio data is added to the audio data included in the one audio-attached data designated in step S114.

  (Step S117) The voice adding unit 111 adds an attribute value designated in advance to the voice-attached data in which the voice data is added in step S116. Then, the process returns to step S101.

  In the flowchart of FIG. 2, the process is terminated by powering off or a process termination interrupt.

  Hereinafter, a specific operation of the data processing apparatus 1 in the present embodiment will be described. Here, it is assumed that the processing performed by the processing unit 103 is a search for data with voice using the voice received by the voice receiving unit 102.

  FIG. 3 is a diagram illustrating an appearance of an example of the data processing apparatus 1 according to the present embodiment. In the present embodiment, a case where the data processing device 1 is a so-called tablet terminal will be described as an example. The data processing device 1 includes a microphone 102a and a monitor 104a. Further, it is assumed that a touch panel (not shown) is provided on the monitor 104a.

  FIG. 4 is a voice-attached data management table for managing voice-attached data stored in the voice-added data storage unit 101. The data management table with sound has attributes of “ID with sound” and “data with sound”. “ID with sound” is an identifier of data with sound, and here it is assumed to be a file name. “Data with sound” is data with sound, and here is a file of data with sound.

  First, suppose that the user displays an audio input screen (not shown) or the like on the monitor 104a and performs an operation for starting input of audio in order to search for data with audio by voice. Then, it is assumed that the user generates a sound “third wave” toward the microphone 102a (not shown). The voice receiving unit 102 receives a voice called “third wave” (here, a voice signal) via the microphone 102a. The voice reception unit 102 performs sampling processing or the like on the received voice signal and acquires voice data indicating the voice “third wave”. Here, for the sake of explanation, this audio data is referred to as acquired audio data.

  The processing unit 103 performs a process of searching for voice data including voice that matches the voice received by the voice receiving unit 102 from the voice data included in the voice-added data stored in the voice-added data storage unit 101 as follows. To do.

  First, the processing unit 103 acquires a plurality of feature amounts specified in advance for the audio data acquired by the audio receiving unit 102. The acquired feature amount is temporarily stored in a storage unit (not shown).

  The processing unit 103 reads the voice-attached data associated with the voice-added ID “001” from the first record (row) on the voice-attached data management table shown in FIG. Hereinafter, for convenience of explanation, the data with sound associated with the sound ID “001” is referred to as data with sound 001. The same applies to voice-attached data associated with other voice-attached IDs.

  The processing unit 103 also acquires a plurality of predesignated feature amounts similar to the above for the audio data included in the read audio-added data 001. Then, the feature amount acquired from the acquired speech data acquired above and the feature amount acquired for the speech data of the speech-added data 001 are collated, and the portion having the feature amount with the matching degree equal to or greater than the threshold is the speech-attached data. It is determined whether or not the audio data included in 001 is included. Here, assuming that at least one portion having a feature amount whose fitness is equal to or greater than a threshold is detected, the processing unit 103 includes the voice-added data 001 including voice that matches the voice received by the voice receiving unit 102. Acquired as audio data having audio data.

  Similarly, the processing unit 103 also processes data with sound acquired from the second and subsequent records (rows) from the top of the data with sound management table shown in FIG. 4, for example, data with sound 002 and data with sound 003. Then, the same processing as described above is performed, and it is determined whether or not a portion having a feature amount whose fitness is equal to or greater than a threshold value is included in the audio data included in the audio data 002, the audio data 003, and the like. Then, if included, the data with sound determined to be included is acquired as data with sound having sound data including sound that matches the sound received by the sound receiving unit 102, and if not included, It is determined that the voice data has voice data that does not include the matching voice, and the data with voice is not acquired.

  Here, for example, as a result of performing the above processing on all the data with sound in the data with sound management table shown in FIG. 4, the processing unit 103 receives only the sound-added data 001 as the sound received by the sound receiving unit 102. Suppose that it has been acquired as voice-attached data having voice data including voice that matches the above.

  The output unit 104 displays the sound-added data 001 acquired by the processing unit 103 on the monitor 104a. Specifically, the display data included in the audio-added data 001 is displayed on the monitor 104a. Here, for example, if the display data of the audio-added data 001 is data having a plurality of pages, the output unit 104 displays one page of display data, for example, the first page.

  Then, it is assumed that the user operates the menu screen or the like to end the display of the display data included in the voice-attached data 001.

  FIG. 5 is a display data management table for managing display data stored in the display data storage unit 105. The display data management table has attributes of “display ID” and “display data”. “Display ID” is an identifier of display data, and here is a file name. “Display data” is display data, and here is a display data file. Here, the display data is data having text data.

  Next, in order to generate data with sound using the display data stored in the display data storage unit 105, the user instructs to display one display data stored in the display data storage unit 105. It is assumed that a menu (not shown) is operated and given to a reception unit (not shown). Here, it is assumed that an instruction to display display data whose “display ID” is “D02” is given. The display unit 106 displays the display data D02 on the monitor 104a in response to the above instruction. The display data whose “display ID” is “D02” is hereinafter referred to as display data D02. The same applies to other display data.

  Here, the monitor 104a used by the output unit 104 is also used for display by the display unit 106.

  It is assumed that the user performs an operation for starting voice input by displaying a voice input interface or the like (not shown) on the monitor 104a in order to create data with voice using the displayed display data D02. Then, if the user generates a voice such as “Seattle-based ...”, which has been mainstream until now, on the microphone 102a, the input voice reception unit 107 receives the voice input via the microphone 102a. (In this case, audio signals) are sequentially received.

  FIG. 6 is a diagram illustrating a state in which the input voice receiving unit 107 receives voice for the display data displayed on the display unit 106. In the figure, a voice input interface 60 for receiving voice input includes a button 61 for starting voice input, a bar 62 indicating recording time, and the like.

  The voice reception unit 102 performs sampling processing or the like on the received voice signal, and acquires voice data indicating the input voice.

  Here, the microphone 102a used by the voice receiving unit 102 is also used by the input voice receiving unit 107 and the additional voice receiving unit 110.

  The data-with-sound storage unit 109 generates data with sound having the sound data received by the input sound receiving unit 107 and the display data D02 being displayed, and stores the data with sound storage unit 101. When accumulating, according to a rule or the like specified in advance, the above “ID with audio” is acquired and stored in the audio data storage unit 101 in association with the acquired “ID with audio”. The rules designated in advance are, for example, rules for assigning sequential numbers.

  Further, the user designates one display data stored in the display data storage unit 105, for example, display data D03, automatically generates audio data using the display data D03, and generates data with audio. When an instruction for automatic generation is given to the data processing apparatus 1, the audio data generation unit 108 reads the display data D 03 from the display data storage unit 105. Here, it is assumed that the display data D03 is data having a plurality of pages. The voice data generation unit 108 converts the first 20 characters of the text data included in the read display data D03 into voice data. For example, if the first 20 characters of the text data on the first page is a character string “how to put delicious coffee is the right appliance”, the voice data generation unit 108 acquires this character string, The character string is converted into voice data indicating the voice read out. Similarly, voice data of the first 20 characters is acquired for other pages.

  Then, the data-with-sound storage unit 109 acquires display data D03 and the data with sound having the sound data acquired for each page of the display data D03 in association with each page, and the acquired data with sound is stored in the above-described manner. In the same manner as described above, the data is stored in the voice-attached data storage unit 101 in association with the “sound-added ID”. Having audio data associated with each page means that, for example, the audio data acquired for each page is stored in association with the page number of each page, or the audio acquired for each page. For example, data indicating the start of each page is associated with information indicating the position at the beginning of each page of the audio data combined with the data (for example, the reproduction start position).

  The user wants to add voice to the voice-attached data stored in the voice-attached data storage unit 101, and the user designates the voice-added data 003 to be added and inputs the voice. Assume that the user generates a voice to be added by displaying a voice input interface (not shown).

  The additional voice receiving unit 110 receives the input voice via the microphone 102a and acquires voice data from the received voice. The voice adding unit 111 reads the voice-attached data from the voice-added data storage unit 101, and adds the acquired voice data to the existing voice data included in the read voice-added data 003. In addition, the voice adding unit 111 acquires the current date and time from a clock (not shown) or the like, and accumulates data indicating the date and time in the voice-added data 003 as an attribute value indicating the date and time when the audio data is received. The voice adding unit 111 reads the device identifier and the user identifier of the data processing device 1 stored in advance in a storage unit or the like (not shown), and indicates the device and user that received the voice data to be additionally written. The attribute value is stored in the voice-added data 003.

  The additional voice receiving unit 110 adds voice data, and further adds the voice-attached data 003 stored in the voice-added data storage unit 101 with the voice-added data 003 accumulating the date and time, the device identifier, and the user identifier. Overwrite. As a result, the voice data is added to the voice-attached data 003.

  If the audio data of the audio-added data 003 is stored in association with the page of audio-added data, the additional audio data is added to the audio data associated with the page specified by the user or the like. Alternatively, it may be added to the audio data designated by default or the like.

  As described above, according to the present embodiment, it is possible to perform a process designated in advance using audio data included in the audio-attached data for the audio-attached data having the audio data and the display data integrated. Therefore, the display data associated with the audio data can be appropriately processed using the audio data.

  In the above embodiment, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.

  In the above-described embodiment, the case where the data processing apparatus is a stand-alone has been described. However, the data processing apparatus may be a stand-alone apparatus or a server apparatus in a server / client system. In the latter case, the output unit or the reception unit receives an input or outputs a screen via a communication line.

  In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. At the time of execution, the program execution unit may execute the program while accessing a storage unit (for example, a recording medium such as a hard disk or a memory).

  The software that realizes the data processing apparatus in the above embodiment is the following program. In other words, this program is a computer that can access an audio data storage unit in which audio data, which is data having display data as display data and audio data integrated with each other, is stored. This is a program for functioning as a processing unit that performs pre-designated processing on voice-attached data using voice data included in the voice-attached data, and an output unit that outputs a processing result of the processing unit.

  In the program, the functions realized by the program do not include functions that can be realized only by hardware. For example, a function that can be realized only by hardware such as a modem or an interface card in an acquisition unit that acquires information or an output unit that outputs information is not included in the function realized by the program.

  Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  In addition, the data structure of the data with sound realized in the present embodiment is the following data with sound. That is, the data structure of the data with sound is a data structure of data with sound used in a data processing apparatus having a processing unit and an output unit, and the data structure is display data that is display data, Audio data that is audio data is integrated and the processing unit performs pre-designated processing on the audio data with the audio data included in the audio data, and the output unit It is a data structure which outputs the process result of the said process part.

  FIG. 7 is a schematic diagram showing an example of the appearance of a computer that executes the program and realizes the data processing apparatus according to the embodiment. The above-described embodiment can be realized by computer hardware and a computer program executed on the computer hardware.

  In FIG. 7, a computer system 900 includes a computer 901 including a CD-ROM (Compact Disk Read Only Memory) drive 905, a keyboard 902, a mouse 903, and a monitor 904.

  FIG. 8 is a diagram showing an internal configuration of the computer system 900. In FIG. 8, in addition to the CD-ROM drive 905, a computer 901 is connected to an MPU (Micro Processing Unit) 911, a ROM 912 for storing a program such as a bootup program, and the MPU 911, and receives instructions of an application program. A RAM (Random Access Memory) 913 that temporarily stores and provides a temporary storage space, a hard disk 914 that stores application programs, system programs, and data, and a bus 915 that interconnects the MPU 911, ROM 912, and the like Prepare. The computer 901 may include a network card (not shown) that provides connection to the LAN.

  A program that causes the computer system 900 to execute the functions of the data processing apparatus and the like according to the above-described embodiments may be stored in the CD-ROM 921, inserted into the CD-ROM drive 905, and transferred to the hard disk 914. Instead, the program may be transmitted to the computer 901 via a network (not shown) and stored in the hard disk 914. The program is loaded into the RAM 913 when executed. The program may be loaded directly from the CD-ROM 921 or the network.

  The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 901 to execute the functions of the data processing device according to the above-described embodiment. The program may include only a part of an instruction that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 900 operates is well known and will not be described in detail.

  The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the data processing device according to the present invention is suitable as a device for processing data having display data, and in particular, processes for data having display data using audio data. It is useful as a device to perform.

DESCRIPTION OF SYMBOLS 1 Data processing apparatus 101 Data storage part with audio | voice 102 Audio | voice reception part 102a Microphone 103 Processing part 104 Output part 104a Monitor 105 Display data storage part 106 Display part 107 Input audio | voice reception part 108 Audio | voice data generation part 109 Voice data storage part 110 Addition Voice reception part 111 Voice addition part

Claims (6)

A display data storage unit that stores display data that is display data having characters;
A data-with-speech storage unit that stores data with sound, which is data having display data and sound data that is sound data integrated;
About the display data stored in the display data storage unit, using the characters that the display data has, an audio data generation unit that generates audio data;
Audio-attached data that accumulates in the data-with-sound storage unit audio-attached data having the display data stored in the display data storage unit and the sound data generated by the sound data generation unit with respect to the display data. A data storage unit ;
A voice reception unit for receiving voice;
Voice data including voice that matches the voice received by the voice receiving unit is searched from voice data included in the voice-attached data stored in the voice-added data storage unit, and voice having voice data including the matching voice is searched. A processing unit for acquiring attached data;
A data processing apparatus comprising: an output unit configured to display display data included in the data with sound acquired by the processing unit or to output sound represented by the sound data included in the data with sound . The audio data generation unit, for each character group composed of the characters included in the display data, the data processing apparatus according to claim 1, wherein generating the audio data for a character string of up number of characters specified in advance from the beginning. An additional voice receiving unit that receives voice for one or more data with voice stored in the data-with-sound storage unit;
The add audio sound of the audio data accepting unit accepts data processing apparatus according to claim 1 or claim 2, wherein with a sound adding section that appended to the voice data forming the data with sound that has received the audio. The data processing apparatus according to claim 3 , wherein the voice adding unit further adds an attribute value related to reception of the voice to the data with voice that has received the voice. Data with sound for storing data with sound, which is data having a display data storage unit for storing display data, which is display data having characters, and sound data, which is display data and sound data, integrated An information processing method performed using a storage unit, a voice data generation unit, a voice-attached data storage unit, a voice reception unit, a processing unit, and an output unit ,
A voice data generation step in which the voice data generation unit generates voice data for the display data stored in the display data storage unit, using characters of the display data;
The voice-attached data storage unit integrates the voice-attached data having the display data stored in the display data storage unit and the voice data generated by the voice data generation unit for the display data. A data accumulating step with sound to be accumulated in the data storage unit ;
A voice receiving step in which the voice receiving unit receives voice;
The processing unit searches the audio data included in the audio-attached data stored in the audio-added data storage unit for audio data including the audio that matches the audio accepted in the audio accepting step, and includes the matching audio A processing step of acquiring voice-attached data having voice data;
An information processing method comprising: an output step in which the output unit displays display data included in the data with sound acquired in the processing step or outputs sound indicated by the sound data included in the data with sound . A display data storage unit for storing display data that is display data having characters, and a data with sound for storing data with sound that is data having display data and sound data that is sound data integrated. A data storage and a computer accessible to the
About the display data stored in the display data storage unit, using the characters that the display data has, an audio data generation unit that generates audio data;
Audio-attached data that accumulates in the data-with-sound storage unit audio-attached data having the display data stored in the display data storage unit and the sound data generated by the sound data generation unit with respect to the display data. A data storage unit ;
A voice reception unit for receiving voice;
Voice data including voice that matches the voice received by the voice receiving unit is searched from voice data included in the voice-attached data stored in the voice-added data storage unit, and voice having voice data including the matching voice is searched. A processing unit for acquiring attached data;
Program for operating as an output unit for outputting a voice represented by voice data display of the display data with sound data to which the processing unit has acquired has, or the audio-data has.

Images