JP2006127443A - E-mail transmitting terminal and e-mail system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an e-mail system with richer expression using voice. <P>SOLUTION: The voice inputted from a microphone is decomposed into recording data elements V1, V2, V3 by noise gate processing. Timing markers by every bar are set in BGM data (musical piece data with marker) and each of the recording data elements V1, V2, V3 is combined at head positions of each bar of the BGM data. In addition, sound effect data is combined in optional timing according to a user specification operation. The combined sound data generated in this way is transmitted as voice e-mail after compression processing is performed to it in a compression form such as an MP3, for example. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for creating and sending e-mail content.

  E-mail using a mobile phone terminal has an advantage that messages can be exchanged easily without selecting time and place. In e-mail using a mobile phone terminal, in addition to text characters, pictograms and the like for the purpose of enhancing expressiveness are used. Recently, along with the enhancement of functions of mobile phone terminals, it is also possible to exchange messages including still images and moving images.

  In this way, in mobile phone mail that is routinely used as a means of easy communication, users seek different expressions from others, exchange emails full of originality using pictograms and emoticons, or Exchanges emails using images. However, there are few e-mail services using audio media, and many e-mail services are aimed at visual effects.

  As an example of a mail service using audio media, “singing ♪ mail” of Non-Patent Document 1 below can be cited. This uses an application program to convert an input kana character or the like into a singing voice. The arrangement of voice quality, melody and accompaniment can be freely changed.

http://www.g-search.or.jp/release/2004/20040726.html

  In e-mail using a mobile phone terminal, diversity of expressiveness for messages is further demanded. While mobile phones play a major role as one of the means of communication, it is significant to enhance the expressiveness of e-mail.

  In the said nonpatent literature 1, although the input character is converted into an artificial voice using an application program, it is difficult to express an accent and emotion peculiar to a person with an artificial voice.

  Accordingly, an object of the present invention is to provide an expressive e-mail system using audio media.

  In order to solve the above-mentioned problem, the invention described in claim 1 is characterized in that voice input means, means for recording the voice input by the voice input means and storing it as recorded data, and synthesizing the recorded data and music data. And a synthesis processing means for generating synthesized voice data and a sending means for sending the synthesized voice data as an e-mail.

  According to a second aspect of the present invention, in the electronic mail transmitting terminal according to the first aspect, the music data includes sound effect data, and the synthesis processing means is designated on a reproduction time axis of the synthesized voice data. Means for synthesizing the sound effect data at a point.

  According to a third aspect of the present invention, in the electronic mail transmitting terminal according to the first or second aspect, the music data can be connected to a recording medium removable from the electronic mail transmitting terminal or the electronic mail transmitting terminal. It is stored in a storage device of a server on the network.

  According to a fourth aspect of the present invention, in the electronic mail transmitting terminal according to the third aspect, the music data stored in the recording medium or the storage device is encrypted, and the composition processing means is encrypted. Means for decoding the music data that has been recorded.

  According to a fifth aspect of the present invention, in the electronic mail transmitting terminal according to any one of the first to fourth aspects, the music data is a marker in which one or more timing markers are set in advance on the reproduction time axis. The composition processing means includes a portion below the predetermined volume of the recording data as a silent portion, and decomposes the recording data into a plurality of recording data elements separated by a silent portion; and Means for synthesizing each of a plurality of recording data elements in synchronism with the time set by the timing marker of the music data with marker.

  According to a sixth aspect of the present invention, in the electronic mail transmitting terminal according to the fifth aspect, the music data with a marker includes music data in which a timing marker is set at the head of each measure.

  According to a seventh aspect of the present invention, in the electronic mail sending terminal according to any one of the first to sixth aspects, the synthesizing processing unit responds to a designation operation by a user on the time axis of the synthetic voice data. Means for synthesizing video data at an arbitrary point and generating synthesized audio data with video data, and the transmission means includes means for transmitting the synthesized audio data with video data as an e-mail. To do.

  According to an eighth aspect of the present invention, in the electronic mail transmitting terminal according to any one of the first to sixth aspects, the electronic mail transmission terminal further comprises means for compressing and converting the synthesized voice data.

  According to a ninth aspect of the present invention, in the electronic mail transmitting terminal according to the seventh aspect, the electronic mail transmitting terminal further comprises means for compressing and converting the synthesized audio data with video data.

  According to a tenth aspect of the present invention, in the electronic mail transmitting terminal according to any one of the first to sixth aspects, the electronic mail transmitting terminal is a mobile phone terminal, and the synthesized voice data is defined in the mobile phone terminal. It is converted to a standard music data format.

  An eleventh aspect of the present invention is the electronic mail transmitting terminal according to any one of the first to tenth aspects, further comprising means for generating scenario data in which a rule for the synthesis process is recorded, wherein the transmitting means And means for transmitting the recording data and the scenario data as an electronic mail.

  A twelfth aspect of the present invention is the electronic mail transmitting terminal according to any one of the first to eleventh aspects, further comprising processing means for processing the voice input by the voice input means, wherein the processing means is Means for modulating the sound and / or means for giving a special effect to the sound, wherein the synthesis processing means synthesizes the recording data and the music data after being processed by the processing means.

  According to a thirteenth aspect of the present invention, in the electronic mail transmitting terminal according to the twelfth aspect, the processing means includes means for executing a tempo change process and / or a pitch shift process of the voice input by the voice input means. It is characterized by including.

  According to a fourteenth aspect of the present invention, in the electronic mail transmitting terminal according to the twelfth aspect, the processing means applies any one of equalizer processing, harmonization processing, and echo processing to the voice input by the voice input means. Means for executing one or more processes.

  A fifteenth aspect of the present invention is the electronic mail transmitting terminal according to any one of the twelfth to fourteenth aspects, wherein a plurality of setting information corresponding to a plurality of themes is prepared in advance, The content of the processing executed by the processing means is defined, and the content of the processing by the processing means is determined by selecting one setting information.

  According to a sixteenth aspect of the present invention, in the electronic mail transmitting terminal according to any one of the first to fifteenth aspects, the music data defined by the music data and / or the music data is further defined. Means for giving special effects to music.

  The invention according to claim 17 is a system for transferring electronic mail, comprising a terminal and a synthesis server, wherein the terminal records voice input means and voice input by the voice input means, Means for storing as data, and means for transmitting the recording data to the synthesis server, wherein the synthesis server synthesizes the received recording data and music data and generates synthesized voice data. And means for storing the synthesized voice data as an e-mail in a storage device.

  The invention according to claim 18 is a system for transferring an electronic mail, comprising a terminal and a synthesis server, wherein the terminal inputs voice input means and voice input by the voice input means to the synthesis server. Means for transmitting, wherein the synthesis server records the received voice as recorded data, synthesis processing means for synthesizing the recorded data and music data, and generating synthesized voice data, and the synthesized voice Means for storing data in a storage device as electronic mail.

  According to a nineteenth aspect of the present invention, in the electronic mail system according to the seventeenth or eighteenth aspect, the music data includes sound effect data, and the synthesis processing means is on a reproduction time axis of the synthesized voice data. Means for synthesizing sound effect data at designated points.

  According to a twentieth aspect of the present invention, in the electronic mail system according to any one of the seventeenth to nineteenth aspects, the music data has a marker in which one or more timing markers are set in advance on a reproduction time axis. The synthesizing processing means includes a means for decomposing the recording data into a plurality of recording data elements having a portion below a predetermined volume as a silence portion and separating the recording data into a silence portion as a delimiter; And means for synthesizing each of the recording data elements in synchronism with the time set by the timing marker of the music data with the marker.

  The invention according to claim 21 is the electronic mail system according to claim 20, wherein the music data with a marker includes music data in which a timing marker is set at the head of each measure.

  According to a twenty-second aspect of the present invention, in the electronic mail system according to any one of the seventeenth to twenty-first aspects, an image is displayed at an arbitrary point on the time axis of the synthesized voice data in response to a designation operation by a user. Means for synthesizing data and generating synthesized audio data with video data.

  A twenty-third aspect of the present invention is the electronic mail system according to any one of the seventeenth to twenty-first aspects, further comprising means for compressing and converting the synthesized voice data.

The invention according to claim 24 is the electronic mail system according to claim 22, further comprising:
Means for compressing and converting the synthesized audio data with video data into a moving image.

  According to a twenty-fifth aspect of the present invention, in the electronic mail system according to any one of the seventeenth to twenty-first aspects, the terminal is a mobile phone terminal, and the synthesized voice data is a standard defined in the mobile phone terminal. It is converted into a music data format.

  According to a twenty-sixth aspect of the present invention, in the electronic mail system according to any one of the seventeenth to twenty-fifth aspects, the synthesizing server further includes processing means for processing a voice received from the terminal. The means includes means for modulating sound and / or means for giving a special effect to the sound, and the composition processing means synthesizes the recording data and the music data after being processed by the processing means. To do.

  According to a twenty-seventh aspect of the present invention, in the electronic mail system according to the twenty-sixth aspect, the processing means includes means for executing a tempo change process and / or a pitch shift process of the voice input by the voice input means. It is characterized by that.

  According to a twenty-eighth aspect of the present invention, in the electronic mail system according to the twenty-sixth aspect, the processing means applies any one of equalizer processing, harmonization processing, and echo processing to the voice input by the voice input means. Alternatively, it includes means for executing a plurality of processes.

  According to a twenty-ninth aspect of the present invention, in the electronic mail system according to any one of the twenty-sixth to twenty-eighth aspects, a plurality of setting information corresponding to a plurality of themes is prepared in advance, The content of the processing executed by the processing means is defined, and the content of the processing by the processing means is determined by selecting one setting information.

  A thirty-third aspect of the invention is the electronic mail system according to any one of the seventeenth to thirty-ninth aspects, further comprising means for modulating music defined by the music data and / or the music data. Means for giving special effects to music.

  The present invention not only reproduces recorded voices and music files on a terminal such as a mobile phone, but also synthesizes them by superimposing their own voice on music. Thereby, it is possible to create a synthesized voice mail full of originality.

  In addition, since the music data stored in the recording medium is encrypted and needs to be read by a predetermined program for decryption, it is possible to prevent unauthorized use of content.

  Further, when synthesizing the recording data and the music data, it is possible to synthesize the recording data without a sense of incongruity by adding timing marker information to the music data. In addition, by performing noise gate processing during or after recording, it is possible to create a content that is easy to hear in terms of music by eliminating outdoor environmental noise.

  In addition, by inserting video data into the synthesized audio data, it is possible to generate multimedia mail with more originality.

  Further, by converting the synthesized audio data into an audio compression file such as AAC / MP3 or a moving image compression file such as MPEG4, it is possible to generate a low-capacity and high-quality content.

  In addition, by converting the synthesized voice data into a general-purpose music data format standardized by a mobile phone terminal, it can be provided as a versatile mail system that extends across mobile phone carriers regardless of the playback environment.

  In addition, since voice data is subjected to various processes and then synthesized with music data, voice mail with more expressiveness can be created.

{First embodiment}
<Configuration of mobile phone device>
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a mobile phone terminal 100 according to the first embodiment of the present invention. The mobile phone terminal 100 includes a microphone device 101 for inputting voice, a microphone interface (Mic I / F) 102, a ROM card 103 as a card medium, and a card control interface (Card Cont I / F) 104 for accessing the card medium. A memory 106 used as a temporary storage area for various data or a storage area for various application programs, a memory management unit (MMU) 105 that controls signals between the CPU 112 and the memory 106, and various user operations on the mobile phone device 100. An input operation unit 107, an audio processing unit 108 for encoding and decoding audio signals, a video processing unit 109 for encoding and decoding video signals, and the mobile phone terminal 100 When performing a voice call and when the mobile phone terminal 100 performs data communication, a communication unit 110 that executes communication processing with the base station via the antenna 111 and a CPU 112 that controls the mobile phone terminal 100 are provided. Yes. In addition, the mobile phone terminal 100 includes a monitor 113 and a speaker 114.

  As the ROM card 103, for example, a compact flash (registered trademark) card, smart media (registered trademark), SD memory card (registered trademark), or the like can be used. As the memory 106, an SDRAM or the like can be used. The audio processing unit 108 has a function of encoding and decoding audio signals based on standards such as MP3 and AAC, and the video processing unit 109 has a function of encoding and decoding video signals based on standards such as MPEG4. It has.

  The ROM card 103 stores BGM data BD and sound effect data ED. The BGM data BD is background music data to be synthesized with the voice input from the microphone in voice electronic mail (voice mail) or multimedia mail transmitted in the present embodiment. The BGM data BD is preferably data without vocal voice in terms of making it easy to hear the content of the voice, but is not limited to such data, and various music data including music with vocals. Is available. The sound effect data ED is data for adding a sound effect of a relatively short time to the synthesized sound data obtained by synthesizing the sound input from the microphone and the BGM data BD. For example, sound data such as clapping, shout, and cymbal are included.

  The BGM data BD and the sound effect data ED are stored in the ROM card 103 in a file format compressed by, for example, MP3 or AAC standards. Alternatively, it is stored in the ROM card 103 in a file format such as a ringing melody format that is standard in mobile phone terminals.

  The BGM data BD and sound effect data ED stored in the ROM card 103 are encrypted and decrypted by the authoring program AP stored in the memory 106. The authoring program AP has decryption key information necessary for decrypting the BGM data BD and the sound effect data ED, and BGM and sound effects can be reproduced by decrypting these data.

  The authoring program AP stored in the memory 106 can execute various functions for creating a voice electronic mail or multimedia mail in the present embodiment, including a voice synthesis process. Specifically, the authoring program AP has a function for recording voice input from the microphone device 101, a process for reading and decoding BGM data BD and sound effect data ED stored in the ROM card 103, and a microphone input voice. Noise gate processing function during or after recording, synthesized processing function of recorded voice data and BGM data BD, synthesized processing function of synthesized voice data and sound effect data ED, synthesized process of synthesized voice data and video data Function, saving function of synthesized voice data with synthesized voice data and video data, playback function of synthesized voice data with synthesized voice data and video data, function to send synthesized voice data and synthesized voice data with video data to other terminals as e-mail , Etc. are executable. The above-described functions executed by the authoring program AP are functions that are realized by the authoring program AP using hardware resources such as the CPU 112 and RAM (not shown).

<Noise gate function and BGM data synthesis function>
Next, the noise gate function and the BGM data synthesis function will be described with reference to FIG. As described above, the noise gate function and the BGM data synthesis function are processing functions by the authoring program AP. The noise gate function includes a noise gate function during voice recording and a noise gate function executed after voice recording. The noise gate function during voice recording is a function for generating recording data by starting recording when the volume of voice input by the microphone device 101 exceeds a predetermined level and stopping recording when the volume is lower than the predetermined level. . The noise gate function that is executed after voice recording discards the data where the volume is lower than the predetermined level in the recording data that has already been recorded and generated, and holds only the location where the volume exceeds the predetermined level as the recording data It is a function.

For example,
"(Silence) (Hello) (silence) (is nice weather today) (Let's go somewhere to play!) (Silence)"
If you talk to
"Hello",
"It is a good weather today",
"Let's go play somewhere!"
These three data are recorded. These three pieces of data will be referred to as recording data elements. V1, V2, and V3 shown as the recording data in FIG. 2 correspond to the above three recording data elements.

  Next, the BGM data BD is divided into a plurality of bars as shown in the figure. In the BGM data BD in the present embodiment, the head position of each bar (this is a point on the time axis). ) Is recorded. The BGM data BD in which the timing marker designating the start position of the measure is set is specifically referred to as marker-added music data. In the music data with marker in this embodiment, a marker is set at the head position of each measure, but this is an example. Markers may be set at different points from the bar break.

  Then, the authoring program AP links each recording data element disassembled as described above to each timing marker position of the music data with marker. That is, each recording data element is synthesized at the head of each measure of the music data with marker. In this way, the synthesized voice data is generated by synthesizing the recording data and the BGM data BD. In this manner, when the play the synthesized speech data that is generated, for each measure head of the background music to be played on the basis of the BGM data BD, regardless of the actual recording timing, "Hello" from the first measure of the head And say “Today is a good weather” from the head of the second bar, and say “Let's go play somewhere!” From the head of the third bar. In this way, the background music and the recorded data are synchronized.

  Here, as shown in the figure, the phrase of the recording data element V3 extends to the fourth bar area of the BGM data BD. In that case, the third measure of the BGM data BD may be loop-reproduced to synthesize the background music and the recorded data without a sense of incongruity.

  As described above, in the present embodiment, by using the noise gate processing and the music data with the marker, it is possible to automate the synthesis processing of the recording data and the BGM data BD. Therefore, the number of operation buttons is limited, and even a mobile phone terminal with a small monitor size can execute speech synthesis processing without requiring complicated operations.

  However, a method may be used in which the user utters in accordance with the rhythm and performs synthetic recording while reproducing the BGM data BD. In that case, it is possible to synthesize the recorded voice and BGM on an arbitrary time axis by operating the keys while recording.

<Sound effect data synthesis function>
The sound effect data synthesis function is also a processing function by the authoring program AP. As shown in FIG. 2, the sound effect data ED can be synthesized at an arbitrary point on the reproduction time axis of the synthesized voice data. The synthesis method is as follows. The authoring program AP inputs a sound effect insertion instruction by the user while reproducing the BGM data BD (or reproducing the synthesized voice data). While listening to the BGM, the user operates the operation unit 107 at a point where a sound effect is to be inserted, and issues a sound effect insertion instruction. In this way, it is possible to generate synthesized voice data in which BGM data BD, recording data, and sound effect data ED are synthesized.

<Video data composition function>
Further, the authoring program AP can synthesize video data such as a still image or a moving image with the generated synthesized audio data to generate synthesized audio data with video data. The synthesis method is as follows. The first method is a method of linking each video data to each bar of the BGM data BD as shown in FIG. As shown in the figure, one video data may be linked over a plurality of bars. As a second method, the authoring program AP is executed to reproduce the synthesized audio data and allow the user to specify a video data switching point.

  By the method described above, synthesized audio data in which the BGM data BD, the recording data, and the sound effect data ED are synthesized is generated, and further, the synthesized audio data with video data is generated by synthesizing the video data.

<Data format>
The authoring program AP uses the audio processing unit 108 to compress the synthesized voice data generated by the above processing based on a standard such as MP3 or AAC. Further, the synthesized audio data with video data generated by the above processing is subjected to moving image compression processing based on a standard such as MPEG4 using the audio processing unit 108 and the video processing unit 109. In this manner, the synthesized voice data in the present embodiment is stored in the memory 106 as voice electronic mail compressed based on a standard such as MP3 or AAC. Alternatively, the synthesized audio data with video data is stored in the memory 106 as multimedia mail compressed based on a standard such as MPEG4 (based on a standard such as MP3 or AAC).

  Further, the authoring program AP can also convert the synthesized voice data into a general-purpose music data format (for example, mmf) that is normally used in a mobile phone terminal. In this case, the authoring program AP does not use the audio processing unit 108, and converts the synthesized voice data into a general-purpose music data format by processing on the CPU 112. The data generated in this way is stored in the memory 106 as voice electronic mail.

  As shown in FIG. 4, most of general-purpose music data used in recent mobile phone terminals is composed of an FM sound source channel for reproducing BGM music and sound effects and a PCM channel for reproducing recorded sound. Therefore, the FM sound source channel may be used for BGM music and sound effects, and the synthesized voice data may be generated using the PCM channel for the recorded voice data.

  When voice e-mail or multimedia mail is generated by the above processing, the CPU 112 designates the destination address of the e-mail and transmits the voice e-mail or multimedia mail to the destination address. . A terminal that has received the voice electronic mail can view the voice synthesized in accordance with the BGM. Since this voice is not artificial voice but data actually recorded by the sender, it is possible to accurately convey the emotional expression. Further, since such recorded voice is reproduced in accordance with the BGM, it is possible to produce the transmitted content in various modes, unlike a normal voice call. Furthermore, it is possible to further enhance the expressiveness of voice mail by using sound effects inserted at arbitrary points. In addition, in multimedia mail, video is reproduced in addition to these expressive voice mails, so communication with richer expressiveness can be achieved.

<Scenario data>
As shown in FIGS. 2 and 3, the synthesized audio data or the synthesized audio data with video data is composed of a plurality of data layers. That is, the synthesized audio data or the synthesized audio data with video data can be defined by the name of each data file (data identification information) synthesized with the BGM data BD and the synchronization information (time information etc.) for the BGM data BD. .

  Therefore, as a modification of the present embodiment, the data identification information and the synchronization information are used as scenario data, and only the recording data and scenario data are transmitted to the destination address. In other words, the scenario data includes information specifying the BGM data BD, sound effect data ED, and video data, and information indicating the timing at which these data and recording data are combined. In other words, when this method is adopted, each receiving side data file specified by the data identification information is held by the receiving side terminal, or the receiving side terminal needs to receive each data file from a server on the network. There is. However, the transmitting terminal does not need to transmit each data file, and the data transfer amount can be reduced.

  When playing back voice e-mail or multimedia mail using scenario data on the receiving side, a predetermined application program is executed, and BGM data and sound effect data are executed according to the data identification information described in the scenario data. The video data is read from the recording medium and reproduced according to the described synchronization information.

<Flow of composition processing>
The synthesis process described above will be described with reference to the flowcharts of FIGS. As shown in FIG. 5, first, noise gate processing (step S10) is performed, and the recorded data is decomposed into recorded data elements.

  FIG. 6 is a flowchart showing the processing contents of the noise gate processing (step S10). In step S11, the noise gate processing timing is selected. If it is set to perform noise gate processing after recording, microphone recording processing (step S12) is performed, and then noise gate processing (step S13) is performed on the recording data. If it is set to perform noise gate processing during recording, noise gate recording processing (step S14) is executed. The recording data element is generated by the above processing. It should be noted that the setting of whether to perform noise gate processing after recording or to perform gate processing during recording can be freely changed by the user.

  Returning to FIG. 5, next, the BGM process (step S20) is performed. FIG. 7 is a flowchart showing the processing contents of the BGM processing (step S20). First, in step S21, a reading destination of the BGM data BD is selected. When the reading destination of the BGM data BD is a card medium, a reading process of the card medium (ROM card 103) is performed (step S22). On the other hand, when the read destination of the BGM data BD is a public server, a connection process to the public server is performed (step S25). Next, the BGM data BD that is the target of the synthesis process is selected from a plurality of BGM data BD (steps S23 and S26). Then, the obtained BGM data BD is decrypted (steps S24 and S27).

  Returning to FIG. 5, next, a synthesis process is performed (step S30). The contents of the synthesizing process are as described above, and the recording data elements are synthesized so as to synchronize with the timing markers of the music data with markers. As described above, the data after the synthesis processing has a layer configuration as shown in FIG. 2, and the BGM data BD and the recording data are synthesized in a separable state. For example, link information between BGM data BD and recorded data is generated. Alternatively, synthesized voice data including a plurality of tracks (channels) is generated, and each data is stored in each track.

  The recording data element is synthesized at each timing marker position of the BGM data BD. At this time, a specific recording data element may be synthesized at a plurality of timing marker positions. For example, it is possible to repeat a specific message by synthesizing a specific recording data element at a plurality of successive timing marker positions, and to emphasize a particularly important phrase in the recording message. It is.

  Next, a sound effect insertion process (step S40) is performed. The sound effect insertion method is as described above, and the sound effect is synthesized at an arbitrary point on the reproduction time axis of the synthesized voice data in response to the user's designated operation. The synthesized voice data after the sound effect is inserted also has a layer configuration as shown in FIG. 2, and is synthesized in a state where each data can be separated. The method for obtaining the sound effect data ED is the same as the method for obtaining the BGM data BD shown in FIG.

  Next, in step S50, it is selected whether or not to add a video. When a video is added, multimedia mail creation processing (step S60) is performed, and when no video is added, voice electronic mail (boil mail) creation processing (step S70) is performed. Selection of whether or not to add video is specified by the user.

  FIG. 8 is a flowchart showing the contents of the multimedia mail creation process. First, a file format is selected (step S61). If a moving image compression format such as MPEG is set as the file format, the specified video data is added and the compression process is executed according to the set file format (step S62). If scenario data is set as the file format, scenario data defining the synthesized audio data with video data is generated (step S63). This scenario data includes information for designating video data.

  The multimedia mail generated in this way is stored in the memory 106 (step S64). Here, in the multimedia mail stored in the memory 106, the data after the processing in step S62 is the synthesized audio data with compressed video data, and the data after the processing in step S63 is scenario data and recording data. The multimedia mail stored in the memory 106 is transmitted to another terminal with the destination address designated (step S65).

  FIG. 9 is a flowchart showing the contents of voice electronic mail (voice mail) creation processing. First, a file format is selected (step S71). If an audio compression format such as MP3 or AAC is set as the file format, compression processing is executed according to the set file format (step S72). When the general-purpose music file format standardized by the mobile phone terminal is set as the file format, conversion processing to the set file format is performed (step S73). If scenario data is set as the file format, scenario data defining the synthesized speech data is generated (step S74).

  The voice electronic mail generated in this way is stored in the memory 106 (step S75). Here, the voice e-mail stored in the memory 106 is the synthesized voice data in which the data after the processing in steps S72 and S73 is compressed or converted, and the data after the processing in step S74 is scenario data and recorded data. The voice e-mail stored in the memory 106 is transmitted to another terminal with a destination address designated (step S76).

{Second Embodiment}
Next, a second embodiment will be described. In the first embodiment described above, the synthesizing process for generating voice electronic mail or multimedia mail is executed in the mobile phone terminal 100. That is, as shown in FIG. 10, the mobile phone terminal 100 executes all the processes of voice input, voice recording, synthesis, and mail transmission, and transmits the synthesized e-mail to the receiving terminal 200.

  In the second embodiment, the synthesis process is executed in the synthesis server 300 connected via a network. Specifically, as shown in FIG. 11, the cellular phone terminal 100 executes only voice input processing and voice recording processing. Then, the recording data is transmitted from the mobile phone terminal 100 to the composition server 300, and the composition processing is executed in the composition server 300. Here, information indicating the conditions of the composition process may be transmitted from the mobile phone terminal 100 to the composition server 300. The information indicating the conditions for the synthesis process may be information of the same type as the scenario data described in the first embodiment. When the synthesizing server 300 generates a voice e-mail or multimedia mail by executing a synthesizing process similar to that of the first embodiment, the e-mail is stored in a storage device. Then, information specifying a URL is transmitted to the mail receiving terminal 200 as access path information to the electronic mail stored in the storage device. The mail receiving terminal 200 can receive voice electronic mail or multimedia mail by specifying this URL.

  In the example shown in FIG. 11, the access path information (URL information) is transmitted from the composition server 300 to the mail receiving terminal 200. However, the composition server 300 once transmits the access path information to the mail transmission terminal. It may be transmitted to a certain mobile phone terminal 100, and the access path information may be transmitted from the mobile phone terminal 100 to the mail receiving terminal 200.

  As described above, according to the second embodiment, since the cell phone terminal 100 that is a mail transmitting terminal does not execute the combining process, the processing load on the terminal can be reduced.

{Third embodiment}
Next, a third embodiment will be described. The third embodiment is different from the second embodiment in that voice recording is also performed by the synthesis server 300. The other points are the same as in the second embodiment. Specifically, first, the cellular phone terminal 100 that is a mail transmission terminal connects a telephone line to the synthesis server 300. Then, the user utters a message toward the microphone device 101. This message is transferred to the synthesis server 300 through the telephone line, and recording processing is performed in the synthesis server 300. Subsequent processing is the same as in the second embodiment.

  Also in the third embodiment, since the combining process is not executed in the mobile phone terminal 100 that is a mail transmitting terminal, it is possible to reduce the processing load on the terminal.

{Fourth embodiment}
Next, a fourth embodiment of the present invention will be described. The fourth embodiment is different from the first embodiment in that various audio processing is added to the audio input from the microphone device 101, and the audio after the audio processing and the BGM data BD are synthesized. .

  The processing flow is substantially the same as that described with reference to the flowcharts of FIGS. However, the synthesis process in step S30 is different. That is, in the synthesis process of step 30 in the first embodiment, the recorded voice is directly synthesized with the BGM data BD. However, in this embodiment, after the recorded voice is processed, the BGM data BD Synthesize. FIG. 13 is a flowchart of the synthesizing process (step S30) in this embodiment. This synthesizing process is also a process executed by the authoring program AP. In the synthesis processing flow, first, voice processing (step S31) is executed. The sound processing is processing for modulating the recorded sound and processing for giving a special effect to the recorded sound.

  The process for modulating the sound includes a process for changing the tempo (speed) of the sound, a process for shifting the pitch (pitch) of the sound, and the like. For example, the microphone device 101 is uttered and recorded at a slow tempo, but can be modulated into a light tone voice message by changing the voice to an up-tempo by modulation processing. Further, by shifting the pitch of the recorded voice and shifting the pitch of the message to the lower side as a whole, it is possible to produce a heavy atmosphere and a bad mood.

  In addition, processing for giving special effects to sound includes equalizer processing, harmonization processing, echo processing, and the like. The equalizer process is a process of changing the frequency characteristics of a voice message by emphasizing a high sound range, emphasizing a low sound range, or cutting a specific sound range. The harmonization process is a process of adding another sound that becomes a chord to the pitch of the recorded voice to make the voice message a chord. The echo process is a process for reproducing the recorded voice with a time difference and making the sound resonate. For example, a message with a magnificent atmosphere can be generated by the harmonization process. In addition, a fantastic atmosphere can be produced by echo processing.

  With reference to FIG. 14, the flow of the audio processing (step S31) will be described. First, it is determined whether or not to change the tempo of the recorded sound (step S301). When the tempo is not changed, the process proceeds to step S305.

  When changing the tempo (Yes in step S301), it is determined whether or not to perform the synchronization process (step S302). Two processing methods are prepared as tempo change processing. One is manual tempo change processing, and the other is automatic tempo change processing (synchronization processing). The manual tempo changing process is a process of changing the tempo of the recorded sound in accordance with the tempo setting designated by the user. The automatic tempo change process is a process in which the tempo of the recorded sound is changed so as to be synchronized with the BGM data BD.

  Before explaining the tempo changing process, first, a method for obtaining a tempo at the time of recording the recorded sound (before changing the tempo) will be described. There are two methods for obtaining the tempo during recording. One is a method determined by the guide voice. When voice is input by the user, a guide voice with a constant rhythm is reproduced like a metronome. The user records the voice while listening to the guide voice. The recorded voice includes information on the guide voice, and the tempo of the recorded voice is determined by the guide voice. The other method is a method for automatically acquiring the rhythm of the recorded voice. A voice analysis process is performed on the recorded voice, and the voice is divided into syllable units. The tempo of the recorded voice is automatically acquired from the utterance timing of each syllable. By the above two methods, the tempo at the time of recording the recorded voice is determined.

  Next, the tempo change process will be described. The manual tempo change processing method is a method of correcting the tempo at the time of recording according to the tempo setting designated by the user. For example, if the recording tempo is determined by the guide sound, the tempo of the guide sound is corrected to the set tempo. Accordingly, the tempo of the recorded sound is also changed. Alternatively, when the recording tempo is automatically acquired, the automatically acquired tempo is changed according to the set tempo setting.

  The automatic tempo change processing method is a method in which the tempo at the time of recording of the recorded voice is automatically changed so as to synchronize with the BGM data BD. If the tempo of the BGM data BD is recorded in advance in the BGM data BD, it can be used. Alternatively, the BGM data BD can be acquired by performing voice analysis. For example, it is possible to analyze the tempo based on a voice that engraves a rhythm such as a drum. If the tempo at the time of recording is determined by the guide voice, the tempo of the guide voice is changed to match the tempo of the BGM data. In accordance with this, the tempo of the recorded sound is also changed to synchronize with the BGM. Alternatively, when the recording tempo is automatically acquired, the automatically acquired tempo is automatically changed to synchronize with the tempo of the BGM data BD.

  Returning to the flowchart of FIG. When the synchronization process is performed (Yes in step S302), the tempo of the recorded sound is automatically changed to synchronize with the BGM data BD (step S303). When the sync process is not performed (No in step S302), the tempo of the recorded sound is changed according to the set value (step S304).

  Next, it is determined whether or not to perform pitch shift processing (step S305). If it is set to perform the pitch shift process (Yes in step S305), the pitch shift process is executed according to the set value (step S306). The set value of the pitch shift process is a pitch shift amount. The shift amount of the recorded voice can be set by the user.

  Next, it is determined whether or not to perform the equalizer process (step S307). If it is set to perform the equalizer process (Yes in step S307), the equalizer process is executed according to the set value (step S308). The set value of the equalizer process is information on a sound range to be emphasized or information on a sound range to be cut, and can be set by the user.

  Next, it is determined whether or not to perform harmonization processing (step S309). If it is set to perform the harmonization process (Yes in step S309), the harmonization process is executed according to the set value (step S310). The setting value of the harmonization process can be set by the user. For example, a third sound is added to the bass sound. Alternatively, settings such as adding 3rd and 5th sounds to the bass sound may be made possible.

  Next, it is determined whether or not to perform echo processing (step S311). If it is set to perform echo processing (Yes in step S311), echo processing is executed according to the set value (step S312). The setting value of the echo process is information for designating a time difference between echo sounds, a duration time of the echo sound, and the like, and can be set by the user.

  As described above, various processes are executed in the audio process (step S31), and what audio process is to be executed may be set by the user. For example, the user is set to execute automatic tempo change and harmonization processing, or is set to execute manual tempo change processing, pitch shift processing and echo processing, or freely selects a combination of audio processing It is possible. In addition, it is only necessary that the user can designate a point on the time axis of the voice to which the voice processing is applied. As a result, for example, the first half of the sound can produce a fantastic atmosphere by echo processing, and the second half can be made up-tempo and a message overflowing in speed.

  In this way, what kind of audio processing is performed can be freely set by the user, but in order to free the user from the burden of making such detailed settings, multiple types corresponding to multiple themes are available. It is desirable to have a set of settings. In this setting set, combinations of voice processing to be executed, setting values for each voice processing to be executed, and the like are defined. For example, it is convenient to prepare a setting set that matches the tune, such as rock, ballad, and lap. Alternatively, it is convenient to prepare a set that matches emotions such as sadness, anger, and joy. The user can easily create a rock-tone voice message by selecting rock-like music as the BGM data BD and selecting the lock-like as the setting set.

  Returning to FIG. 13, when the audio processing is completed in step S31, the synthesis processing of the audio subjected to the audio processing and the BGM data BD is performed (step S32). When the synthesizing process is completed, steps S40, S50, and S60 (or S70) shown in FIG. 5 are executed, and multimedia mail or voice mail is transmitted.

  As described above, according to the present embodiment, the recorded voice is modulated or synthesized with BGM data BD after various special effects are given. As a result, it is possible to further enhance the expressive power of voice mail and voice mail with video. In particular, it is possible to reproduce a message in accordance with the atmosphere and tone of the BGM data BD by devising a combination of the voice processing setting and the BGM data BD.

  According to the present invention, by the noise gate process and the tempo change process, the BGM is combined with the recorded voice instead of the sound that flows independently of the recorded voice to constitute one sound. In other words, the recorded voice phrase or sentence is linked to the BGM bars by the noise gate processing and merged. Furthermore, the tempo change process merges the syllables of the recorded voice into the BGM rhythm within each BGM measure, thereby forming a unified sound.

  In the above embodiment, after recording the voice, the BGM data BD is acquired, and then the voice processing and the synthesis processing of the recorded voice are performed. However, the voice is recorded during the voice recording. Processing may be executed. In this case, voice recording and voice processing are performed in parallel, and then BGM data BD is acquired, and the voice after voice processing and BGM data BD are synthesized. However, since the information of the BGM data BD is necessary to perform the automatic tempo change process by the synchro process, the automatic change process may be executed after obtaining the BGM data BD.

  Further, in the above-described embodiment, a case has been described in which sound processing for modulating a recorded sound or giving a special effect to the recorded sound is performed. However, a sound processing for modulating BGM music or giving a special effect to BGM music. May be executed. As a voice processing method, the same processing as that set for the recorded voice may be performed, or for BGM music, the contents of the voice processing may be set separately.

It is a block diagram of the mobile phone terminal concerning an embodiment. It is a figure which shows the layer structure of a voice electronic mail. It is a figure which shows the layer structure of a multimedia mail. It is a figure which shows a general purpose music file format. It is a flowchart which shows the main process of a synthesis process. It is a flowchart of a noise gate process. It is a flowchart of a BGM data acquisition process. It is a flowchart which shows the production process of multimedia mail. It is a flowchart which shows the production process of a voice mail. It is a figure which shows the flow of the process in 1st Embodiment. It is a figure which shows the flow of the process in 2nd Embodiment. It is a figure which shows the flow of the process in 3rd Embodiment. It is a flowchart of the synthetic | combination process in 4th Embodiment. It is a flowchart of an audio | voice process.

Explanation of symbols

100 Mobile phone terminal 101 Microphone device 103 ROM card 106 Memory (SDRAM)
AP authoring program BD BGM data ED Sound effect data

Claims (30)

Voice input means;
Means for recording the voice input by the voice input means and storing it as recorded data;
Synthesis processing means for synthesizing the recording data and music data and generating synthesized voice data;
Transmitting means for transmitting the synthesized voice data as an e-mail;
An e-mail transmitting terminal comprising: In the e-mail transmission terminal according to claim 1,
The music data is
Sound effect data,
Including
The synthesis processing means includes
Means for synthesizing the sound effect data at a specified point on the playback time axis of the synthesized sound data;
An e-mail transmission terminal comprising: In the e-mail transmission terminal according to claim 1 or 2,
The e-mail transmission terminal, wherein the music data is stored in a recording medium detachable from the e-mail transmission terminal or a storage device of a server on a network to which the e-mail transmission terminal can be connected. In the e-mail transmission terminal according to claim 3,
The music data stored in the recording medium or the storage device is encrypted,
The synthesis processing means includes
Means for decrypting the encrypted music data;
An e-mail transmission terminal comprising: In the electronic mail transmitting terminal according to any one of claims 1 to 4,
The music data is
Marked music data with one or more timing markers set in advance on the playback time axis,
Including
The synthesis processing means includes
Means for disassembling a portion of the recorded data below a predetermined volume as a silent portion, and decomposing the recorded data into a plurality of recorded data elements separated by a silent portion;
Means for synthesizing each of the plurality of recording data elements in synchronism with a time set by the timing marker of the music data with the marker;
An e-mail transmission terminal comprising: In the e-mail transmission terminal according to claim 5,
The marker-attached music data is
Music data with timing markers set at the beginning of each measure,
An e-mail transmission terminal comprising: In the e-mail transmission terminal according to any one of claims 1 to 6,
The synthesis processing means includes
Means for synthesizing video data at an arbitrary point on the time axis of the synthesized audio data in response to a designation operation by a user, and generating synthesized audio data with video data;
Including
The transmission means includes
Means for transmitting the synthesized audio data with video data as an e-mail;
An e-mail transmission terminal comprising: In the e-mail transmission terminal according to any one of claims 1 to 6,
Means for compressing and converting the synthesized voice data;
An e-mail transmitting terminal comprising: The e-mail transmission terminal according to claim 7, further comprising:
Means for compressing and converting the synthesized audio data with video data into a moving image;
An e-mail transmitting terminal comprising: In the e-mail transmission terminal according to any one of claims 1 to 6,
The e-mail transmission terminal is a mobile phone terminal, and the synthesized voice data is converted into a standard music data format defined in the mobile phone terminal. The electronic mail transmitting terminal according to any one of claims 1 to 10, further comprising:
Means for generating scenario data recording the rules of the synthesis process;
With
The transmission means includes
Means for transmitting the recording data and the scenario data as an e-mail;
An e-mail transmission terminal comprising: 12. The e-mail transmission terminal according to claim 1, further comprising:
Processing means for processing the voice input by the voice input means;
With
The processing means includes
Means for modulating the sound and / or for providing a special effect to the sound;
Including
The e-mail transmission terminal characterized in that the synthesizing processing unit synthesizes the recording data and the music data after being processed by the processing unit. In the e-mail transmission terminal according to claim 12,
The processing means includes
Means for executing a tempo change process and / or a pitch shift process of the voice input by the voice input means;
An e-mail transmission terminal comprising: In the e-mail transmission terminal according to claim 12,
The processing means includes
Means for executing any one or more of equalizer processing, harmonization processing, and echo processing on the voice input by the voice input means;
An e-mail transmission terminal comprising: In the e-mail transmission terminal according to any one of claims 12 to 14,
A plurality of setting information corresponding to a plurality of themes is prepared in advance, each setting information defines the contents of processing executed by the processing means, and when one setting information is selected, An e-mail transmission terminal characterized in that contents of processing by the processing means are determined. The electronic mail transmitting terminal according to any one of claims 1 to 15, further comprising:
Means for modulating the music defined by the music data and / or means for giving a special effect to the music defined by the music data;
An e-mail transmission terminal comprising: A system for forwarding email,
A terminal,
A synthesis server;
With
The terminal
Voice input means;
Means for recording the voice input by the voice input means and storing it as recorded data;
Means for transmitting the recording data to the synthesis server;
With
The synthesis server
Synthesis processing means for synthesizing the received recording data and music data and generating synthesized voice data;
Means for storing the synthesized voice data as an e-mail in a storage device;
An e-mail system comprising: A system for forwarding email,
A terminal,
A synthesis server;
With
The terminal
Voice input means;
Means for transmitting the voice input by the voice input means to the synthesis server;
With
The synthesis server
Means for recording the received voice as recorded data;
A synthesis processing means for synthesizing the recording data and the music data and generating synthesized voice data;
Means for storing the synthesized voice data as an e-mail in a storage device;
An e-mail system comprising: The electronic mail system according to claim 17 or claim 18,
The music data is
Sound effect data,
Including
The synthesis processing means includes
Means for synthesizing sound effect data at a specified point on the playback time axis of the synthesized sound data;
An e-mail system characterized by including: The electronic mail system according to any one of claims 17 to 19,
The music data is
Marked music data with one or more timing markers set in advance on the playback time axis,
Including
The synthesis processing means includes
Means for disassembling a portion of the recorded data below a predetermined volume as a silent portion, and decomposing the recorded data into a plurality of recorded data elements separated by a silent portion;
Means for synthesizing each of the plurality of recording data elements in synchronism with a time set by the timing marker of the music data with the marker;
An e-mail system characterized by including: The electronic mail system according to claim 20,
The marker-attached music data is
Music data with timing markers set at the beginning of each measure,
An e-mail system characterized by including: The e-mail system according to any one of claims 17 to 21, further comprising:
Means for synthesizing video data at an arbitrary point on the time axis of the synthesized audio data in response to a designation operation by a user, and generating synthesized audio data with video data;
An e-mail system comprising: The e-mail system according to any one of claims 17 to 21, further comprising:
Means for compressing and converting the synthesized voice data;
An e-mail system comprising: The e-mail system according to claim 22, further comprising:
Means for compressing and converting the synthesized audio data with video data into a moving image;
An e-mail system comprising: The electronic mail system according to any one of claims 17 to 21,
The electronic mail system according to claim 1, wherein the terminal is a mobile phone terminal, and the synthesized voice data is converted into a standard music data format defined in the mobile phone terminal. The electronic mail system according to any one of claims 17 to 25,
The synthesis server further includes:
Processing means for processing audio received from the terminal;
With
The processing means includes
Means for modulating the sound and / or for providing a special effect to the sound;
Including
The e-mail system, wherein the synthesizing unit synthesizes the recording data and the music data after being processed by the processing unit. The e-mail system of claim 26.
The processing means includes
Means for executing a tempo change process and / or a pitch shift process of the voice input by the voice input means;
An e-mail system characterized by including: The e-mail system of claim 26.
The processing means includes
Means for executing any one or more of equalizer processing, harmonization processing, and echo processing on the voice input by the voice input means;
An e-mail system characterized by including: The e-mail system according to any one of claims 26 to 28,
A plurality of setting information corresponding to a plurality of themes is prepared in advance, each setting information defines the contents of processing executed by the processing means, and when one setting information is selected, An electronic mail system characterized in that contents of processing by a processing means are determined. 30. The electronic mail system according to claim 17, further comprising:
Means for modulating the music defined by the music data and / or means for giving a special effect to the music defined by the music data;
An e-mail system characterized by including:

Images