JP5287735B2 - Information processing device - Google Patents

Description

  The present invention relates to an information processing apparatus capable of performing speech translation between a plurality of languages.

  As the world becomes more globalized, opportunities for communication with people with different native languages are increasing. For example, it may be shopping or restaurant conversation during overseas travel, business conversation or telephone conference. Speech translation technology is a technology that facilitates communication between different languages, and recently, many translation devices have been commercialized.

  For example, according to Patent Document 1, a speech translation apparatus capable of realizing smooth communication has been proposed. This speech translation apparatus displays a character string of a translation target language obtained by speech recognition in area A of the display unit, displays a translation language character string in area B, and translates the character string or translation language of the translation target language. If a keyword for document search is extracted from the character string of, and the language of the document searched using the searched keyword is the translation target language, it is translated into the translation language, and the language of the searched document is the translation language The document translated into the language to be translated is displayed in the area C, and the document obtained by translating the document is displayed in the area D.

JP 2009-205579 A

  To use speech translation in conversations between other languages, bidirectional speech translation is required. Take bi-directional speech translation between Japanese and English as an example. Japanese speech recognition processing module, Japanese-English machine translation processing module, English speech synthesis processing module, English speech recognition processing module, English-Japanese machine translation processing module Japanese speech synthesis processing modules are roughly divided into six processing modules. In the processing technology of “speech recognition”, “machine translation”, and “speech synthesis”, it is common to use dictionary data in which linguistic information and waveform information are learned in advance. Requires an initialization process such as reading dictionary data from storage or the like in advance.

  If such a translation technology is used on a PC (Personal Computer), if the data transfer speed with the storage is high and the RAM (Random Access Memory) size is large, all processing is performed when the speech translation application is started. Even if the module is initialized, the processing time does not increase so much, and the occupation of the RAM by the dictionary data is not particularly problematic.

  However, when a speech translation application is generally operated on a low-spec machine such as a portable terminal, for example, a situation occurs in which all modules cannot be initialized when the speech translation application is activated because the RAM size is small. To do. Even if a RAM size sufficient to initialize all modules can be secured, other processes such as incoming voice calls and mail reception may operate. It is necessary to secure it.

  On the other hand, there is also a method in which initialization is not performed when the speech translation application is activated, but initialization is performed only when the process is operated, and the dictionary data is released from the RAM when the process is completed. However, this method requires an initialization process every time voice translation is executed by the voice translation application. In particular, a portable terminal often has a slower data transfer rate than a PC or the like, and has a low calculation capability, so that a considerable amount of time is required to execute speech translation.

  There is also a method for realizing processing that requires a lot of RAM size and computation in a server via a communication network in order to realize a speech translation application. However, this method cannot achieve speech translation unless the environment is communicable. For example, in overseas countries where speech translation is most active, there are many environments where communication is not possible, and even if communication is possible, a high communication fee is incurred. There are many cases.

  For these reasons, conventionally, it has been difficult to shorten the processing time when executing the speech translation function in a stand-alone manner.

  The present invention has been made in view of the above problems, and is an information processing apparatus having a stand-alone speech translation function, and the module initialization timing can be set even when the usable RAM size is limited. It is an object of the present invention to provide an information processing apparatus in which the speech translation processing time is minimized by controlling.

  In order to solve the above-described problem, an information processing apparatus according to the present invention is an information processing apparatus having a plurality of modules related to translation processing. When the translation processing is executed, a free RAM area that can be used for translation processing is allocated. First calculation means for calculating, a maximum size that occupies a storage area of the RAM when each of the free area and the plurality of modules performs processing, and storage of the RAM necessary for initialization of each of the plurality of modules Selection means for selecting a module that can be initialized from the plurality of modules based on the area and the time required for initialization of each of the plurality of modules, and translation processing is performed on the module selected by the selection means A first initialization unit that initializes before, and when any of the modules performs a process in accordance with the execution of the translation process, If that has not been initialized by the initialization means, characterized in that it comprises a second initialization means for releasing during translation processing ends as well as the processing start module initialization when this module.

  According to the information processing apparatus according to the present invention, the voice translation processing time can be minimized by controlling the initialization timing of the module even when there is a restriction on the usable RAM size as well as the stand-alone speech translation function. It becomes possible to limit to the limit.

The perspective view which shows the state which the information processing apparatus (mobile telephone) based on this invention opened. 1 is a block diagram showing a configuration of an information processing apparatus (mobile phone) according to the present invention. The figure which shows an example of the screen at the time of the translation control process being performed in the information processing apparatus (cellular phone) which concerns on this invention. The data block diagram which shows an example of module list information. The data block diagram which shows an example of module initialization time information. The data block diagram which shows an example of initialization pattern information. The flowchart which shows the procedure of the translation control process in the information processing apparatus (mobile telephone) which concerns on this invention. The flowchart which shows the procedure of the pre-initialization process in the information processing apparatus (cellular phone) concerning this invention. (A) is a correspondence table showing the relationship between identification codes and initialization states, and (B) is a data configuration diagram showing an example of module initialization state information. The flowchart which shows the procedure of the initialization process at the time of a calculation process at the time of the information processing apparatus (mobile telephone) which concerns on this invention performing a translation control process. The data block diagram which shows an example of log | history information. The data block diagram which shows an example of the additional information of an initialization pattern.

  Embodiments of an information processing apparatus according to the present invention will be described with reference to the accompanying drawings. A mobile phone 1 will be described as an example of the information processing apparatus according to the present invention.

  FIG. 1 is a perspective view of the mobile phone 1. As shown in FIG. 1, the mobile phone 1 has a rectangular plate-like upper casing 10 and a lower casing 11 that is substantially the same shape as the upper casing 10 in a closed state. It is formed by being laminated so as to cover. The upper housing 10 and the lower housing 11 are coupled so as to be slidable by a predetermined distance from each other in a predetermined direction (for example, the X direction in FIG. 1). By sliding, the deformed state is changed from the closed state to the open state, or from the open state to the closed state.

  A display device 12 such as a liquid crystal display for displaying data, a speaker 13 for outputting sound, and a microphone 14 for inputting sound are provided on the outer surface of the upper housing 10 (the surface not facing the lower housing 11). Yes. An input device 15 such as operation keys for inputting data when the user presses is provided on the inner surface of the lower housing 11 (the surface facing the upper housing 10).

  FIG. 2 is a block diagram showing the configuration of the mobile phone 1. As shown in FIG. 2, the cellular phone 1 includes a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, a communication device 23, the display device 12, and the speaker 13 described above. The microphone 14 and the input device 15 are connected by a bus 24 so that they can communicate with each other.

  The CPU 20 performs overall control of the mobile phone 1 and performs a translation control process, which will be described later, and various other processes. Further, the CPU 20 includes an input interface for the input device 15. When the CPU 20 detects that an operation key provided as the input device 15 is pressed, for example, the CPU 20 performs processing corresponding to the operation key. Further, the CPU 20 generates screen data based on the control of the application program being executed and displays the screen data on the display device 12.

  CPU20 produces | generates an analog audio | voice signal from the audio | voice collected with the microphone 14, and converts this analog audio | voice signal into a digital audio | voice signal. Further, when acquiring the digital audio signal, the CPU 20 converts the digital audio signal into an analog audio signal and outputs the analog audio signal as audio from the speaker 13.

  The RAM 21 is a storage device that temporarily stores data as a work area when the CPU 20 performs processing. The ROM 22 is a storage device that stores a processing program (processing program for translation control processing, etc.) when the CPU 20 performs processing, data used for processing (module list information 40, which will be described later), and the like. It is assumed that dictionary data used for speech translation is also stored in the ROM 22 in advance. The mobile phone 1 may include a storage device such as a hard disk and a nonvolatile memory in addition to the ROM 22, and the module list information 40 may be stored in the storage device instead of the ROM 22.

  Based on the control of the CPU 20, the communication device 23 restores data by performing spectrum despreading processing on a received signal received from a base station (not shown) via the antenna 23 a. This data is output from the speaker 13 under the control of the CPU 20, displayed on the display device 12, or recorded in the RAM 21. In addition, the communication control unit 26 acquires sound data collected by the microphone 14, data input via the display device 12 or the input device 15, or data stored in the RAM 21 or the ROM 22 based on the control of the CPU 20. These data are subjected to spread spectrum processing and transmitted to the base station via the antenna 23a.

  The mobile phone 1 is provided with a speech translation function that translates language A into language B or translates language B into language A, for example. Further, when performing the translation control process, the mobile phone 1 uses the RAM size of the RAM 21 required for the initialization process of each module used for translation control, the RAM size required for the calculation process of each module, and each module. A function for controlling the timing of the initialization process of each module based on the time required for the initialization process and the free RAM size of the RAM 21, thereby limiting the free RAM size of the RAM 21 Even so, speech translation processing time can be minimized.

  The CPU 20 includes a language A speech recognition unit 30, an AB machine translation unit 31, and a language B speech synthesis unit 32 as modules that translate language A into language B, and translates language B into language A. The module includes a language B speech recognition unit 33, a B-A machine translation unit 34, and a language A speech synthesis unit 35.

  When the language A speech is input to the microphone 14, the language A speech recognition unit 30 generates a language A sentence indicating the input speech. The AB machine translation unit 31 performs machine translation of the generated language A text into language B text. The language B speech synthesizer 32 synthesizes speech based on the machine-translated sentence and generates speech data for output.

  When the language B speech is input to the microphone 14, the language B speech recognition unit 33 generates a language B sentence indicating the input speech. The B-A machine translation unit 34 machine-translates the generated language B text into language A text. The language A speech synthesizer 35 synthesizes speech based on the machine-translated sentence and generates speech data for output.

  As an example, an operation for speech translation from language A (Japanese) to language B (English) will be briefly described. For example, if the user utters “Please make it a little cheaper” in Japanese, the language A speech recognition unit 30 recognizes the speech waveform and generates a character string “Please make it a little cheaper” to display it. 12 is displayed. FIG. 3 is a diagram illustrating an example of a screen displayed on the display device 12 when the mobile phone 1 performs the translation control process. As shown in FIG. 3, the language A speech recognition unit 30 displays the speech-recognized character string in the pre-translation language display field 12 b on the screen 12 a of the display device 12. Further, as shown in FIG. 3, the AB machine translation unit 31 machine translates the generated character string into English to generate a character string “Please make it a little chainer”, and translates the screen 12a. It is displayed in the rear language display column 12c. The language B speech synthesizer 32 synthesizes speech based on the machine-translated character string, and outputs a “Please make it a little chainer” character string as speech.

  In the case of speech translation from English to Japanese, the above reverse processing is executed. Note that in the mobile phone 1, the process of translating speech from Japanese to English and the process of translating speech from English to Japanese do not operate in parallel.

  When the language A speech recognition unit 30, the AB machine translation unit 31, the language B speech synthesis unit 32, the language B speech recognition unit 33, the BA machine translation unit 34, and the language A speech synthesis unit 35 perform processing, In the RAM 21, it is necessary to occupy a storage area for reading dictionary data (RAM for initialization) and a storage area for calculation processing (RAM for calculation). By performing initialization before the arithmetic processing in advance, it is possible to execute the subsequent processing at high speed. However, in that case, the storage area of the RAM 21 is continuously occupied. On the other hand, if the initialization is not executed in advance, it is avoided that the storage area of the RAM 21 is continuously occupied by the dictionary data. However, since the initialization is required every time processing is performed, the processing time becomes long.

  The ROM 22 stores module list information 40 in which modules used when the CPU 20 performs translation control processing are listed. FIG. 4 is a data configuration diagram illustrating an example of the module list information 40. As shown in FIG. 4, the module list information 40 includes, for each module information 40a, type information 40b indicating the type of processing, and the maximum size (peak) of the storage area used when the module performs processing in the RAM. RAM size information 40c indicating (RAM size) is associated with each other. When a module performs processing, it is necessary to perform initialization processing that expands data necessary for arithmetic processing, which is the original processing of the module, from the ROM 22 to the RAM 21. There are two types of processing: “processing” and “calculation” processing that is the original processing of the module.

  Specifically, the RAM size of “M_Aa” for initialization of the language A speech recognition unit 30 (ASR_A) for initialization is “0” for the computation of the language A speech recognition unit 30 (ASR_A) for computation. The RAM sizes of “M_Aa_w” are associated with each other. The RAM size of “M_Ma” corresponds to the initialization of the AB machine translation unit 31 (MT_AB), and the RAM size of “M_Ma_w” corresponds to the operation of the AB machine translation unit 31 (MT_AB). It is attached. The RAM size of “M_Ta” is associated with the initialization of the language B speech synthesizer 32 (TTS_B), and the RAM size of “M_Ta_w” is associated with the operation of the language B speech synthesizer 32 (TTS_B). ing.

  Similarly, the language B speech recognition unit 33, the BA machine translation unit 34, and the language A speech synthesis unit 35 are also associated with the RAM sizes necessary for the initialization process and the calculation process.

  The ROM 22 stores initialization time information 41 in which the initialization time of each module is recorded. FIG. 5 is a data configuration diagram showing an example of the initialization time information 41. As shown in FIG. 5, in the initialization time information 41, the time required for the initialization process is associated with the module information 40a indicating each module.

  FIG. 6 shows an example of initialization pattern information 42 indicating the initialization timing pattern of each module used when speech translation is performed from language A to language B (for example, initialization timing of a module that translates from language A to language B). FIG. As shown in FIG. 6, the initialization pattern information 42 includes pre-initialization availability information 42b indicating whether or not the language A speech recognition unit 30 is pre-initialized for each pattern identification information 42a, AB. Pre-initialization availability information 42c indicating whether the machine translation unit 31 is pre-initialized, pre-initialization availability information 42d indicating whether the language B speech synthesis unit 32 is pre-initialized, peak RAM size information 42e, The additional processing time information 42f indicating the time taken to initialize each module at the time of calculation is associated with each module.

  The initialization pattern information 42 includes a peak RAM size and an initial value at the time of execution of module calculation processing for all patterns that can be considered as combinations of necessity of pre-initialization of each module used when speech translation from language A to language B is performed. The time (additional processing time) required for the conversion is registered. In FIG. 6, the pre-initialization availability information of a module to be pre-initialized is represented by “◯”, and the pre-initialization availability information of a module that is not pre-initialized is represented by “×”. Although not shown, initialization pattern information for speech translation from language B to language A is also stored in advance, as is the initialization pattern for speech translation from language A to language B. Here, the prior initialization is module initialization that is performed in advance at the start of the speech translation function.

  The peak RAM size and the additional processing time when executing the translation control process from language A to language B will be described. In the speech translation process, when the module initialization process is performed in advance, the storage area of the RAM 21 used for the initialization process is occupied as it is when the calculation process of each module is executed. Therefore, the more modules to be initialized in advance, the more storage areas of the RAM 21 are required. The peak RAM size of the initialization pattern information 42 represents the maximum RAM value required in the entire speech translation processing when processing is performed with each pattern. In speech translation processing, as described above, machine translation is executed after completion of speech recognition, and speech synthesis is executed after completion of machine translation. Therefore, the modules do not operate in parallel. For this reason, the peak RAM size is calculated as the sum of the maximum RAM size required for the arithmetic processing of each module and the RAM size required for initialization of all the modules to be pre-initialized.

  On the other hand, modules that are not initialized in advance are initialized each time the arithmetic processing of each module is performed. Therefore, the more modules that are not initialized in advance, the more processing time is required for additional initialization that occurs each time the arithmetic processing of each module is performed. The additional processing time of the initialization pattern information 42 represents the total processing time for the additional initialization processing in the entire speech translation processing. That is, the additional processing time is calculated as the time required for the initialization processing of the module that is not initialized in advance.

  For example, among the patterns registered in the initialization pattern information 42, the pattern that occupies the most storage area of the RAM 21 is a pattern (pattern 1 in FIG. 6) that is initialized in advance by all modules. Is defined as a maximum value max {M_Aa_w, M_Ma_w, M_Ta_w} = M_w of the usage amount of the RAM 21 when each module performs arithmetic processing, the peak RAM size M_Pk_max is “M_Aa + M_Ma + M_Ta + M_w”. Further, the additional processing time Tpa1 is 0 because no additional initialization processing is required when performing the arithmetic processing by each module.

  On the other hand, in the pattern in which all modules are not initialized in advance (pattern 8 in FIG. 6), the RAM is not occupied by the initialization. Therefore, the peak RAM size is “Mpa8 = max {M_Aa + M_Aa_w, M_Ma + M_Ma_w, M_Ta + M_Ta_w}”. On the other hand, since initialization is performed at the time of execution of arithmetic processing of each module, the additional processing time becomes “Tpa8 = T_Aa + T_Ma + T_Ta”.

  The initialization pattern is selected based on the peak RAM size and additional processing time information stored in the initialization pattern information 42 and the free RAM size that can be used for the translation control process.

  If the free RAM size in the storage area of the RAM 21 is defined as “M_Free” and the minimum RAM size reserved for other applications when executing the translation control processing is defined as “M_etc”, the free space that can be used for the translation control processing The RAM size M_max is defined as “M_Free-M_etc”.

  If the peak RAM size M_Pk_max required for speech translation from language A to language B is smaller than the free RAM size M_max, it is possible to execute initialization processing for all modules in advance before speech translation. By using pattern 1, the translation control process can be executed with a minimum processing time. However, when the peak RAM size M_Pk_max is smaller than the free RAM size M_max, all the modules cannot be initialized in advance, so only some of the modules are initialized in advance. In this case, the peak RAM size is reduced, but the processing time is longer by the time required for the initialization processing of the modules that have not been pre-initialized, so the processing time is longer than when all the modules are initialized. Become.

  Hereinafter, in order to minimize the processing time by controlling the timing of initialization of each module, a method for selecting a module to be pre-initialized will be described. When the translation control processing is performed by the mobile phone 1, it is selected which module to pre-initialize according to the free RAM size. Considering three types of modules for translating from language A to language B, there are eight patterns 1 to 8 as shown in the initialization pattern information 42 for determining whether or not to pre-initialize.

The optimization of the initialization process for selecting the module to be initialized in advance and the module to be initialized at the timing of executing the arithmetic process is performed based on the initialization pattern information 42 in the following procedure.
(1) The free RAM size M_max is calculated.
(2) A pattern whose peak RAM size is equal to or less than the free RAM size M_max is selected from patterns 1 to 8 (Mpa1 to MPa8).
(3) Of the patterns selected in (2) above, select the pattern with the shortest additional processing time (Tpa1 to Tpa8).

  The procedure of the translation control process in which the mobile phone 1 performs the translation process while optimizing the initialization timing of each module will be described in detail based on the flowcharts shown in FIGS. Note that a case where translation from the language A to the language B is first performed will be described as an example.

  FIG. 7 is a flowchart showing an overall procedure when the mobile phone 1 performs the translation control process, and FIG. 8 is a flowchart showing a procedure when the mobile phone 1 performs the pre-initialization process. As shown in FIG. 7, first, the CPU 20 performs a pre-initialization process for each module that performs translation from language A to language B (S101). This pre-initialization process will be described based on the flowchart shown in FIG.

  The CPU 20 first calculates an empty RAM size (M_max) (S201). Further, the CPU 20 sets the minimum value (Tmin) of the additional processing time to the initial value “infinity (∞)”, sets the initial value “1” to the variable i for pattern counting, and the variable indicating the optimum pattern. An initial value “0” is set to n (S203). Here, the initial value “infinity” means a sufficiently large value, and more specifically, it may be a value larger than Tpa8.

  The CPU 20 determines whether or not the peak RAM size (Mpa (i)) of the pattern i is smaller than the free RAM size (M_max) (S205). At this time, for example, when the variable i is “1”, the CPU 20 determines the peak RAM size of the pattern 1. When the peak RAM size of the pattern i is smaller than the free RAM size (Yes in S205), the CPU 20 determines whether or not the additional processing time (Tpa (i)) of the pattern i is smaller than the minimum value Tmin of the additional processing time. Judgment is made (S207).

  When the additional processing time (Tpa (i)) of the pattern i is smaller than the minimum value Tmin of the additional processing time (Yes in S207), the CPU 20 adds the additional processing time of the pattern i to the minimum value (Tmin) of the additional processing time. (Tpa (i)) is set (S209). When the minimum value of the additional processing time is set to infinity, the additional processing time of the pattern i at this time is always set to the minimum value of the additional processing time. Then, the CPU 20 sets “i” at this time to the variable n (S211). The variable n is a variable indicating an optimal pattern, and setting “i” to the variable n means that the pattern i is optimal at the present time.

  When the peak RAM size of the pattern i is equal to or larger than the free RAM size (No in S205), and when the additional processing time (Tpa (i)) of the pattern i is equal to or larger than the minimum value of the additional processing time (No in S207) Alternatively, after completing the processes of steps S209 and S211, the CPU 20 adds “1” to the variable i (S213).

  Further, the CPU 20 determines whether or not the variable i is larger than “8” which is the total number of patterns (S215). If the variable i is 8 or less (No in S215), that is, if the processes in steps S205 to S211 have not been performed for all patterns, the process returns to step S205, and the CPU 20 performs the processes in steps S205 to S215.

  When the variable i is greater than 8 (Yes in S215), that is, when the processes of Steps S205 to S211 have been performed for all patterns, the CPU 20 determines whether or not the variable n is greater than 0 (S217). This is because when the variable n is “0”, an optimal pattern has not been selected and pre-initialization cannot be performed.

  When the variable n is larger than 0 (Yes in S217), the CPU 20 executes the initialization of the pattern n based on the initialization pattern information 42 (S219). At this time, for example, if the variable n is “1”, the CPU 20 based on the pattern 1 of the initialization pattern information 42, the language A speech recognition unit 30, the AB machine translation unit 31, the language B speech synthesis. When the unit 32 is initialized and the pattern 2 is selected, the language A speech recognition unit 30 and the AB machine translation unit 3 are initialized.

  At this time, the CPU 20 stores the initialization state of each module in the initialization state information 44 indicating the initialization state of each module (S221). FIG. 9A is a correspondence table showing the relationship between the identification codes and the initialization states, and FIG. 9B is a data configuration diagram showing an example of the initialization state information 44. As shown in FIG. 9A, the identification code “−1” indicates that the module should not be preinitialized but is initialized, and the identification code “0” indicates that the module is not preinitialized. An identification code “1” is assigned to a state that should be pre-initialized but has not yet been assigned, and an identification code “2” is assigned to a state where the module has already been pre-initialized.

  The identification code “−1” is an identification code for unnecessarily performing the release process. For example, memory usage is generally released immediately after a module is executed, but if this module is likely to be used repeatedly, it is inefficient to repeat initialization many times. It is. Therefore, by setting an identification code “−1” for a module that is likely to be used repeatedly, and releasing the module at the timing when another module is executed, wasteful initialization is performed. Processing can be omitted.

  Then, based on the selected pattern, the module is not pre-initialized but is initialized (the module whose identification code is set to “−1” in advance) is “−1”, the module “0” is set for a module that is not pre-initialized, and “2” is set for a module that has already been pre-initialized. In addition, for example, when performing bidirectional translation from “A language to B language” and “B language to A language”, a module pattern for bidirectional is selected. For example, “1” is set for a module that should be pre-initialized but not currently performed.

  As shown in FIG. 9B, in the initialization state information 44, the state information 44b indicating the initialization state is associated with the module information 44a. The state information 44b is written with the identification code described above. For example, according to FIG. 9B, the identification code “2” (initialized state) is assigned to the language A speech recognition unit 30 (ASR_A), and the identification code “1” is assigned to the AB machine translation unit 31 (MT_AB). (A state that should be initialized but not done) is associated. In step S221, the CPU 20 updates the state information 44b for each module.

  In this manner, the CPU 20 determines whether or not an end is instructed by the user after performing the pre-initialization process for each module (S103). At this time, the CPU 20 determines that the termination has been instructed based on, for example, a predetermined input via the input device 15. When the end is instructed (Yes in S103), the CPU 20 ends the translation control process.

  If termination is not instructed (No in S103), the CPU 20 determines whether or not switching of the translation direction is instructed (S105). At this time, for example, the CPU 20 is instructed to switch when the translation from the language A to the language B is switched to the translation from the language B to the language A based on a predetermined input via the input device 15. to decide.

  When the switching of the translation direction is not instructed (No in S105), the CPU 20 determines whether or not the user has spoken in order to translate from the language A to the language B (S107). For example, the user instructs the mobile phone 1 to start voice recognition by selecting the recognition start / end button 12 displayed on the screen 12a, and selects the recognition start / end button 12 again to end the voice recognition. Instruct. The CPU 20 determines whether or not an utterance has been made based on the selection of this button, for example.

  When the utterance is made (Yes in S107), the CPU 20 performs language A speech recognition by the language A speech recognition unit 30 (S109). For example, if the user utters "Hello" in language A (Japanese) is, CPU20 generates a character string "Hello". When the CPU 20 detects a user operation indicating that machine translation processing is to be executed, the AB machine translation unit 31 machine translates the character string recognized in step S109 from language A to language B. (S111). For example, in the case where the character string "Hello" is generated by the speech recognition, CPU20 generates a string of "Hello" and then machine translation in language B (English) "Hello".

  In this case, a voice recognition string to the translation language display column 12b of the screen 12a may character string "Hello" is displayed. Moreover, it is good to display the character string of "Hello" which is the character string machine-translated in the post-translation language display column 12c of the screen 12a.

  When the CPU 20 detects a user operation indicating that the speech synthesis process is to be executed, the language B speech synthesizer 32 performs speech synthesis on the character string machine-translated in step S111 to generate speech data (S113). ). For example, in the case where the generated character string is “Hello”, voice data that generates the word “Hello” is generated. When the CPU 20 detects a user operation indicating the output of the generated audio data, the CPU 20 outputs the audio data generated in step S113 through the speaker 13 (S115). For example, the user instructs the mobile phone 1 to start audio output by selecting the output start button 12e displayed on the screen 12a. The CPU 20 may start audio output based on the selection of this button. Thereafter, the process returns to step S103, and the CPU 20 determines again whether or not the end is instructed.

  On the other hand, when switching of the translation direction is instructed (No in S105), the CPU 20 switches the translation direction and performs advance processing shown in steps S201 to S221 for each module that performs translation from language B to language A. Initialization processing is performed (S117). Thereafter, the CPU 20 determines whether or not the user has instructed termination (S119). At this time, the CPU 20 determines that the termination has been instructed based on, for example, a predetermined input via the input device 15. When the end is instructed (Yes in S119), the CPU 20 ends the translation control process.

  If termination is not instructed (No in S119), the CPU 20 determines whether or not switching of the translation direction is instructed (S121). At this time, for example, when the CPU 20 switches from translation from the language B to the language A to translation from the language A to the language B based on a predetermined input via the input device 15, the switching is instructed. to decide. If switching of the translation direction is instructed (Yes in S121), the process returns to step S101, and the CPU 20 switches the translation direction and performs pre-initialization for each module that performs translation from language A to language B. Do.

  When switching of the translation direction is not instructed (No in S121), the CPU 20 determines whether or not the user has spoken in order to translate from the language B to the language A (S123). For example, the user instructs the mobile phone 1 to start voice recognition by selecting the recognition start / end button 12 displayed on the screen 12a, and selects the recognition start / end button 12 again to end the voice recognition. Instruct. The CPU 20 determines whether or not an utterance has been made based on the selection of this button, for example.

  When the utterance is made (Yes in S123), the CPU 20 performs language B speech recognition by the language B speech recognition unit 33 (S125). For example, when the user speaks “Hello” in language B (English), the CPU 20 generates a character string “Hello”. When the CPU 20 detects a user operation indicating that machine translation processing is to be executed, the B-A machine translation unit 34 machine translates the character string recognized in step S125 from language B to language A. (S127). For example, in the case where the character string "Hello" is generated by the speech recognition, CPU20 generates a string of by machine translation "Hello" to the language A (Japanese) "Hello".

  When the CPU 20 detects a user operation indicating execution of the speech synthesis process, the language A speech synthesizer 35 performs speech synthesis on the character string machine-translated in step S127 to generate speech data ( S129). Voice data that emits the words of "Hello" is generated in the case, for example, is generated string was "Hello". When the CPU 20 detects a user operation indicating the output of the generated audio data, the CPU 20 outputs the audio data generated in step S129 through the speaker 13 (S131). For example, the user instructs the mobile phone 1 to start audio output by selecting the output start button 12e displayed on the screen 12a. The CPU 20 may start audio output based on the selection of this button. Thereafter, the process returns to step S119, and the CPU 20 determines whether or not the end is instructed again.

  In addition, although CPU20 demonstrated that each process (arithmetic process) of step S109, S111, S113, S125, S127, S129 mentioned above was performed according to a user's input operation, when there is no user's input operation, May proceed to the processing of the next step without executing each processing. Further, machine translation, speech synthesis, and speech output may be automatically executed without being based on a user input operation.

  Further, the CPU 20 performs initialization processing according to the initialization state of each module when performing each processing (calculation processing) of steps S109, S111, S113, S125, S127, and S129 described above. FIG. 10 is a flowchart showing a procedure of initialization processing at the time of calculation processing when the mobile phone 1 performs translation control processing. For example, the case where the language A speech recognition unit 30 performs speech recognition in step S109 will be described.

  As shown in FIG. 10, the CPU 20 determines the initialization state of the module of the language A speech recognition unit 30 based on the initialization state information 44 (S301). When the status information 44b is “0” (“0” in S301), the module of the language A speech recognition unit 30 is a module that should be initialized at the time of calculation without being pre-initialized. The module of the recognition unit 30 is initialized (S302).

  When the status information 44b is “1” (“1” in S301), the module of the language A speech recognition unit 30 is a module that should be pre-initialized, but is not initialized at this time. The module of the voice recognition unit 30 is initialized (S303). Then, since the language A speech recognition unit 30 has been initialized, the CPU 20 sets the state information 44b of the language A speech recognition unit 30 to “2” in the initialization state information 44 (S304).

  When the status information 44b is “2” (“2” in S301), the module of the language A speech recognition unit 30 is a module to be pre-initialized and has already been pre-initialized. Proceed to the next step without doing it. When the status information 44b is “−1” (“−1” in S301), the module of the language A speech recognition unit 30 is a module that should not be pre-initialized, but has already been initialized. The CPU 20 proceeds to the next step without performing initialization.

  Then, the CPU 20 causes the language A speech recognition unit 30 to execute arithmetic processing (that is, speech recognition processing) (S305). After execution of the arithmetic processing, the CPU 20 again determines the initialization state of the module of the language A speech recognition unit 30 as in step S301 (S307).

  When the status information 44b is “0” (“0” in S307), the module of the language A speech recognition unit 30 has been initialized in step S302, so the data expanded in the RAM 21 by the initialization is deleted from the RAM 21. As a result, the module of the language A speech recognition unit 30 is released (S309). On the other hand, when the status information 44b is “−1” or “2” (“−1”, “2” in S307), the CPU 20 ends the process without releasing the module of the language A speech recognition unit 30. . Note that a module for which the identification code “−1” is set may be released at a timing when another module is executed.

  Such processing in steps S301 to S309 is performed by the language A speech recognition unit 30, the AB machine translation unit 31, the language B speech synthesis unit 32, the language B speech recognition unit 33, the BA machine translation unit 34, and the language A. This is performed every time each module of the speech synthesizer 35 executes processing.

  As described above, the relationship between the free RAM size that can be used for speech translation and the RAM size required when pre-initializing each speech translation module, and additionally required if each module is not pre-initialized Whether to initialize each module in advance is determined based on the processing time. As a result, even with a limited RAM size, speech translation can be executed with less processing time.

  By using the frequency of use of each module for the initialization process determination process as described above, the processing time of the translation control process can be further shortened. FIG. 11 is a data configuration diagram showing an example of the history information 45 in which the usage history of each module is recorded. As illustrated in FIG. 11, the history information 45 is information stored for the last 100 modules executed recently, for example. The CPU 20 stores the executed module in the history information 45 each time one of the modules is executed (for example, when each of steps S109, S111, S113, S125, S127, and S129 is performed). . Based on the history information 45, the usage frequency of each module is determined.

  FIG. 12 is a data configuration diagram showing an example of the additional information 46 of the initialization pattern. As illustrated in FIG. 12, the additional information 46 of the initialization pattern is usage frequency information indicating a ratio of the usage frequency of the language A speech recognition unit 30 with respect to each pattern identification information 46 a common to the initialization pattern information 42. 46b, usage frequency information 46c indicating the usage frequency ratio of the AB machine translation unit 31, usage ratio information 46d indicating the usage frequency ratio of the language B speech synthesizer 32, and additional processing time reflecting the usage frequency. The extended additional processing time information 46e indicating the extended additional processing time is information associated with each other. The extended additional processing time is a sum obtained by weighting the frequency ratio to the additional processing time of each module. When the usage frequency of each module is used, the pattern is selected so that the extended additional processing time is minimized.

  For example, according to FIG. 12, the usage frequency ratio of the language A speech recognition unit 30 is “R_Aa”, the usage frequency ratio of the AB machine translation unit 31 is “R_Ma”, and the usage frequency ratio of the language B speech synthesis unit 32. Is defined as “R_Ta”. Further, extended addition processing times Tpa 'to Tpa8' for patterns 1 to 8 are defined.

  For example, consider the case where M_Aa = 10 MB, M_Ma = 10 MB, M_Ta = 1 MB, T_Aa = 8 seconds, T_Ma = 4 seconds, T_Ta = 1 second, M_Aa_w = 5 MB, M_Ma_w = 4 MB, and M_Ta_w = 3 MB. When the usage frequency of each module is not used, patterns 3 and 5 are selected from the memory constraints when the free RAM size M_max is 16 MB. The additional processing time at the time of executing the translation processing is Tpa3 = 4 seconds for pattern 3 and Tpa5 = 8 seconds for pattern 5, so that the additional processing time is the minimum for pattern 3. In other words, the language A speech recognition unit 30 (8 seconds), which requires a lot of additional processing time, is initialized in advance, and the AB machine translation unit 31 is initialized (4 seconds) at a timing that requires machine translation processing. The processing procedure to execute is selected.

  However, for example, assuming a situation where a user edits a character string obtained by speech recognition with a text editor and repeats the process of performing machine translation on the text string, initialization of machine translation (4 seconds) is performed. Repeating many times is inefficient. In order to reduce this inefficient additional processing time, the frequency ratio of each module is used.

  For example, in the above case, the frequency ratios of the language A speech recognition unit 30, the AB machine translation unit 31, and the language B speech synthesis unit 32 are R_Aa = 0.1, R_Ma = 0.8, and R_Ta = 0.1, respectively. If there is, the extended additional processing time is pattern 3, Tpa3 ′ = 4 * 0.8 = 3.2 seconds, and pattern 5 has Tpa5 ′ = 8 * 0.1 = 0.8 seconds. 5 In this way, it is possible to optimize the initialization processing timing of each module by automatically adapting to the use case of the user.

  Note that the method of weighting the frequency ratio to the additional processing time of each module is not limited to the above-described method, and may be a method of calculating the frequency ratio with a weighted sum that increases the weight of the recent history.

  Also, the modules of the language A speech recognition unit 30, the AB machine translation unit 31, the language B speech synthesis unit 32, the language B speech recognition unit 33, the BA machine translation unit 34, and the language A speech synthesis unit 35 are executed. If the frequencies are equal, it is preferable to select a pattern with reference to the additional processing time information 42f of the initialization pattern information 42.

  Further, when the AB machine translation unit 31 is not pre-initialized (state “0” or “1”) and the user edits the translation original text (the output text in the speech recognition unit) with a text editor or the like ( Since it is highly likely that the user will next execute the machine translation process (not at the timing after the editing is finished, but at the timing when the content changes even a little), the AB machine translation unit is performed in parallel processing behind the user's text editing. 31 may be initialized.

  Similarly, when the language B speech synthesizer 33 is not pre-initialized (state “0” or “1”) and the user edits the translation result (the text to be read out by the speech synthesizer) with a text editor or the like. Since it is highly likely that the user will next execute the speech synthesis process (not at the timing after the editing is finished, but at the timing when the content changes even a little), the language B speech synthesis unit is performed in parallel processing behind the user's text editing. 33 may be initialized.

  Further, in the mobile phone 1, not only the pre-initialization process for each direction but also the both directions may be performed. In this case, when the pre-initialization process for the currently selected translation direction is performed, if there is a vacant RAM size in the RAM 21, the other-direction pre-initialization process may be performed. However, the initialization pattern of the translation direction that is not currently selected is not determined based on a value obtained by removing the peak RAM size of the currently selected translation direction from the size of the RAM 21 that can be used for the translation process. The determination is made based on the size of the RAM 21 that can be used for the translation process.

  This is because pre-initialization of the translation direction that is not currently selected is necessary when the translation direction is switched. This is because it may be released when switched. If the initialization pattern of the translation direction that is not currently selected is determined based on the value obtained by removing the peak RAM size of the currently selected translation direction from the RAM size that can be used for the translation process, the translation direction is When switching, the pre-initialization pattern that shortens the processing time may not be selected. For example, there are a pattern A with an additional processing time of 8 seconds and a pattern B of 0 seconds as pre-initialization patterns in the translation direction that are not currently selected, and both are determined based on the RAM size that can be used for translation processing. It is assumed that these patterns can be selected. The translation direction is selected when only the pattern A can be selected based on a value obtained by removing the peak RAM size of the currently selected translation direction from the RAM size that can be used for the translation process. In some cases, a module in a translation direction that is not currently selected based on the pattern A is pre-initialized. However, when the translation direction is switched, the pattern that has the shortest additional processing time among the patterns that can be selected is the pattern B. Therefore, the pattern B needs to be pre-initialized. It may be necessary to release a module that has been done. The initialization pattern of the translation direction that is not currently selected as described above is inefficient if it is determined based on the value obtained by removing the peak RAM size of the currently selected translation direction from the size of the RAM 21 that can be used for the translation process. Pre-initialization may be performed, leading to an increase in processing time.

  A process for selecting an initialization pattern in the translation direction that is not currently selected based on the size of the RAM 21 that can be used for the translation process will be described in detail. For example, when the currently selected translation direction is the direction from language B to language A, the size of the RAM 21 that can be used for the translation process is 20 MB, and the initialization pattern is set in the currently selected translation direction. It is determined that the peak RAM size required for the initialization pattern is 14 MB. Further, the RAM size required for the initialization process of each module in the translation direction not currently selected (language A → language B) is M_Aa = 10 MB, M_Ma = 10 MB, M_Ta = 6 MB, and the translation direction not currently selected The RAM size required for the calculation processing of each module in (Language A → Language B) is M_Aa_w = 5BM, M_Ma_w = 4MB, M_Ta_w = 3MB. 4 seconds and T_Ta = 1 second. In this case, the selection of the initialization pattern in the translation direction that is not currently selected has a peak RAM size that falls within the range of the size of the RAM 21 that can be used for the translation process, and a pattern with a short additional initialization process time. Selected. Since the maximum value M_w of the RAM size necessary for the calculation processing of each module is max (5 MB, 4 MB, 3 MB) = 5 MB, the RAM size of the RAM 21 that can be used for the translation processing has a peak RAM size that falls within the range of 20 MB. The pre-initialization pattern is a pattern for pre-initializing any one of the modules in the translation direction not currently selected. Among these patterns, a pattern for initializing the language A speech recognition unit 30 is selected as a pattern with a short additional initialization processing time. However, when the translation direction of language B → language A is selected, a RAM size of 20 MB that can be used for translation processing is used to initialize a translation direction that is not currently selected (language A → language B). Is 6 MB excluding the peak RAM size of 14 MB in the currently selected translation direction (language B → language A). Therefore, the initialization of the language A speech recognition unit 30 that requires 10 MB cannot be executed, the initialization state corresponding to the language A speech recognition unit 30 in the initialization state information 44 is set to “1”, and the translation direction After is switched, pre-initialization is performed.

  In the above-described example, the case where the pre-initialization of the other translation direction cannot be performed when one translation direction is selected is shown. However, the case where the other translation direction can be pre-initialized when one translation direction is selected is also conceivable. For example, in the above example, when M_Ta = 1 MB, a pattern for initializing the language A speech recognition unit 30 and the language B speech synthesis unit 32 is selected as the initialization pattern. In this case, since the pre-initialization of the language B speech synthesizer 32 can be executed even when the language B → language A is selected, the pre-initialization is performed when the language B → language A is selected. Also good. In addition, if one of the translation directions can be pre-initialized when the other translation direction is selected, a module that can be pre-initialized in either translation direction is There is no need to release initialized data. For example, when the pre-initialization of the language B speech synthesizer 32 can be executed even when language B → language A is selected, even if the translation direction is switched from language A → language B to language B → language A, The initialization data of the language B speech synthesizer 32 need not be released.

  The initialization state information 44 may be updated when the translation function is activated / when the translation direction is switched / when the application screen is active.

  Further, after determining initialization timing for one direction (for example, translation from language A to language B) and performing initialization based on the timing, the other direction is generated when there is a margin in the free RAM size of the RAM 21. Initialization (for example, translation from language B to language A) may be performed. For example, while the translation from the language A to the language B is being performed in parallel, the processing is not running (for example, the CPU 20 is only performing the display processing and is waiting for the user's input operation). Sometimes, initialization in the other direction may be performed.

  Alternatively, the pre-initialization is not necessarily performed in advance. It is also possible to execute the initialization process at a timing when an arithmetic process (speech recognition process, machine translation process, etc.) is actually required, and not release the initialization data until the translation direction changes.

  In addition, when performing translation control processing in the mobile phone 1, if initialization of a module that has not been initialized is possible according to the free RAM size of the RAM 21, information for prompting initialization is displayed and initialization is performed for the user. You may be made to prompt the instruction. For example, if there is another 3 MB of free RAM size in the RAM 21, if it is possible to pre-initialize another module, the fact may be displayed to the user.

  Specifically, for example, a value obtained by adding 3 MB to the actual free RAM size M_max is calculated, and based on this value, the result of determining the pre-initialization pattern (patterns 1 to 8) by the method shown in the flowchart of FIG. It is determined whether it is the same as when 3 MB is not added (when the pre-initialization pattern is determined based on M_max). For example, when the pre-initialization pattern determined based on the value obtained by adding 3 MB to M_max is pattern 3 and the pre-initialization pattern determined based on M_max is also pattern 3, regardless of whether 3 MB is added or not. The modules that can be initialized in advance do not change. On the other hand, for example, when the pre-initialization pattern determined based on the value obtained by adding 3 MB to M_max is pattern 2 and the pre-initialization pattern determined based on M_max is also pattern 3, the free RAM size is If 3 MB can be secured, the number of modules that can be pre-initialized increases and the processing time is reduced. A value obtained by subtracting the additional processing time of the pre-initialization pattern (pattern 3) determined based on M_max from the additional processing time of the pre-initialization pattern (pattern 2) determined based on the value obtained by adding 3 MB to M_max. It is assumed that the processing time can be shortened by securing 3 MB. “If you reserve 3 MB of free RAM, the speech translation operation time can be increased by 2 seconds. "Is displayed.

  In the above example, the value added to the actual free RAM size has been described as 3 MB. However, the value is not limited to 3 MB, and may be any value. For example, a value designated in advance or a value designated by the user may be used. Further, instead of making a determination based on one type of value, a pre-initialization pattern may be determined using a plurality of values (3 MB, 5 MB, 10 MB, etc.), and the determination result may be displayed. Further, the value to be added to the actual free RAM size is gradually increased from 0 MB (for example, by 1 MB) to determine the pre-initialization pattern, and the initial value is initialized in advance from the pre-initialization pattern determined based on the real free RAM size M_max. The minimum value to be added to the substantially empty RAM size may be obtained for all patterns having many modules to be converted.

  For example, when the pre-initialization pattern determined by the real free RAM size M_max is the pattern 3, the real free RAM size M_max is selected to select the pattern 2 and the pattern 1 that have more modules to be pre-initialized than the pattern 3. Judge at least how many MBs are needed and display the additional free RAM size required to pre-initialize pattern 1 and the additional free RAM size required to pre-initialize pattern 2 To do.

  At this time, it may be displayed to the user whether or not the free RAM size of the RAM 21 can be secured by 3 MB or more by terminating any of the running application programs (tasks). Alternatively, the application program (task) being executed may be forcibly terminated to secure the free RAM size of the RAM 1. These may be configured so that the user can set the operation mode in advance.

  The information processing apparatus (mobile phone 1) according to the present invention has a stand-alone speech translation function and the initialization timing of each module even when the RAM size of the usable RAM 21 is limited. By controlling, it is possible to minimize the speech translation processing time.

  As the description of the present invention, the mobile phone 1 has been described. However, the present invention is not limited to this, and has a translation function such as a PHS (Personal Handyphone System), a PDA (Personal Digital Assistant), a portable game machine, and a portable music recording / reproducing machine. Any information processing apparatus may be used as long as it is an information processing apparatus. Further, the present invention can be applied to all low-spec devices, and is not limited to portable terminals, and can be applied to general embedded devices and general electric devices such as televisions and personal computers.

  DESCRIPTION OF SYMBOLS 1 ... Mobile phone, 10 ... Upper housing | casing, 11 ... Lower housing | casing, 12 ... Display apparatus, 12a ... Screen, 12b ... Language display column before translation, 12c ... Language display column after translation, 12d ... Recognition start / end button, 12e ... Output start button, 13 ... Speaker, 14 ... Microphone, 15 ... Input device, 20 ... CPU, 21 ... RAM, 22 ... ROM, 23 ... Communication device, 23a ... Antenna, 30 ... Language A speech recognition unit (ASR_A) , 31 ... AB machine translation unit (MT_AB), 32 ... language B speech synthesis unit (TTS_B), 33 ... language B speech recognition unit (ASR_B), 34 ... BA machine translation unit (MT_B), 35 ... language A speech synthesizer (TTS_A), 40 ... module list information, 41 ... initialization time information, 42 ... initialization pattern information, 43 ... correspondence table showing relationship between identification codes and initialization states, 44 ... Synchronize state information, 45 ... history information 46 ... additional information initialization pattern.

Claims (6)

An information processing apparatus having a plurality of modules related to translation processing,
A first calculating means for calculating an available RAM area that can be used for the translation process when the translation process is executed;
The free space and the RAM size required when each of the plurality of modules performs processing, the RAM size occupied when each of the plurality of modules is initialized in advance, and each of the plurality of modules are determined in advance. Selection means for selecting a module that can be initialized in advance from the plurality of modules based on the time required for initialization
First initialization means for initializing a module selected by the selection means before translation processing is executed;
When any module performs processing in accordance with the execution of the translation processing, if this module has not been initialized by the first initialization means, the second module that initializes this module at the start of processing of this module Initialization means;
An information processing apparatus comprising:   The selection means is a module or a plurality of modules that can be initialized in advance in a free RAM area that can be used for translation processing, and the module or modules that have the shortest additional processing time are initialized in advance. The information processing apparatus according to claim 1, wherein the information processing apparatus is selected as a module that performs conversion.   The information processing apparatus according to claim 1, wherein the plurality of modules include a module that performs speech recognition, a module that performs machine translation, and a module that performs speech synthesis. Storage means for storing the frequency of use of a plurality of modules related to the translation processing for each module;
The selecting means selects one module or a plurality of modules having the shortest additional processing time based on the time required for the modules to perform initialization in advance and the frequency of use of each of the plurality of modules. The information processing apparatus according to claim 2, wherein the determination is made. The translation process is a bidirectional translation process,
A translation direction selection means for selecting a direction of translation processing;
The selection means is a RAM free space that can be used for the translation processing calculated by the first calculation means to select a module to be initialized in advance among a plurality of modules used for translation processing in a direction not selected. The information processing apparatus according to claim 1, wherein the information processing apparatus uses a region.   The first initialization means is a module used for translation processing in a direction not selected when translation processing is executed, and is a module that performs initialization in advance selected by the selection means. Initialize a module that can be initialized while executing a translation process in the selected direction, and a module that cannot be initialized while executing a translation process in the selected direction 6. The information processing apparatus according to claim 5, wherein prior initialization is performed after a direction is selected.

Images