JP5643545B2 - Communication terminal and e-mail reading method - Google Patents

Description

  The present invention relates to a communication terminal and a mail reading method.

  There is known a communication terminal having a function of reading a text included in a received e-mail, a WEB page or the like by voice for the purpose of assisting a low vision user. Japanese Patent Application Laid-Open No. H10-228561 proposes a technique for collating image data received by data broadcasting with a character pattern, extracting characters included in the image data, and reading them out.

JP 2006-33741 A

  In some cases, image data is attached to an e-mail in addition to the body text. In many cases, image data attached to an e-mail is a photograph of a person or a landscape associated with the content of the text, and is indispensable for fully understanding the content of the e-mail. However, in the above-described conventional technology, even if the text of the e-mail can be read out, it is impossible to read out the person or the landscape in the attached image data.

  The present invention has been made in view of the above, and an object of the present invention is to provide a communication terminal and a mail reading method that can read out the contents of image data attached to an e-mail by voice.

  In order to solve the above-described problems and achieve the object, the present invention is a communication terminal, which is a communication unit that receives an e-mail to which image data is attached, and an e-mail that is received by the communication unit A data generation unit that analyzes the image data that has been generated and generates image read-out data for reading out an object included in the image data by voice; and a voice that reads out the image read-out data generated by the data generation unit by voice And a reading section.

  In addition, the data creation unit creates the image reading data by analyzing the image data attached to the email while the voice reading unit reads the body of the email received by the communication unit. Preferably, the speech reading unit reads the image reading data created by the data creation unit after the reading of the text is completed.

  The data creation unit analyzes the image data attached to the e-mail received by the communication unit, extracts feature data of the object included in the image data, and uses the extracted feature data It is preferable that the type or name of the object is specified by making an inquiry to the server, and the image reading data is created based on the specified type or name.

  Further, the data creating unit analyzes the importance of the plurality of objects extracted by analyzing the image data attached to the e-mail received by the communication unit at least for the type, position, and range of each object. It is preferable to make the determination based on a determination criterion related to one, and to create the image reading data so that an object having high importance is read first.

  Further, the data creation unit determines whether or not the type or name of the object extracted by analyzing the image data attached to the email received by the communication unit is included in the body or subject of the email It is preferable to further use as a criterion for determining the importance of the object.

  In addition, the data creation unit analyzes the image data attached to the email received by the communication unit, and the email address of the person extracted as one of the objects is the sender address of the email It is preferable to further use whether or not they match as a criterion for determining the importance of the object corresponding to the person.

  In addition, when a plurality of image data is attached to the e-mail received by the communication unit, the data creation unit is configured to add an object of the same type or name from another image data attached to the e-mail. It is preferable to further use whether or not it has been extracted as a criterion for determining the importance of the object.

  In addition, the data creation unit, the plurality of objects extracted by analyzing the image data attached to the e-mail received by the communication unit, grouped based on the type or name and the importance, It is preferable to create the image reading data so that objects belonging to the same group are read as one sentence or a series of sentences.

  In addition, when a plurality of image data is attached to the e-mail received by the communication unit, the data creation unit creates the image reading data including an explanation of a correspondence relationship between the plurality of image data. Is preferred.

  In order to solve the above-described problems and achieve the object, the present invention is a communication terminal, which is a communication unit that receives image data from an external device, and that analyzes the image data received by the communication unit. A data creation unit that creates image reading data for reading out the object included in the image data by voice, and a voice reading unit that reads the image reading data created by the data creation unit by voice. It is characterized by.

  In order to solve the above-described problems and achieve the object, the present invention is a mail reading method executed by a communication terminal having an e-mail receiving function, wherein the communication terminal is attached with image data. A receiving step for receiving the e-mail, and an image reading for the communication terminal to analyze the image data attached to the e-mail received in the receiving step and to read out the content included in the image data by voice A data creating step for creating data, and a voice reading step in which the communication terminal reads out the image reading data created in the data creating step by voice.

  INDUSTRIAL APPLICABILITY The communication terminal and the mail reading method according to the present invention have an effect that the contents reflected in the image data attached to the electronic mail can be read out by voice.

It is a block diagram which shows schematic structure of the function of a communication terminal. It is a figure which shows the example of a display of the email containing image data. It is a figure which shows an example of address book data. It is a figure which shows an example of importance determination data. It is a figure which shows an example of analysis data. It is a figure which shows an example of image reading-out data. It is a sequence diagram showing a flow of a reading process of an e-mail attached with image data. It is a flowchart which shows the whole flow of an image reading data creation process. It is a flowchart which shows the process sequence of a person analysis process. It is a flowchart which shows the process sequence of an object analysis process. It is a flowchart which shows the process sequence of an importance calculation process. It is a flowchart which shows the process sequence of a grouping process. It is a flowchart which shows the process sequence of a data output process.

  Hereinafter, the present invention will be described in detail with reference to the drawings. The present invention is not limited to the following description. In addition, constituent elements in the following description include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those in a so-called equivalent range. In the following, a mobile phone is taken as an example of a communication terminal, but the application target of the present invention is not limited to a mobile phone. The present invention can also be applied to, for example, a PHS (Personal Handy-phone System), a personal computer, a portable information terminal, and the like.

  First, a schematic configuration of functions of the communication terminal 10 will be described. FIG. 1 is a block diagram showing a schematic configuration of functions of the communication terminal 10. The communication terminal 10 is a mobile phone having a wireless communication function and an e-mail voice reading function. As illustrated in FIG. 1, the communication terminal 10 includes a control unit 22, a storage unit 24, a communication unit 26, an operation unit 28, an audio processing unit 30, and a display unit 32.

  The control unit 22 is a processing unit that centrally controls the overall operation of the communication terminal 10, for example, a CPU (Central Processing Unit). That is, the communication unit 26, the display unit 32, and the like so that various processes of the communication terminal 10 are executed in an appropriate procedure according to the operation of the operation unit 28 and the software stored in the storage unit 24 of the communication terminal 10. To control the operation. The control unit 22 executes processing based on a program (for example, an operating system program, application program, etc.) stored in the storage unit 24. The control unit 22 can execute a plurality of programs (applications, software) in parallel.

  The control unit 22 executes an e-mail reading process as one of various processes. The control unit 22 includes a reading data creation unit 22a and a voice reading unit 22b as functional units for realizing an e-mail reading process. The reading data creation unit 22a and the voice reading unit 22b are realized by the control unit 22 executing a predetermined program.

  The reading data creation unit 22a analyzes the image data attached to the e-mail and creates image reading data for reading an object included in the image data. An object here is a person, an animal, a plant, a building, a machine, a mountain, etc., for example.

  The voice reading unit 22b reads the text of the email and the image reading data created by the reading data creation unit 22a by voice. As described above, the communication terminal 10 reads out not only the text of the email but also the object included in the image data attached to the email. Note that the voice reading by the voice reading unit 22b may be realized using any technique.

  The memory | storage part 24 is a memory | storage device which memorize | stores various information, for example, memorize | stores information in semiconductors, such as flash memory. The storage unit 24 stores application programs and data used for processing in the control unit 22. Specifically, the storage unit 24 stores, as an application program, a mail reading program 24a for realizing the reading data creation unit 22a and the voice reading unit 22b. The mail reading program 24a may be divided into a plurality of programs, such as a program for realizing the reading data creation unit 22a and a program for realizing the voice reading unit 22b.

  The storage unit 24 also stores, as data, e-mail 24b, address book data 24c, importance determination data 24d, analysis data 24e, image reading data 24f, and the like.

  The email 24b is an email received by the communication terminal 10. The e-mail 24b may have one or more image data attached in addition to a header portion including a subject, a transmission source address, and the like, and a text. A display example of the e-mail 24b to which image data is attached is shown in FIG. In the example shown in FIG. 2, the subject S1 and the source address D1 are displayed at the top of the display unit 32, and the text T1, image data P1, text T2, and image data P2 are displayed below it.

  The layout of the email 24b to be displayed in FIG. 2 is defined in the HTML format, and the communication terminal 10 displays the text of the email and the image data attached to the email according to the definition. ing. Note that e-mails with attached image data do not necessarily have a layout defined in HTML format or the like, and e-mails with no layout defined display image data one by one under the body text. Is displayed in the default format.

  The address book data 24c is data in which telephone numbers and the like of the user of the communication terminal 10 are registered in advance. An example of the address book data 24c is shown in FIG. As shown in the example of FIG. 3, in the address book data 24c, a name, a telephone number, a mail address, and a face image are registered for each user acquaintance. The face image is image data in which the face of the user's acquaintance is photographed, and is displayed on the display unit 32, for example, to indicate the caller when a call or e-mail is received.

  The importance determination data 24d is data in which information for determining the importance of an object included in image data is registered in advance. An example of the importance determination data 24d is shown in FIG. As shown in the example of FIG. 4, the importance determination data 24d includes a plurality of determinations regarding the type of object, the area occupied, the distance from the center, the degree of relevance to the e-mail, the appearance frequency in other image data, and the like. The condition and the importance value to be added when the determination condition is satisfied are registered. Note that the determination condition of the importance determination data 24d shown in FIG. 4 and the importance value are both examples, and may be changed as appropriate.

  The analysis data 24e is data in which an analysis result of an object included in the image data is stored. An example of the analysis data 24e is shown in FIG. As shown in the example of FIG. 5, the analysis data 24e includes, for each object included in the image data, an image number, object number, contour, feature data, type, name, accuracy, email address, facial expression, gender, age, Stores importance, group number, and the like.

  The image number is a number for identifying the image data that includes the object. The object number is a number for identifying the object. The contour is information representing the shape, position, and range (size) of the object in the image data. The outline may be, for example, information indicating the shape, position, and range of an object by a set of vectors, or the shape, position, and range of an object may be determined by a combination of an expression indicating a shape such as a square or a circle, position information, and dimensions. It may be the information shown. The feature data is information indicating the feature of the object in the image data, and is, for example, color or texture data.

  The type is the type name of the object determined by the analysis, and the name is the proper noun or product name of the object determined by the analysis. The accuracy is a value representing the accuracy of the determination result of the type or name. The mail address is the mail address of a person corresponding to the object, and is set when it is determined that the object is a specific person. The expression, gender, and age are information indicating the analysis result of the person corresponding to the object, and are set when the object is determined to be human.

  The importance is a value indicating the importance of the object, and is calculated based on the importance determination data 24d. The group number is a number indicating to which group the objects are grouped by grouping performed based on the strength of association.

  The image reading data 24f is data used by the voice reading unit 22b to read the content of the image data, and is created based on the analysis data 24e. An example of the image reading data 24f is shown in FIG. In the example shown in FIG. 6, the contents of the image data are stored in the image reading data 24f as plain text. The format of the image reading data 24f does not need to be plain text as shown in FIG. 6, and may be any format suitable for the voice reading unit 22b to read out.

  The communication unit 26 includes an antenna 26a, establishes a radio signal line by a CDMA system or the like with a base station via a channel assigned by the base station, and performs telephone communication and information communication with the base station. . When the communication unit 26 receives an email, the control unit 22 stores the received email in the storage unit 24 as an email 24b.

  The operation unit 28 includes various keys. When these keys are input by a user operation, a signal corresponding to the operation content is generated. The generated signal is input to the control unit 22 as a user instruction.

  The audio processing unit 30 executes processing of an audio signal input to the microphone 15 and an audio signal output from the receiver 16. The display unit 32 includes a display panel composed of a liquid crystal display (LCD), an organic EL (Organic Electro-Luminescence) panel, and the like, and video and images corresponding to video data supplied from the control unit 22. An image corresponding to the data is displayed on the display panel.

  Next, the operation of the communication terminal 10 when reading an e-mail attached with image data will be described with reference to FIG. FIG. 7 is a sequence diagram showing a flow of reading processing of an e-mail attached with image data.

  As shown in the example of FIG. 7, when the communication terminal 10 is instructed to read out an electronic mail to which image data is attached, the voice reading unit 22b starts reading out the text of the electronic mail. At the same time, the read-out data creation unit 22a executes image read-out data creation processing. In the image reading data creation process, a process for analyzing the image data attached to the e-mail and creating the image reading data 24f is performed.

  When the voice reading unit 22b completes reading the body of the e-mail, the voice reading unit 22b starts reading the image reading data 24f created by the reading data creation unit 22a. In this way, by executing the image reading data creation process while reading the text of the e-mail, the user can be aware that the image reading data creation process is being executed, and the text and image data. Can be read continuously.

  Next, the image reading data creation process shown in FIG. 7 will be described with reference to FIGS. FIG. 8 is a flowchart showing the overall flow of image reading data creation processing. As shown in FIG. 8, in the image reading data creation process, first, the reading data creation unit 22a extracts and analyzes a person out of the objects included in the image data attached to the e-mail in step S10. The person analysis process is executed. Subsequently, the read-out data creation unit 22a executes an object analysis process for extracting and analyzing an object other than a person among objects included in the image data attached to the e-mail in step S11.

  Then, the read-out data creation unit 22a executes an importance calculation process for calculating the importance of each object extracted from the image data in step S12, and in step S13, the object is determined based on the strength of the relation. Execute grouping processing for grouping. Then, the read-out data creating unit 22a executes a data output process for outputting the image read-out data 24f as step S14.

  FIG. 9 is a flowchart showing a processing procedure of the person analysis processing shown in FIG. As shown in FIG. 9, the read-out data creation unit 22a first extracts all face areas of a person included in the image data attached to the e-mail as step S20. Note that the face area extraction may be realized using any face detection technique.

  Then, in step S21, the reading data creation unit 22a selects one unselected face area from the extracted face areas. When an unselected face area can be selected (step S22, Yes), the reading data creation unit 22a performs facial expression analysis as step S23 and gender analysis as step S24 for the selected face area, In step S25, the age is analyzed. The analysis of facial expression, sex and age may be realized using any image analysis technique.

  Then, the read-out data creation unit 22a collates the selected face area with the face image of the address book data 24c in step S26. Here, when a face image that matches with a certain degree of accuracy is registered in the address book data 24c (step S27, Yes), the read-out data creation unit 22a, as step S28, the name and mail corresponding to the face image The address is acquired as personal information of a person corresponding to the selected face area.

  Then, in step S29, the reading data creation unit 22a adds the analysis result obtained for the selected face area to the analysis data 24e in association with the outline of the selected face area.

  Through the above processing, for example, the face area of the person OB1 included in the image data P1 shown in FIG. 2 is analyzed in step S23 as the expression “smile”. In step S24, the gender is “male”. In step S25, it is analyzed that the age is “20s”. Further, for example, the face area of the person OB1 matches the face image in the first row of the address book shown in FIG. 3 with an accuracy of “90%” in step S26, and “Taro Tanaka” as the name in step S28. Is acquired, and “taro@example.com” is acquired as the mail address.

  In this case, as shown in the first line of the example of the analysis data 24e shown in FIG. 5, “Human” is set as the type of analysis, “Taro Tanaka” is set as the name, and “90%” is set as the accuracy. Is set, “taro@example.com” is set as the mail address, “smile” is set as the facial expression, “male” is set as the gender, and “20s” is set as the age. If the corresponding face image is not registered in the address book data 24c, the name and e-mail address are blank as shown in the third line of the example of the analysis data 24e shown in FIG. In the extraction of the face area in S20, the accuracy determined that the area is a person's face is set.

  After adding information to the analysis data 24e, the reading data creation unit 22a re-executes Step S21 and the subsequent steps. When it becomes impossible to select an unselected face area, that is, when the analysis of all the extracted face areas has been completed (No in step S22), the reading data creation unit 22a ends the person analysis process. .

  FIG. 10 is a flowchart showing the processing procedure of the object analysis processing shown in FIG. As shown in FIG. 10, the reading data creation unit 22a first extracts all the outlines of the objects included in the image data attached to the e-mail in step S40. Then, in step S41, the reading data creation unit 22a excludes the contour corresponding to the person extracted by the person analysis process from the extracted contours. Note that the extraction of the contour of the object in step S40 may be realized using any image analysis technique.

  Then, in step S42, the reading data creation unit 22a selects one unselected contour from the extracted contours. When an unselected contour can be selected (step S43, Yes), the read-out data creation unit 22a acquires feature data such as the color and texture data of the region surrounded by the selected contour as step S44. Subsequently, in step S45, the read-out data creation unit 22a transmits the contour and feature data to a predetermined server via wireless communication, and inquires about what object is present in the region surrounded by the contour.

  The inquiry destination server is an apparatus that stores a huge number of object types, contours, and feature data in association with each other. This server stores names associated with contours and the like for objects having names.

  If the server has information on an object that matches the contour and feature data with a certain degree of accuracy (Yes in step S46), the read-out data creation unit 22a, in step S47, includes detailed information on the object such as type and name. Get verification accuracy from the server.

  Then, in step S48, the reading data creation unit 22a adds the analysis result obtained for the selected contour to the analysis data 24e in association with the selected contour.

  Through the above processing, for example, the object included in the outline of the object OB4 included in the image data P1 shown in FIG. 2 has the accuracy of “90%”, the type is “mountain”, and the name by the inquiry to the server. Is found to be “Mt. Fuji”. In this case, as shown in the fourth line of the example of the analysis data 24e shown in FIG. 5, as the analysis result, “Mountain” is set as the type, “Mt. Fuji” is set as the name, and “90%” is set as the accuracy. Is set.

  When the name is not specified by the server inquiry, the name is blank as in the fifth line in the example of the analysis data 24e shown in FIG. When the type is not specified by the server inquiry, the type is “unknown”.

  After adding information to the analysis data 24e, the reading data creation unit 22a re-executes Step S42 and the subsequent steps. Then, when it becomes impossible to select an unselected area, that is, when analysis of all the areas to be processed has been completed (No in step S43), the reading data creation unit 22a ends the object analysis process.

  FIG. 11 is a flowchart showing a processing procedure of importance calculation processing shown in FIG. As shown in FIG. 11, the read-out data creation unit 22a first initializes a variable i for identifying an object registered in the analysis data 24e to 1 in step S60.

  Then, the read-out data creation unit 22a selects the i-th object registered in the analysis data 24e as step S61. When the i-th object can be selected (step S62, Yes), the read-out data creation unit 22a refers to the importance determination data 24d as step S63, and the importance according to the type and name of the i-th object Get the value of.

  Specifically, when the importance determination data 24d is as shown in the example shown in FIG. 4, the reading data creation unit 22a determines the type of the i-th object among the determination conditions whose classification is “type”. And the name satisfying the condition is selected, and the importance value corresponding to the selected determination condition is acquired from the importance determination data 24d.

  For example, as in the case of the object in the first line of the analysis data 24e shown in FIG. 5, when the type of the i-th object is “human” and the name is “Taro Tanaka”, the type and name are As a matching judgment condition, a judgment condition of classification “type” and condition “human (name is specified)” is selected, and “10” is acquired as an importance value corresponding to this judgment condition. .

  Further, for example, when the type of the i-th object is “human” and the name is not specified as in the third row object in the example of the analysis data 24e illustrated in FIG. As a condition, a determination condition that the classification is “type” and the condition is “human (no name is specified)” is selected, and “4” is acquired as the importance value corresponding to this determination condition.

  Further, for example, when the type of the i-th object is “mountain” like the object in the fourth line in the example of the analysis data 24e illustrated in FIG. 5, the classification is set as the determination condition that matches the type and the name. A determination condition of “type” and a condition of “mountain” is selected, and “4” is acquired as an importance value corresponding to this determination condition.

  As in the example of the importance determination data 24d shown in FIG. 4, the importance value corresponding to the type and name should be set higher as it seems that the e-mail recipient is more interested like a human being. Is preferred. Further, even for humans, it is preferable that an acquaintance registered in the address book data 24c is set to have a higher importance value than a stranger. By setting the importance value in this way, it is possible to preferentially read out objects that are important for the e-mail recipient among objects included in the image data.

  Subsequently, in step S64, the read-out data creation unit 22a refers to the importance determination data 24d, and acquires the importance value corresponding to the exclusive area of the i-th object.

  Specifically, when the importance determination data 24d is as shown in the example shown in FIG. 4, the read-out data creation unit 22a determines the i-th object among the determination conditions whose classification is “exclusive area”. The exclusive area satisfying the condition is selected, and the importance value corresponding to the selected determination condition is acquired from the importance determination data 24d. In the example of the importance determination data 24d shown in FIG. 4, it is assumed that the exclusive area is calculated as a ratio of the area surrounded by the outline to the entire area of the image data. This is not necessary.

  As in the example of the importance determination data 24d shown in FIG. 4, the importance value corresponding to the exclusive area is preferably set higher as the exclusive area is larger. In general, a photographer takes a picture so that a subject to be photographed is large. For this reason, by setting a higher importance value for an object having a larger exclusive area, it is possible to increase a possibility that an object intentionally photographed by the photographer is preferentially read out.

  Subsequently, in step S65, the read-out data creation unit 22a refers to the importance determination data 24d, and acquires the importance value corresponding to the distance from the center of the image data of the i-th object.

  Specifically, when the importance level determination data 24d is as shown in the example shown in FIG. 4, the reading-out data creation unit 22a is the i-th of the determination conditions whose classification is “distance from the center”. The object satisfying the distance between the object and the center of the image data is selected, and the importance value corresponding to the selected determination condition is acquired from the importance determination data 24d. In the example of the importance determination data 24d shown in FIG. 4, it is assumed that the distance from the center is calculated as a relative value when the length of the long side of the image data is 1. The distance from the center does not have to be this way.

  As in the example of the importance determination data 24d shown in FIG. 4, the importance value corresponding to the distance from the center is preferably set higher as the distance from the center is shorter. In general, a photographer takes a picture so that an object to be photographed is captured near the center. For this reason, by setting a higher importance value for an object having a shorter distance from the center of the image data, it is possible to increase a possibility that an object intentionally photographed by the photographer is preferentially read out.

  Subsequently, in step S66, the read-out data creating unit 22a collates the analysis result such as the type, name, and mail address of the i-th object with the subject, body, and source mail address of the email. Then, in step S67, the read-out data creation unit 22a refers to the importance determination data 24d, determines the importance value according to the collation result, and the importance according to the degree of association with the e-mail of the i-th object. Get the value of.

  Specifically, when the importance determination data 24d is as in the example illustrated in FIG. 4, the read-out data creation unit 22a determines that i among the determination conditions whose classification is “degree of association with e-mail”. The one that satisfies the condition of the type, name, or mail address of the second object is selected, and the importance value corresponding to the selected determination condition is acquired from the importance determination data 24d.

  In addition, when there are a plurality of determination conditions that satisfy the condition, name, or mail address of the i-th object, the sum of importance values corresponding to the determination conditions that satisfy the condition is determined according to the degree of association with the e-mail. May be acquired as the importance value, or the maximum importance value corresponding to each determination condition that satisfies the condition may be acquired as the importance value according to the degree of association with the e-mail. Good.

  For example, like the object in the fourth line in the example of the analysis data 24e shown in FIG. 5, the type of the i-th object is “mountain” and the name is “Mt. Fuji”, the subject of the email, Assume that the source address and the text are as shown in FIG. Further, the sum of importance values corresponding to the determination conditions that satisfy the condition is acquired as the importance value corresponding to the degree of association with the e-mail. Further, when there are a plurality of determination conditions that satisfy the condition of the name, the determination conditions that satisfy the condition of the type are not searched.

  In this case, since “Mt. Fuji” is included in the subject S1, the condition “the name is included in the subject” is satisfied, and “10” is acquired as the importance value corresponding to the determination condition that satisfies this condition. . Since “Mount Fuji” is included in the text T1, the condition “name is included in the text” is satisfied, and “8” is acquired as the importance value corresponding to the determination condition that satisfies this condition. As a result, “18”, which is the sum of “10” and “8”, is acquired as the importance value corresponding to the degree of association with the e-mail.

  In general, the sender of an e-mail refers to the subject or text of the e-mail about what he / she wants to convey with image data attached to the e-mail. For this reason, when the object type or name is included in the subject or body of an email, the importance of the object is highly evaluated, so that the object to be conveyed in the image data attached by the sender of the email has priority. It is possible to increase the possibility of being

  Further, for example, as in the first line object in the example of the analysis data 24e shown in FIG. 5, the type is “human”, the name is “Taro Tanaka”, and the mail address is “taro@example.com”. Suppose that the subject, source address, and text of the e-mail are as shown in FIG.

  In this case, since “taro@example.com” matches the source address D1, the condition that “the mail address matches the source address” is satisfied, and the importance level corresponding to the determination condition that satisfies this condition is satisfied. “6” is acquired as the value. As a result, “6” is acquired as the importance value corresponding to the degree of association with the e-mail.

  In general, when an e-mail sender wants to convey his / her state by e-mail, the image data of the e-mail is often attached to the e-mail. For this reason, by evaluating the importance of an object whose mail address matches the sender address, it is possible to increase the possibility that the object to be conveyed by the image data attached by the sender of the email is preferentially read out. it can.

  Subsequently, in step S68, the read-out data creation unit 22a refers to the importance determination data 24d, and acquires the importance value corresponding to the number of appearances of the i-th object in the other image data.

  Specifically, when the importance determination data 24d is as shown in the example shown in FIG. 4, the reading data creation unit 22a determines that the classification is “number of appearances in other image data”. , The one that satisfies the condition of the number of appearances in the other image data of the i-th object is selected, and the importance value corresponding to the selected determination condition is acquired from the importance determination data 24d. In the example of the importance determination data 24d shown in FIG. 4, it is assumed that the number of objects extracted from other image data and having the same name is acquired as the number of appearances in other image data. However, the number of objects extracted from the other image data and having the same type may be acquired as the number of appearances in the other image data.

  As in the example of the importance determination data 24d illustrated in FIG. 4, it is preferable that the importance value corresponding to the number of appearances in other image data is set higher as the number of appearances increases. In general, when an e-mail sender wants to convey something about an e-mail, he or she may attach a plurality of image data showing what he / she wants to convey to the e-mail. For this reason, by evaluating the importance of objects included in a large amount of image data, it is possible to increase the possibility that an object to be conveyed by image data attached by the sender of the e-mail is preferentially read out.

  Subsequently, in step S69, the reading data creation unit 22a records the sum of importance values acquired for the i-th object in the analysis data 24e in association with the i-th object. In step S70, the read-out data creation unit 22a adds 1 to the variable i, and re-executes step S61 and the subsequent steps.

  When the i-th object can no longer be selected, that is, when the calculation of importance of all objects has been completed (No in step S62), the reading data creation unit 22a ends the importance calculation processing. . Note that the importance calculation method described above is merely an example, and some determination conditions may not exist, or other determination conditions may be added.

  FIG. 12 is a flowchart showing a processing procedure of the grouping process shown in FIG. As shown in FIG. 12, the read-out data creation unit 22a first initializes a variable i for identifying an object registered in the analysis data 24e to 1 in step S80.

  Then, the read-out data creation unit 22a selects the i-th object registered in the analysis data 24e as step S81. When the i-th object can be selected (Yes in step S82), the reading-out data creation unit 22a determines in step S83 whether the i-th object has been grouped, that is, sets a group number for the i-th object. It is determined whether it has been completed. If the i-th object has already been grouped (step S83, Yes), the read-out data creation unit 22a adds 1 to the variable i as step S89, and re-executes step S81 and subsequent steps.

  On the other hand, when the i-th object has not been grouped (No in step S83), the read-out data creation unit 22a determines whether the type of the i-th object is known in step S84. Here, when the type of the i-th object is not known (step S84, No), the read-out data creation unit 22a adds 1 to the variable i as step S89, and re-executes step S81 and the subsequent steps.

  On the other hand, when the type of the i-th object is known (Yes in step S84), the read-out data creation unit 22a, in step S85, has the same type and importance from the objects extracted from the same image data. An object whose difference is within a predetermined range is selected. If no object is selected (No at Step S86), the read-out data creation unit 22a adds 1 to the variable i as Step S89, and re-executes Step S81 and the subsequent steps.

  On the other hand, when one or more objects are selected (step S86, Yes), the read-out data creation unit 22a issues a new group number as step S87, and the issued group number as step S88. Are recorded in the analysis data 24e in association with the object selected in step (i) and the i-th object. Then, the read-out data creation unit 22a adds 1 to the variable i in step S89, and re-executes step S81 and the subsequent steps.

  When the i-th object can no longer be selected, that is, when the grouping of all objects has been completed (No in step S82), the read-out data creation unit 22a ends the grouping process.

  For example, consider the objects OB6, OB7, OB8, and OB9 included in the image data P2 shown in FIG. The person corresponding to the objects OB6, OB7, and OB8 is a family, but the person corresponding to the object OB9 is another person. Here, in order to simplify the description, the importance is calculated based only on the distance from the center of the image.

  In this case, all of the objects OB6, OB7, OB8, and OB9 are determined to be “human”, but the objects OB6, OB7, and OB8 are substantially equal in distance from the center of the image data. In contrast, the object OB9 is evaluated to be less important because the object OB9 is far from the center of the image data. Therefore, the objects OB6, OB7, and OB8 are grouped into the same group, but the object OB9 is not grouped. In this way, by performing grouping based on the type and importance, it is possible to group strongly related objects.

  FIG. 13 is a flowchart showing a processing procedure of the data output process shown in FIG. As shown in FIG. 13, the read-out data creation unit 22a first initializes a variable j for identifying image data attached to an e-mail to 1 in step S100. Then, in step S101, the reading data creation unit 22a determines whether the value of the variable j is larger than the number of image data attached to the e-mail.

  When the value of the variable j is less than or equal to the number of image data attached to the e-mail (step S101, No), the read-out data creation unit 22a has a plurality of image data attached to the e-mail as step S102. It is determined whether or not. If there is a plurality of image data attached to the e-mail (step S102, Yes), the read-out data creation unit 22a indicates the correspondence between the jth image data and other image data as step S103. The text is output to the image reading data 24f.

  The sentence indicating the correspondence between the image data and the other image data is, for example, sentences such as sentence T3 and sentence T6 in the example of the image reading data 24f shown in FIG. 6, and what is the image data? It is also for explaining where the image data is to be displayed in a defined layout. By outputting such a sentence, the listener can grasp which image data is currently being read out.

  Subsequently, in step S104, the read-out data creation unit 22a acquires from the analysis data 24e all rows that hold information on the object extracted from the j-th image data. Then, the read-out data creation unit 22a sorts the acquired rows in descending order of importance in step S105.

  After completion of the sorting, the read-out data creation unit 22a initializes a variable k for identifying one of the sorted rows to 1 in step S106. In step S107, the read-out data creation unit 22a selects the kth row from the sorted rows, that is, the kth row having the highest importance value.

  Here, when the k-th row can be acquired (step S108, Yes), the reading data creation unit 22a determines whether the importance value of the acquired row is larger than a predetermined value as step S109. judge. When the importance value of the acquired line is larger than a predetermined value (Yes at Step S109), the reading data creating unit 22a sets the acquired line contents to the image reading data 24f as Step S110. It is determined whether it has been output.

  When the content of the acquired line has not been output to the image reading data 24f (No at Step S110), the reading data creating unit 22a determines whether or not a group number is set for the acquired line, as Step S111. Then, it is determined whether or not the object corresponding to the acquired row belongs to the same group as other objects.

  When the group number is set to the acquired line (step S111, Yes), the reading data creation unit 22a sets the same group number as the k-th line and the content of the k-th line as step S112. The contents of the other lines are output to the image reading data 24f in one sentence or a series of sentences. When a group number is not set for the acquired line (step S111, No), the read-out data creation unit 22a outputs the contents of the k-th line to the image read-out data 24f in one sentence as step S113. Here, a series of sentences refers to a plurality of sentences connected by a conjunction indicating that the contents of the sentences are continued, such as “and” and “further”.

  The output to the image reading data 24f includes the type and name of the identified object, the shape, position and range of the object derived from the contour, the color of the object derived from the feature data, and the human expression obtained by the analysis. It is performed by combining expressions derived from sex, age, etc. The output to the image reading data 24f will be described in more detail with reference to FIGS. 2, 5, and 6. FIG.

  The sentence T4 “Large Mt. Fuji is reflected in the center” shown in FIG. 6 is the output of the analysis result of the object OB4. The sentence T4 is output by combining “Mt. Fuji” identified as the name and “center” and “large” words derived from the position and range of the contour.

  In addition, the sentence T5, “Taro Tanaka, Hanako Tanaka, and a boy are smiling with a smile in the vicinity of the center” is an analysis of the objects OB1, OB2, and OB3 grouped into a group with a group number of “G1”. The result is output. Sentence T5 consists of the word “boy” derived from the combination of “Taro Tanaka” and “Hanako Tanaka” identified as names, the type of “human”, the sex of “male”, and the age of “child”. The word “near the center” derived from the position of “and the word“ smile ”that is the analysis result of the facial expression is output in combination. The same applies to the sentence T7.

  Further, a sentence T8 that “a thing like a temple is slightly larger on the left” is an output of the analysis result of the object OB10. The sentence T8 is output by combining the word “Tera” specified as the type and the words “left” and “somewhat larger” derived from the position and range of the contour.

  Note that in the sentence T8, “thing like a temple” instead of “temple is” is because the accuracy of the OB10 is relatively low at “60%”. In this way, for an object whose accuracy value is lower than a certain value, an ambiguous expression such as “something” can reduce the possibility of misunderstanding the listener. For an object with low accuracy or an object for which the type could not be specified, a sentence may be output by specifying the object not by name or type but by shape or color, such as “red and round”.

  After completing the output in step S112 or step S113, the read-out data creation unit 22a adds 1 to the variable k as step S114, and re-executes step S107 and subsequent steps. Also, if the content of the acquired line has already been output to the image reading data 24f in step S110 (step S110, Yes), the reading data creation unit 22a adds 1 to the variable k as step S114, Step S107 and subsequent steps are re-executed.

  If the k-th row cannot be acquired in step S108 (No in step S108), or if the importance value of the acquired row is equal to or smaller than a predetermined value in step S109 (step S108) (S109, No), since the output of the contents of the object of the j-th image data has been completed, the read-out data creation unit 22a adds 1 to the variable j as step S115, and re-executes step S101 and the subsequent steps. .

  When the value of the variable j becomes larger than the number of image data attached to the e-mail, that is, when the output of the object contents of all the image data has been completed (Yes in step S101), it is read out. The data creation unit 22a ends the data output process.

  As described above, in this embodiment, the image reading data 24f is created based on the analysis result of the object included in the image data attached to the e-mail, and the image reading data 24f is read out by voice. Therefore, not only the text of the email but also the content of the image data attached to the email can be read out by voice.

  The configuration of the communication terminal 10 can be variously changed without departing from the gist of the present invention. For example, the object inside the contour may be analyzed in more detail and read out. Specifically, when the outline of a short cake is extracted, the outline inside the outline may be further analyzed to extract strawberries and sponges and read out.

  In addition, when an object that produces a sound like a musical instrument or an object that produces a cry like a cat is extracted, it does not just read out the type of the object, but also reproduces the sound or cry of the instrument as a sound effect. Also good. In addition, sound or music corresponding to the extracted object may be played along with the sound reading, such as playing a sound sung by a small bird when a forest is extracted as an object. Note that the sound of a musical instrument to be played, squealing, music, or the like may be downloaded via wireless communication.

  In the above-described embodiment, the extracted face area is collated with the face image of the address book data 24c in order to identify the individual. However, the face area information is transmitted to the server via wireless communication. The server may be requested to specify an individual. The requested server may be a server that provides an SNS (Social Network Service) function in which a member registers his / her face image as profile information. In this case, the verification target is registered as a friend of the requester. It may be limited to existing members.

  In the above-described embodiment, the image reading data 24f is created while the text of the e-mail is being read. However, when the communication terminal 10 receives the e-mail or when the communication terminal 10 is in the idle state. In some cases, the image reading data 24f may be created in advance.

  Further, when the text and the image data are alternately displayed as the text T1, the image data P1, the text T2, and the image data P2 as in the e-mail shown in FIG. 2, the text T1 is displayed. The image reading data 24f of the image data P1 is generated during the reading of the image data P1, and the image reading data 24f of the image data P2 is generated during the reading of the image reading data 24f of the image data P1 and the reading of the text T2. The image reading data 24f may be created for each image data.

  In the embodiment described above, the content of the image data attached to the e-mail is read out. However, the present invention can be applied to the case where the content of the image data received by means other than the e-mail is read out. .

  As described above, the communication terminal and the mail reading method according to the present invention are useful for reading an e-mail, and are particularly suitable for reading image data attached to an e-mail.

DESCRIPTION OF SYMBOLS 10 Communication terminal 15 Microphone 16 Receiver 22 Control part 22a Reading data preparation part 22b Voice reading part 24 Storage part 24a Mail reading program 24b Email 24c Address book data 24d Importance determination data 24e Analysis data 24f Image reading data 26 Communication part 26a Antenna 28 Operation unit 30 Audio processing unit 32 Display unit

Claims (16)

A communication unit for receiving an e-mail attached with image data;
A data creation unit that analyzes image data attached to an e-mail received by the communication unit and creates image reading data for reading out an object included in the image data by voice;
A speech reading unit that reads out the image reading data created by the data creation unit by voice ;
The data creating unit analyzes the image data attached to the e-mail and creates the image reading data while the voice reading unit reads the body of the e-mail received by the communication unit,
The speech reading unit, after reading of the text is complete, the communication terminal characterized by Rukoto reading the image reading data created by the data creation unit. A communication unit that receives image data and text data from an external device;
Analyzing the image data received by the communication unit, a data creation unit for creating image reading data for reading out an object included in the image data by voice;
A speech reading unit that reads the image reading data and text data created by the data creation unit by voice;
The data creation unit, the text data received by the communication unit while reading section reading the sound, create a data reading the image by analyzing the equivalent 該画 image data,
The speech reading unit, the text after the reading of data is completed, communication terminal you characterized in that read aloud the image reading data created by the data creation unit. A communication unit for receiving an e-mail attached with image data;
A data creation unit that analyzes image data attached to an e-mail received by the communication unit and creates image reading data for reading out an object included in the image data by voice;
A speech reading unit that reads out the image reading data created by the data creation unit by voice;
The data creation unit creates the image reading data including an explanation of a correspondence relationship between the plurality of image data when a plurality of image data is attached to the electronic mail received by the communication unit. Communication terminal. A communication unit that receives image data from an external device;
Analyzing the image data received by the communication unit, a data creation unit for creating image reading data for reading out an object included in the image data by voice;
A speech reading unit that reads out the image reading data created by the data creation unit by voice;
The data creation unit, when a plurality of image data is received by the communication unit, creates the image reading data including an explanation of a correspondence relationship between the plurality of image data. The data creation unit, on the basis of the importance of a plurality of objects extracted by analyzing the received said image data by the communication unit, the type of each object, the position, the fewer criteria for one of the ranges communication terminal according to claim 1, any one of 4 to determine, characterized in that to create the image reading data to be spoken first from high importance object Te. A communication unit for receiving an e-mail attached with image data;
A data creation unit that analyzes image data attached to an e-mail received by the communication unit and creates image reading data for reading out an object included in the image data by voice;
A speech reading unit that reads out the image reading data created by the data creation unit by voice;
The data creating unit analyzes the importance of the plurality of objects extracted by analyzing the image data attached to the e-mail received by the communication unit, at least one of the type, position, and range of each object. The communication terminal is characterized in that the image read-out data is created so as to be read out first from an object having high importance. The data creation unit creates the image reading data including an explanation of a correspondence relationship between the plurality of image data when a plurality of image data is attached to the electronic mail received by the communication unit. The communication terminal according to claim 1 or 6 . A communication unit that receives image data from an external device;
  Analyzing the image data received by the communication unit, a data creation unit for creating image reading data for reading out an object included in the image data by voice;
A speech reading unit that reads out the image reading data created by the data creation unit by voice;
The data creation unit analyzes the importance of a plurality of objects extracted by analyzing the image data received by the communication unit, based on a determination criterion regarding at least one of the type, position, and range of each object. The communication terminal is characterized in that the image reading data is created so that an object having high importance is read out first. The data creation unit is configured to determine whether an object type or name extracted by analyzing image data attached to an email received by the communication unit is included in the body or subject of the email. The communication terminal according to claim 1 , wherein the communication terminal is further used as a determination criterion for determining the importance level. The data creation unit analyzes the image data attached to the email received by the communication unit, and the email address of the person extracted as one of the objects matches the source address of the email The communication terminal according to any one of claims 1, 3, 6, 7 and 9, wherein the communication terminal is further used as a determination criterion for determining the importance of the object corresponding to the person. . When a plurality of image data is attached to the email received by the communication unit, the data creation unit extracts an object of the same type or name from other image data attached to the email. The communication terminal according to any one of claims 1, 3, 6, 7, 9 and 10 , further using as a determination criterion for determining the importance of an object. The data creation unit, a plurality of objects extracted by analyzing the pre-outs image data, and the type or name, and grouped based on the severity, objects belonging to the same group one sentence or a series The communication terminal according to any one of claims 1 to 11 , wherein the image reading data is created so as to be read as a sentence. The data creation unit analyzes the images data received by the communication unit, it extracts the feature data of the object included in the image data, queries the server using the extracted the feature data has been Accordingly, to identify the type or name of the object, a communication terminal according to claim 1, any one of 12, wherein the creating the image reading data based on the type or name specified. An e-mail reading method executed by a communication terminal having an e-mail receiving function,
A receiving step in which the communication terminal receives an e-mail attached with image data;
The communication terminal analyzes the image data attached to the e-mail received in the receiving step, and creates a data reading step for generating image reading data for reading out the content included in the image data by voice;
Wherein the communication terminal, after reading the text of the e-mail received in said receiving step with audio, seen including a reading voice step reads aloud by voice the image reading data generated in said data creation step,
In the data creation step, the communication terminal analyzes the image data attached to the email and creates the image readout data while reading the body of the email Reading method. An e-mail reading method executed by a communication terminal having an e-mail receiving function,
A receiving step in which the communication terminal receives an e-mail attached with image data;
The communication terminal analyzes the image data attached to the e-mail received in the receiving step, and creates a data reading step for generating image reading data for reading out the content included in the image data by voice;
The communication terminal includes a voice reading step of reading the image reading data created in the data creation step by voice,
In the data creation step, the communication terminal analyzes the importance of a plurality of objects extracted by analyzing the image data attached to the e-mail received in the reception step, and determines the type, position, and range of each object. Based on criteria for at least one of
In the voice reading step, the mail reading method is characterized in that the object having the high importance determined in the data creation step is read first. An e-mail reading method executed by a communication terminal having an e-mail receiving function,
A receiving step in which the communication terminal receives an e-mail attached with image data;
The communication terminal analyzes the image data attached to the e-mail received in the receiving step, and creates a data reading step for generating image reading data for reading out the content included in the image data by voice;
The communication terminal includes a voice reading step of reading the image reading data created in the data creation step by voice,
When the communication terminal includes a plurality of image data attached to the e-mail received in the reception step in the data creation step, the image reading data including an explanation of a correspondence relationship between the plurality of image data An e-mail reading method characterized by creating a mail.

Images