JP5120777B2 - Electronic data editing apparatus, electronic data editing method and program - Google Patents

Description

  The present invention relates to an electronic data editing apparatus, an electronic data editing method, and a program having an electronic data editing (creating) function such as a portable terminal with a mail creating function.

  For example, since a mobile terminal with a mail creation function such as a mobile phone can arbitrarily edit (create) mail data (text data) such as text data, an electronic data editing apparatus having an electronic data editing (creation) function can be used. It is an example. In the following, if the mobile terminal with this mail creation function is a “mobile phone”, the mail that can be created with this mobile phone is a text-based mail or a mixed pictogram mail. The pictogram mixed mail can use various pictograms (a wide variety of graphic symbols such as a heart symbol prepared in advance), and can express emotions that are difficult to convey only by letters.

  However, in the mixed pictogram mail, an appropriate pictogram must be selected from the list and the pictogram must be inserted into the mail, and this operation is performed manually.

  Therefore, a technique is known in which when a mail is created, a user's voice is picked up by a microphone, an emotion is judged, and a pictograph specified from the emotion is automatically inserted (for example, see Patent Document 1).

JP 2006-277567 A

However, in the above prior art, it is possible to automatically insert data such as pictograms that require time for manual input into editing data (electronic data) such as e-mail. Since a certain amount of audio data is required and it takes a certain amount of time from voice data collection to emotion judgment, emotions cannot be judged quickly, and the operability when editing electronic data is degraded. is there.
In addition to the case of inputting pictograms with a mobile phone, there is a problem in operability even when a name consisting of a character string arbitrarily input by a user is added to image data taken with a digital camera. In other words, a normal digital camera does not have a keyboard that is easy to input such as a personal computer or an advanced input support function, and it is necessary to input characters using cursor keys. It was troublesome to input.

  Accordingly, an object of the present invention is to provide an electronic data editing apparatus, an electronic data editing method, and a program capable of improving editing efficiency when data that takes time for manual input is inserted into electronic data being edited. is there.

An electronic data editing apparatus according to the present invention includes an imaging unit that captures an image of a subject and acquires image data, and detects a face in the image data, identifies the facial expression detected, and is classified into a plurality of groups. For each of a plurality of facial expression types that can be determined by the determination unit, a manual input unit that inputs arbitrary text data in response to a user operation The first storage means for storing the text data input in advance by the input means in association with the image data acquired by the imaging means when the text data needs to be input; The determination means determines the expression type for the data, and the text data stored in the first storage means corresponding to the determined expression type is automatically selected. It applies an automatic input means for inputting, acquisition of a new image data by the imaging unit, performs input of text data, including any words or short by the manual input means, the acquired new image data to the First registration means for additionally registering the facial expression type determined by the determination means and text data including the input arbitrary word or short sentence in association with the first storage means; the shall be the feature.
Also, an electronic data editing method according to the present invention includes an imaging unit that captures an image of a subject and acquires image data, and a manual input unit that inputs arbitrary text data in accordance with a user operation. The electronic data editing apparatus detects a face in image data, identifies the detected facial expression, and corresponds to any of the facial expression types classified into a plurality of groups. or a determination processing for a storage process of the for each determinable plurality of facial types by determination processing in the storage means in association with the text data previously input by the input means, the text When image data is acquired by the imaging means when data input is required, the facial expression type is determined by the determination process for the acquired image data. Was carried out, an automatic input process of inputting text data stored in the storage means in response to the determined facial expression types automatically, the acquisition of image data by the imaging means, of any by the manual input means Text data including a word or short sentence is input, and the facial expression type determined by the determination process for the acquired image data is associated with the text data including the input arbitrary word or short sentence. it characterized in that to execute a registration process for additionally registering in the storage means.
In addition, the program according to the present invention detects a face in image data and identifies the detected facial expression by using a computer of an electronic data editing apparatus including an imaging unit that captures an image of a subject and acquires image data. Determination means for determining which facial expression type is classified into a plurality of groups, manual input means for inputting arbitrary text data in response to a user operation, and a plurality of determinations that can be determined by the determination means Image data is acquired by the first storage means for storing the text data previously input by the input means in association with each expression type, and when the text data needs to be input, by the imaging means. And determining the expression type by the determination means for the acquired image data, and corresponding to the determined expression type, the first memory An automatic input means automatically inputs the text data stored in the acquisition of image data by the imaging unit, performs input of text data, including any words or short by the manual input means, is the acquisition First registration means for additionally registering the expression type determined by the determination means for the received image data and the text data including the input arbitrary word or short sentence in association with each other It is made to function as you characterized.

  According to the present invention, it is possible to provide an electronic data editing apparatus, an electronic data editing method, and a program capable of improving editing efficiency when data that takes time for manual input is inserted into electronic data being edited.

  Hereinafter, embodiments of the present invention will be described by taking a camera-equipped mobile phone as an example with reference to the drawings.

  First, terms in this embodiment are defined. The e-mail or e-mail data refers not only to text-based data such as text data, but also to data that includes additional information such as format information and optional information other than characters (image information, etc.) in addition to character information. . This is because, for example, text mail is generally used frequently in mobile phones, but some mobile phones and personal computers use non-text mail (electronic data such as HTML format) that can embed characters and images. ) Can be edited and created. In addition, “editing” or “editing process” in the present embodiment means that the electronic data that has already been created (e.g., the above-mentioned mail data) is modified or modified, and that new electronic data can be created. means. “Inserted data” refers to data that is inserted or synthesized into electronic data to be edited. For example, when the electronic data to be edited is “image data”, it can be inserted by image synthesis or the like. If the image data is image data or the electronic data to be edited is text data such as text data, it is text data (for example, text data) that can be additionally inserted at a specified position of the text data. However, these are merely examples for explanation, and any data can be used as long as it can be inserted into electronic data to be edited.

  FIG. 1 is a block diagram of a camera-equipped mobile phone. In this figure, a camera-equipped mobile phone (hereinafter simply referred to as “mobile phone”) 1 performs voice communication or data communication including mail via an antenna 2 with the nearest mobile phone base station (not shown). A communication unit 3 to perform, an imaging unit 4 including an imaging device such as a CCD or CMOS that captures a still image or a through image of a subject and generates image data thereof, a numeric keypad for both telephone number / character input, an on-hook key, and an off-hook Operation unit 5 including keys, shutter keys, cursor keys and various function keys, display of incoming / outgoing telephone numbers, electronic viewfinder during still image shooting or through image shooting, confirmation / playback display of captured images, creation of e-mail, Display unit 6 composed of a liquid crystal display or the like used for browsing, voice call microphone 7 and speaker 8, image data, and sent / received mail Data and other data such as a telephone directory, an input / output unit 11 that mediates exchange of image data and the like with an external device 10 such as a personal computer, and a primary battery or a secondary battery And a power supply unit 12 including a human face is extracted from given image data, and facial expressions (laughing face, crying face, angry face, expressionlessness,...) Are determined and It includes an emotion determination unit 13 that generates emotion data indicating the determination result, and a control unit 14 that is configured by a microcomputer including a CPU 14a, a PROM 14b, a RAM 14c, an I / O 14d, and the like, and controls the operations of the above-described units.

  FIG. 2 is a conceptual configuration diagram of the emotion determination unit 13. In this figure, as described above, the emotion determination unit 13 extracts “a person's face from the given image data, and expresses the facial expression (laughing face, crying face, angry face, no expression,... Etc.) and generate emotion data indicating the determination result ”. Here, “emotion” accurately refers to the human psychological state, but in practice, it is impossible or difficult to grasp the psychological state, and here, there is a certain correlation with the psychological state. “Facial expression” is defined as emotion. That is, it is assumed that “emotion = facial expression”.

  As a known technique for judging emotion (≈ psychological state) from facial expressions, for example, “Method of Estimating Psychological State from Facial Expression” described in Japanese Patent Application Laid-Open No. 2007-65969 is known. The method may also be used for the determination unit 13.

  By the way, the principle of the method described in this publication is that “the Fourier description that quantitatively represents the shape of the left and right eyes and mouth using a Fourier descriptor from a human image, and represents the shape of these mouth and left and right eyes, respectively. By calculating and normalizing the determination score for a plurality of preset basic emotions for each shape of the mouth and the left and right eyes by a linear discriminant using a child, the above basic emotions in each shape are calculated. Is calculated using a multiple regression equation that takes into account the importance of each facial expression element, and the psychological state of the person is estimated from the result. "

  The emotion determination unit 13 may be given image data G (this image data G may be a still image or a through image captured by the imaging unit 4 or may be a captured image stored in the storage unit 9. A face extraction unit 15 that extracts a human (human) face from a still image or a still image input from the external device 10; and a first determination unit 16 that determines whether a face is extracted. When it is determined that there is face extraction, the extracted facial expression is identified (for example, a smile, a crying face, an angry face, etc.) and the accuracy value for each facial expression (the “probability” of the expression) Of the facial expression identification unit 17 for setting the facial expression identification unit 17, the second determination unit 18 that determines the accuracy value for each facial expression in the facial expression identification unit 17, and the second determination unit 18 A selection unit that outputs a selection signal corresponding to the determination result 9, and an emotion data output unit 20 for outputting removed one emotion data from a plurality of emotion data according to the selection signal.

  Here, the facial expression identification unit 17 has a plurality of facial expression classifiers for each facial expression. Specifically, the illustrated example includes a smile classifier 17a for identifying a smile, a crying face classifier 17b for identifying a crying face, and an angry face classifier 17c for identifying an angry face. However, these are only examples. A classifier corresponding to another facial expression may be provided, or any of the classifiers illustrated may be omitted or replaced. Alternatively, it may be a single configuration that identifies only one facial expression (for example, a smile).

  The “smile”, “crying face”, and “angry face” in the above examples are examples of “expression types” for classifying different facial expressions of the same person. It is not a classification but a classification of facial expressions commonly used by a plurality of persons.

  The emotion data output unit 20 includes the number of facial expressions identified by the facial expression identification unit 17 plus one data output unit. That is, in the illustrated example, the facial expression identification unit 17 includes three classifiers (smiling classifier 17a, crying face classifier 17b, and angry face classifier 17c), so the emotion data output unit 20 stores smile data. A smile data output device 20a for outputting, a crying face data output device 20b for outputting crying face data, and an angry face data output device 20c for outputting angry face data. Is provided with an expressionless data output device 20d for outputting expressionless data when the above does not apply.

  The face data taken out from these four data output devices (smile data output device 20a, crying face data output device 20b, angry face data output device 20c and expressionless data output device 20d) are designed to imitate each expression. It may be a facial expression icon, or may be an emoticon or a pictograph. Alternatively, it may be identification information such as code data for specifying those facial expression icons, emoticons or pictograms.

  In such a configuration, when arbitrary image data G is given to the emotion determination unit 13, first, the face extraction unit 15 extracts a human face, and then the first determination unit 16 determines whether or not there is face extraction. Is done. If there is no face extraction, it is determined that the image data does not include a person's face, and data indicating that fact is output.

  On the other hand, when it is determined that there is face extraction, the facial expression identification unit 17 identifies the facial expression of the extracted face. For example, when the extracted face is a smile, a large value is given to the accuracy value of the smile classifier 17a, and a small value is given to the accuracy value of the other classifiers (crying face classifier 17b and angry face classifier 17c). Is given. The second determination unit 18 compares the respective accuracy values, the selection unit 19 outputs a selection signal for extracting emotion data (smile data) corresponding to the facial expression having the largest accuracy value, and the emotion data output unit 20 In accordance with this selection signal, one emotion data out of a plurality of emotion data, in this case, smile data (smile icon, smile pictogram or emoticon or identification information for specifying them) of the smile data output device 20a. Take out and output.

  As described above, the emotion determination unit 13 extracts the face of a person from the given image data G, and determines the facial expression (laughing face, crying face, angry face, no expression, etc.). Thus, emotion data indicating the determination result can be generated. In addition, when the accuracy value of each facial expression is equivalent, it is regarded as an expressionless and data of the expressionless data output unit 20d (expressionless icon, expressionless pictogram or emoticon or identification information for specifying them) Is output.

  FIG. 3 is a diagram showing a flow of mail creation and transmission processing. This flow is a part of a control program stored in advance in the PROM 14b of the control unit 14, and is read into the RAM 14c of the control unit 14 in response to a user event (an operation event for mail creation or transmission by the user). It is executed by the CPU 14a of the control unit 14.

  This flow can be broadly divided into a text editing block B1 executed immediately after the start of the flow and an emotion mark insertion block B2 executed subsequently. First, in the text editing block B1, it is determined whether or not text data for the mail subject has been input (step S1). If it has been input, the text data is inserted at the specified position (cursor position) of the mail subject. (Step S2). Next, it is determined whether or not text data for the mail body has been input (step S3), and if it has been input, the text data is inserted at the designated position (cursor position) of the mail body (step S4).

  Next, in the emotion mark insertion block B2, first, the presence / absence of image data for mail attachment is determined (step S5). Here, the image data for mail attachment is typically a face photograph (still image) taken by the image pickup unit 4 during mail creation, but other than this, for example, in the storage unit 9 It may be a stored face photograph taken in the past. In short, any suitable face photo for extracting emotions may be used. It may also be a face photo of another person.

  If there is no image data for mail attachment, the process leaves the emotion mark insertion block B2 to determine completion of mail creation (mail editing completion operation by the user) (step S13). If B1 is executed and completed, the mail is transmitted (step S14), and then the flow ends.

  On the other hand, when there is image data for mail attachment, the image data is sent to the emotion determination unit 13, and the emotion determination unit 13 detects a face (step S6). If no face is detected (step S7). NO) exits the emotion mark insertion block B2 and determines that the mail creation is complete (the mail editing completion operation by the user) (step S13). If not completed, the text editing block B1 is executed again and completed. Then, after sending the mail (step S14), the flow ends.

  On the other hand, if a face is detected (YES in step S7), emotions are determined by recognizing facial expressions (step S8), and then it is determined whether the emotion mark automatic synthesis mode is selected. (Step S9). The emotion mark automatic composition mode is a mode selected in advance by the user, and when this mode is selected, the automatic combination of the emotion mark to the image data attached to the mail is allowed. .

  When the emotion mark automatic synthesis mode is selected, an emotion mark corresponding to the determined emotion is generated and synthesized at the designated position in the attached image (step S10), and then the emotion character automatic insertion mode is selected. It is determined whether or not it is performed (step S11). The emotion character automatic insertion mode is a mode selected in advance by the user, and when this mode is selected, the automatic insertion of emotion characters in the mail subject or mail text is allowed. .

  If the emotion character automatic insertion mode is selected, a pictograph corresponding to the determined emotion is generated and combined with the specified subject position (cursor position) of the subject of the mail or the text (step S12). If the emotion mark automatic composition mode is not selected, step S10 is passed, and if the emotion character automatic insertion mode is not selected, step S12 is passed.

  FIG. 4 is a conceptual diagram showing an emotion mark automatic synthesis mode and an emotion character automatic insertion mode. As shown in (a), in the emotion mark automatic composition mode, the emotion mark 22 is placed at a predetermined position (for example, one of the corners indicated by a dotted line) of the target image data 21 (image data attached to an email). Synthesize. Also, as shown in (b), in the emotional character automatic insertion mode, emoticons (or emoticons) 24 and 25 are inserted at designated positions (cursor positions) in the subject or body of the mail 23.

  As described above, according to the present embodiment, the face character (or face pictogram) 24 corresponding to his / her emotion at that time can be obtained simply by taking a still image of his / her face using the image pickup unit 4 during mail creation. 25 can be automatically inserted in the subject and body of the mail 23.

  At this time, the still image shooting of the imaging unit 4 is completed almost simultaneously with the shutter operation, and the face extraction and emotion determination in the emotion determination unit 13 are also completed almost instantly. Compared to emotion judgment based on, a series of processes from emotion judgment to emoji or emoticon insertion can be completed in a very short time. It is possible to provide a mobile terminal with a mail creation function that can be reflected. In addition, since emotion determination is performed from image data attached to an email, it is not necessary to prepare image data for emotion determination again, thereby reducing labor.

  In addition, according to the present embodiment, as shown in FIG. 4A, since an emotion mark can be automatically synthesized at a predetermined position of image data attached to an email, the emotion of the sender is given to the recipient of the email. Can be visualized, and the certainty of emotion transmission can be improved.

  In the above description, emoticons (characters are combined to make a face) 24 and 25 are inserted in the subject and body of the mail 23, but the present invention is not limited to this. For example, graphic data such as a face icon or a pictograph may be inserted if permitted by the specification of the mail system.

  The technology of the present application is not limited to the above-described embodiment, and various modifications and development examples are included within the scope of the idea. For example, the following may be adopted.

  FIG. 5 is a diagram showing a first modified example, and more specifically, is a diagram showing a main part flow replacing step S5 to step S8 of the emotion mark insertion block B2 of FIG. In this first modification, first, it is determined whether image data for mail attachment (hereinafter referred to as image data for attachment) has been photographed (step S15), and when image data for attachment has been photographed, After the image data for attachment is stored in the storage unit 9 (step S16), it is determined whether new image data (hereinafter referred to as emotion determination image data) has been captured (step S17).

  If the image data for attachment has not been photographed, step S16 is passed to determine whether or not the image data for emotion determination has been photographed (step S17). If the emotion determination image data has not been shot, the process passes through the emotion mark insertion block B2 and proceeds to step S13 (email creation completion determination). If the emotion determination image data has been shot, the emotion determination image data is acquired. The image data is sent to the emotion determination unit 13, and the emotion determination unit 13 detects a face from the image data for emotion determination (step S18). If no face is detected (NO in step S19), an emotion mark is inserted. The process leaves block B2 and proceeds to step S13 (mail creation completion determination).

  When a face is detected (YES in step S19), the facial expression is recognized to determine an emotion (step S20), and then an emotion mark indicating the determined emotion is displayed on the screen (step S21). . Then, the user's selection instruction is determined (step S22). If there is no selection instruction, the repetition is determined (step S23), and if there is a repetition, step S17 (photographing determination of image data for emotion determination) is performed again. If there is no repetition, the emotion mark insertion block B2 is skipped and the process proceeds to step S13 (email creation completion determination). If there is a selection instruction, the emotion determined at the timing when the selection instruction is given. Adopt (step S24).

  Thereafter, similarly to the emotion mark insertion block B2 of FIG. 3, it is determined whether or not the emotion mark automatic synthesis mode is selected (step S9). If it is selected, an emotion mark corresponding to the determined emotion is generated and combined with the designated position in the attached image data (step S10), and then the emotion character automatic insertion mode is selected. It is determined whether or not (step S11). If it is selected, a pictograph corresponding to the determined emotion is generated and combined with the designated subject position (cursor position) of the subject of the mail or the text (step S12).

  In this first modification, image data attached to an email (image data for attachment) and image data for emotion determination (image data for emotion determination) are separated, so image data attached to the email ( Arbitrary data can be used for the attached image data, and the degree of freedom of the attached image data is increased.

  In addition, when the desired emotional determination is obtained while repetitively capturing the image data for emotion determination or while capturing a through image, it is only necessary to indicate the selection result of the emotion determination result. Emotion determination can be performed through trial and error, and emoticons and emotion marks that match the desired emotion expression can be selected and combined with image data for attachment or inserted into an email.

  FIG. 6 is a conceptual diagram for selecting a desired emotion determination result. In this figure, a confirmation display area 26 for emotion determination image data is provided on the screen of the display unit 6, and each of one or several emotion determination image data taken retrospectively. An icon indicating the emotion determination result is displayed. For example, in the illustrated example, an expressionless icon 27, a crying face icon 28, and a smiling face icon 29 are displayed in order from the top. The user only has to select the icon of the intended emotion. For example, in this figure, the smile icon 29 where the cursor 30 is located can be selected and designated as the intended icon.

  In this example, it is assumed that the image data for emotion determination is acquired by shooting several times, that is, the image data for emotion determination is a plurality of still image data obtained by shooting the same subject several times. For example, a through image may be used. By changing the facial expression during the acquisition of the through image, the expressionless icon 27, the crying face icon 28, the smile icon 29, etc. can be displayed, and any of them can be selected arbitrarily.

  In the above description, emotion marks (for example, emoticons and pictograms) determined from facial expressions are inserted. However, the present invention is not limited to this. For example, text data corresponding to facial expressions is registered in a dictionary in advance. Alternatively, text data may be read from the dictionary and inserted into the mail whenever necessary.

  FIG. 7 is a diagram showing a flow of dictionary registration processing. In this flow, first, text data (words, short sentences, etc.) to be registered is input (step S31), and then the face is continuously photographed (through image photographing) using the imaging unit 4, and the through image data is obtained. Obtain (step S32). Then, the emotion determination unit 13 is used to determine the emotion from the face in the through image data (step S33), and the determination result is displayed on the display unit 6 to determine the presence / absence of the user's registration instruction (step S34). ). If there is no registration instruction (NO in step S35), the process returns to step S32 again. If there is a registration instruction (YES in step S35), the emotion information (code) determined at the timing of the registration instruction and step S31 After associating with the text data input in step (b) and registering it in the dictionary data table in the PROM 14b or the storage unit 9 (step S36), the flow ends. It should be noted that the “registration instruction” by the user is not only performed by the key operation of the operation unit 5, but other than that, for example, when the facial expression is not changed for a predetermined time toward the imaging unit 4 “Registration instruction”, or “cancellation instruction” when the facial expression is changed during the process. This is preferable because a confirmation or cancellation operation by the user is unnecessary.

  FIG. 8 is a conceptual diagram of the dictionary data table. In this figure, the dictionary data table 31 has an emotion field 31a and a text data field 31b. The emotion field 31a stores information on emotions (laugh, cry, anger, expressionless, etc.), and the text data field 31b stores text data for each emotion. In the illustrated example, the emotion field 31a stores “laugh” indicating a laughing face, “crying” indicating a crying face, “angry” indicating an angry face, and “nothing” indicating no expression. This is for convenience of illustration. Actually, identification information (code) corresponding to each emotion is stored.

  FIG. 9 is a diagram showing a flow of using the dictionary data table, which replaces the mail subject input processing (steps S1 and S2) and the mail text input processing (steps S3 and S4) in the flow of FIG.

  In this flow, first, it is determined whether or not text data has been input (step S41). If it has been input, the text data is inserted at the current cursor position (step S42). Next, it is determined whether to acquire through image data (step S43). If not acquired, the process proceeds to step S5 in FIG. 3. If acquired, emotion information (code) is obtained from the face in the through image data. Determination is made (step S44).

  Then, with reference to the dictionary data table 31 of FIG. 8, the text data registered in association with the emotion information (code) determined in step S44 is read, and the text data is “temporarily” inserted at the current cursor position. (Step S45).

  Next, a confirmation operation or cancellation operation by the user is determined (step S46), and if cancelled, the process proceeds to step S5 in FIG. 3, and if confirmed, the temporarily inserted text data is confirmed and inserted (step S47). Then, the process proceeds to step S5 in FIG. It should be noted that the “confirm” or “cancel” operation by the user is not only performed by the key operation of the operation unit 5, but also, for example, the facial expression is changed toward the imaging unit 4 for a predetermined time. It may be “confirmed” if the user has not been selected, or “cancel” if the expression is changed in the middle.

  According to this flow, emotion information (code) and text data are associated with each other and registered in the dictionary data table 31. After that, it is determined from the image data only by photographing a face as necessary. The text data related to the emotion information (code) can be extracted from the dictionary data table 31 and automatically inserted at the designated position of the mail subject or the mail text. For this reason, text data corresponding to the emotion at that time can be quickly inserted into the mail, and the efficiency of mail creation can be improved.

  In the above explanation, emotions are specified from the face included in the image data. However, in reality, there are individual differences in facial expressions, so emotions must be specified for application to unspecified humans. Some kind of ingenuity is necessary in terms of reliability. The following are the ideas.

  FIG. 10 is a diagram showing a flow of determination data registration processing and a flow of emotion identification processing using the determination data. As shown in (a), in the judgment data registration process, first, a personal name to be registered is input (step S51), and information representing an emotion to be registered is input (step S52). Next, imaging is performed to acquire image data (step S53), and the captured image data or feature data extracted from the image data (feature point data in the image) is input to the personal name in step S51. Are registered in a data table for determination (described later) in association with the emotion (information representing) input in step S52 (step S54), and then the flow ends.

  Next, as shown in (b), in the emotion identification process, the face detected in the captured image is collated with registration information in a determination data table (described later) to identify registration data having the highest degree of matching. (Step S61). Next, after identifying the personal name and emotion corresponding to the identified registration data (step S62), the flow is terminated.

  FIG. 11 is a conceptual diagram of the determination data table. The determination data table 32 includes a user field 32a, an emotion field 32b, and an image data or feature data field 32c. The user field 32a stores the personal name input in step S51, the emotion field 32b stores information representing the emotion input in step S52, and the image data or feature data field 32c stores image data. Alternatively, feature data extracted from the image data is stored.

  According to the flow of the determination data registration process and the emotion identification process using the determination data, the determination data table 32 stores emotions (information representing each person) in advance and image data or feature data of the individual. When the emotion is actually determined, the emotion of each individual is specified by collating with the registered information in the determination data table 32, so that it is not affected by individual differences and is not specified by an unspecified human. It is possible to improve the reliability of emotion identification when applied to.

  In the above description, e-mail is created with a camera-equipped mobile phone as an example. However, the present invention is not limited to this. For example, a user arbitrarily inputs image data taken with a digital camera. You may apply when adding the name which consists of character strings. In addition, it is desirable that a camera function such as an imaging unit is present, but image data can be taken in from an external device such as a personal computer as necessary. In short, any electronic device having a function of acquiring image data from its own imaging unit or an external device and a function of editing electronic data may be used.

It is a block diagram of a mobile phone with a camera. 3 is a conceptual configuration diagram of an emotion determination unit 13. FIG. It is a figure which shows the flow of a mail preparation and transmission process. It is a conceptual diagram which shows the automatic synthetic | combination mode of an emotion mark, and the automatic insertion mode of an emotion character. It is a figure which shows a 1st modification. It is a selection instruction conceptual diagram of a desired emotion determination result. It is a figure which shows the flow of a dictionary registration process. It is a conceptual diagram of a dictionary data table. It is a figure which shows the flow of dictionary data table utilization. It is a figure which shows the flow of the judgment data registration process, and the flow of the emotion specific process using the data for judgment. It is a conceptual diagram of the data table for judgment.

Explanation of symbols

1 Mobile phone with camera (electronic data editing device)
3 Communication unit (transmission means)
4 Imaging unit (imaging means)
14 Control unit (identifying means, inserting means, reading means, image attaching means, specifying means)
15 Face extraction unit (extraction means)
17 Facial expression identification unit (determination means)
19 Selection part (selection means)
31 Dictionary data table (storage means)
32 Judgment data table (registration means)

Claims (8)

Imaging means for capturing an image of a subject and acquiring image data;
A determination unit that detects a face in the image data, identifies the expression of the detected face, and determines which of the expression types classified into a plurality of groups;
A manual input means for inputting arbitrary text data in accordance with a user operation;
First storage means for storing text data input in advance by the input means in association with each of a plurality of facial expression types that can be determined by the determination means;
When image data is acquired by the imaging unit when text data needs to be input, the determination unit determines the facial expression type for the acquired image data, and corresponds to the determined facial expression type. Automatic input means for automatically inputting text data stored in the first storage means,
The acquisition of new image data by the imaging means and the input of text data including an arbitrary word or short sentence by the manual input means, and the facial expression determined by the determination means for the acquired new image data First registration means for additionally registering the type and text data including the input arbitrary word or short sentence in association with the first storage means;
An electronic data editing apparatus comprising: The first storage means corresponds to an emotion mark including an emoticon or a pictograph for each of a plurality of expression types that can be determined by the determination means, and an emotion mark representing an emotion corresponding to each expression type Remember,
The first registration means is capable of additionally registering text data including an arbitrary word or short sentence in the first storage means regardless of the emotion corresponding to each facial expression type. The electronic data editing apparatus according to claim 1. A second storage means for storing facial image data or facial feature data in association with each of the plurality of facial expression types;
The determination unit compares the face image data or the facial feature data detected from the image data with the facial image data or the facial feature data stored in the second storage unit, and determines the degree of match. 3. The electronic data editing apparatus according to claim 2, wherein a facial expression type stored in correspondence with image data or facial feature data having a high face is used as a determination result. An expression type input means for inputting information representing the expression type;
Acquisition of image data by the imaging unit, input of information representing an expression type by the expression type input unit, and detection of facial image data or facial feature data detected in the acquired image data, and the input Second registration means for associating with the information indicating the performed facial expression type, and additionally registering the information in the second storage means,
The electronic data editing apparatus according to claim 3, further comprising: The second storage means stores the expression type and facial image data or facial feature data in association with each other in association with each of a plurality of individuals.
The second registration means inputs the information for identifying the individual in addition to the acquisition of the image data by the imaging means and the input of the information indicating the expression type by the expression type input means, and the information for identifying the individual And additionally registering the face image data or face feature data and the information indicating the expression type in association with the second storage means,
5. The electronic data editing apparatus according to claim 4, wherein the determination unit identifies a facial expression for each individual based on information stored in the second storage unit.   The automatic input means provisionally inputs the text data read from the first storage means in response to the acquisition of the image data by the imaging means, and a new operation by the imaging means is performed until a confirmation operation is performed. 6. The electronic data editing apparatus according to claim 5, wherein acquisition of image data and reading of text data corresponding to newly acquired image data are repeatedly executed. An electronic data editing apparatus control method comprising: an imaging unit that images a subject to acquire image data; and a manual input unit that inputs arbitrary text data in accordance with a user operation,
In the computer of the electronic data editing device,
A determination process for detecting a face in the image data, identifying an expression of the detected face, and determining which of the expression types classified into a plurality of groups;
A storage process for storing in the storage means text data input in advance by the input means for each of a plurality of facial expression types that can be determined by the determination process;
When image data is acquired by the imaging means when text data needs to be input, the facial expression type is determined by the determination process for the acquired image data, and the facial expression type corresponding to the determined facial expression type is determined. Automatic input processing for automatically inputting text data stored in the storage means ;
Acquisition of image data by the imaging unit, input of text data including an arbitrary word or short sentence by the manual input unit, facial expression type determined by the determination process for the acquired image data, and A registration process for additionally registering in the storage means in association with text data including any input word or short sentence;
An electronic data editing method characterized in that A computer of an electronic data editing apparatus comprising an imaging means for capturing an image of a subject and acquiring image data ,
A determination unit that detects a face in the image data, identifies the expression of the detected face, and determines which of the expression types classified into a plurality of groups;
A manual input means for inputting arbitrary text data in accordance with a user operation;
First storage means for storing text data input in advance by the input means in association with each of a plurality of facial expression types that can be determined by the determination means;
When image data is acquired by the imaging unit when text data needs to be input, the determination unit determines the facial expression type for the acquired image data, and corresponds to the determined facial expression type. Automatic input means for automatically inputting text data stored in the first storage means ,
The acquisition of image data by the imaging means, the input of text data including any word or short sentence by the manual input means, the facial expression type determined by the determination means for the acquired image data, and A program that functions as a first registration unit that associates an input arbitrary word or text data including a short sentence with each other and additionally registers the text data in the first storage unit .

Images