JP4600341B2 - Mobile terminal device with camera - Google Patents

Description

  The present invention is a mobile terminal device such as a mobile phone, PHS, PDA (Personal Data Assistance, Personal Digital Assistants), which can be photographed by a camera, and personal information such as a business card and a telephone directory. The present invention relates to a device having a function to manage

  In recent years, information such as telephone numbers, e-mail addresses, URLs, etc. is directly photographed by a camera mounted on a portable information terminal such as a cellular phone, extracted as data by performing image processing, etc. Has been proposed (see Patent Documents 1 to 3, etc.). In addition, there is a method in which a QR code / bar code is printed on a business card or a postcard, the QR code / bar code is read by a camera of the mobile phone device, and registered in a phone book in the mobile phone device.

JP 2002-366463 A JP 2004-080442 A JP 2005-055991 A

  In the above-mentioned prior art, for example, when a company's phone number, change of location only / personal name, or all information except phone number is updated, registration of some information is also registered in the mobile phone. When doing so, since information is registered in a lump, there is always a new phone book registration, and there is a problem that a plurality of information with the same personal name will be registered.

  In addition, when updating the phone book in the mobile phone, there is a method of shooting individual information such as the name, address, and phone number of the name card / postcard cover for each registration information with a camera and registering it in the phone book. It is necessary to activate the camera for each piece of information to be updated and shoot only a limited range of information. The burden caused by the increase in the number of operations by the user of the mobile phone, or the activation of the camera for each piece of information to be updated There is a problem that the usable time is shortened due to an increase in battery consumption of the telephone.

  In view of the above problems, an object of the present invention is to extract character information such as an address, name, phone number, mail address, URL, etc. from image information captured by a camera function of a mobile phone and register it in a phone book address. Is to provide means for preventing a plurality of the same information from being registered.

  The mobile terminal device with a camera according to the present invention has a camera function and a phone book function for recording personal information used when making an e-mail or a call with the mobile phone. A phone book from character information extracted by a mechanism for extracting address information, name, phone number, mail address, URL character information from the image information obtained by photographing the entire table / sender, and a mechanism for extracting character information from the photographed image And if it is already registered in the phone book, the registered contents and the mechanism for extracting character information from the photographed image are searched. The extracted information is compared, and the display of the difference is registered and only the difference information is registered in the telephone directory.

Specifically, the present invention is an individual comprising items such as a mobile phone function, a camera function, and a name, address, telephone number, mail address, URL, etc. In a portable terminal device with a camera having a phone book function for registering information as phone book data, image data recognition means for extracting character information related to the personal information from image information photographed by the camera function, and the image data The phone book data registered in the phone book is searched by the character information extracted by the recognition means to determine whether or not the personal information is registered, and when registered, the corresponding phone in the phone book Compares the book data with the extracted character information and displays the difference so that it can be identified. If not registered, the extracted character information is displayed as it is. That the duplicate data judging means, receiving the registration instruction by the user to the telephone directory of the character information displayed by the duplicate data judging means, if the personal information has been registered in the telephone directory data , Overwriting the duplicate data registered in the phone book with the character information displayed in the difference identification display, and if the personal information is not registered in the phone book data, Phone book registration means for newly registering everything in the phone book.

  Further, the duplicate data determination means compares the “name” item in the phone book data registered in the phone book with the “name” item in the extracted character information, and when both match, A first comparison / determination unit that outputs a match determination to the item, an item other than the “name” item in the phonebook data determined to be a match by the first comparison / determination unit, and the item in the extracted character information Second comparison determination means for comparing items other than the “name” item and determining that the extracted character information is personal information already registered as the telephone directory data when there is a matching item It is realizable as a structure provided with.

In addition, a telephone that records data items such as the name, address, telephone number, mail address, URL, etc. of the other party used when making an e-mail or telephone call with the mobile phone function, camera function, and mobile phone function of the present invention as personal information A program to be executed by a computer (CPU) incorporated in a camera-equipped mobile terminal device having a book function is a process of extracting character information related to the personal information from image information photographed by the camera function, and the extracted A process for determining whether or not the personal information is registered by searching the telephone book data registered in the telephone book based on character information; and when determining that the personal information is registered, the telephone book The corresponding phone book data and the extracted character information are compared and displayed so that the difference can be identified, and the personal information is not registered. And as a process of displaying the extracted character information when it is determined that, to the phone book when the difference between the corresponding telephone book data and the extracted character information in the phone book is displayed distinguishably If the personal information is registered in the phone book data when receiving a registration instruction by a user, the corresponding phone book data registered in the phone book by the character information displayed with the difference identification And when the personal information is not registered in the phone book data, the extraction is performed when a registration instruction to the phone book is received while the extracted character information is displayed as it is. characterized in that the character information including a process of newly registered in the telephone directory.

  The processing for determining whether or not the personal information is registered by searching the phone book data registered in the phone book based on the extracted character information is, for example, the phone book data registered in the phone book. A first comparison / determination process for comparing a “name” item in the extracted item and a “name” item in the extracted character information, and outputting a match determination when both match, and the first comparison determination When items other than the “name” item in the phone book data determined to match by processing are compared with items other than the “name” item in the extracted character information, and there is a matching item In addition, the extracted character information can be realized by a configuration including a second comparison determination process for determining that the telephone book data is personal information already registered.

  According to the present invention, if the personal information is not yet registered in the phone book, just by taking a business card or postcard with the camera of the mobile phone, if the personal information is newly registered and registered in the phone book, Since only the changed data can be added to the already-registered personal information and updated, it is not necessary to newly register in the phone book, so that the phone book can be easily managed and searched.

  In addition, it is possible to extract data from captured image data in a batch and update the phone book data based on the extracted data without individually capturing information corresponding to the data items of the phone book. Therefore, the number of activations of the camera is minimized, and the effect of suppressing the power consumption of the mobile terminal device is obtained.

  FIG. 1 is a principal block diagram of a circuit configuration showing an embodiment of a mobile phone as a mobile terminal device according to the present invention.

  The cellular phone 1 of this embodiment is equipped with a CPU (central processing unit) 2, and the CPU 2 is connected to each part in the apparatus via a bus 3. Among these, the ROM 4 stores a program for telephone directory registration of the present invention in addition to various control programs to be executed by the CPU 2, and also stores read-only data such as various display data. It is memory. The RAM 5 is composed of a random access memory that stores data temporarily required for the CPU 2 to execute the program. In relation to the present invention, image data / phone book data photographed by a camera are mainly stored.

  The input unit 6 is a circuit that detects key inputs from various keys including a decision (shutter) key 17 and up / down / left / right keys 8. Further, the wireless unit 9 that transmits and receives wireless radio waves, the control unit 10 that performs communication control such as voice processing and memory management, and the control unit 10 described above are controlled by the camera finder and the RAM 5 described above. It includes an LCD 11 for displaying stored data, a speaker / microphone 12, a camera module 13, and the like.

  FIG. 2 is a block diagram showing a software configuration of the present embodiment.

  The mobile phone software according to the present invention controls the phone book function 21 for managing personal information owned by the mobile phone user, the recognition engine 22 for recognizing the address of business cards and postcards from the photographed image, and the recognition engine 22. The recognition engine control unit 23, the recognition engine interface (I / F) 24 that controls data between the recognition engine 22 and the recognition engine control unit 23, the recognition engine 22, and the camera function of the mobile phone of the present invention. An image recognition control unit 25, a camera control unit 26 that controls the camera module 13 of the present invention, and a camera interface (I / F) 27 that controls data such as captured images between the camera control unit 26 and the image recognition control unit 25. Extraction of specific person information from the result recognized by the recognition engine 22 and personal information owned by the telephone directory function 21 , And a difference extraction section 28 for extracting a difference between data items.

  The image recognition control unit 25 is activated by the telephone directory function 21 or a camera function (not shown). When activated from the telephone directory function, activation information is passed from the camera I / F 27 to the camera control unit 26 and the camera module 13 is activated. This process is omitted when activated by a camera function (not shown).

  Next, the data of the captured image captured by the camera module 13 is passed to the image recognition control unit 25 through the camera control unit 26 and the camera I / F 27. The image recognition control unit 25 activates the recognition engine control unit 23 based on the image taken by the camera module 13, and the recognition engine control unit 23 activates the recognition engine 22 via the recognition engine I / F 24. Image data photographed by the camera module 13 is passed to the recognition engine 22 via the recognition engine I / F 24.

  The recognition engine 22 recognizes the address of the business card / postcard from the image photographed by the camera module 13, and the name, postal code, address, telephone number, FAX number, mobile phone described in the address of the business card / postcard. Recognition of numbers, company names, URLs, e-mail addresses, etc., and conversion into character strings. A result converted into a character string by the recognition engine 22 (hereinafter referred to as a recognition result) is passed to the recognition engine control unit 23 via the recognition engine I / F 24.

  The recognition engine control unit 23 notifies the image recognition control unit 25 of the recognition result of the recognition engine 22. The image recognition control unit 25 passes the recognition result notified from the recognition engine control unit 23 to the difference extraction processing unit 28, and the difference extraction processing unit 28 confirms whether the recognition result is already registered in the telephone directory 21 or not registered. In the case of data already registered in the telephone directory, the difference portion is extracted.

  3 to 8 are flowcharts showing the operation in the software configuration of the present embodiment, and FIG. 9 is an explanatory diagram showing an example of the difference information extracting operation in the present embodiment. Hereinafter, the operation of the present embodiment will be described with reference to FIGS.

  FIG. 3 shows the entire process from camera shooting to phone book update used in this embodiment. First, the user of the mobile phone 1 performs camera shooting such as a business card by the camera module 13. (Step 1), the recognition engine 22 extracts item information such as name, postal code, address, telephone number, FAX number, mobile phone number, company name, URL, and mail address from the photographed image data (Step 1). 2).

  Next, the image recognition control unit 25 activates the difference extraction processing unit 28 to read the phone book data registered in the phone book 21 in the mobile phone one by one, and at least the item information extracted in step 2 and the item information. A search is performed to determine whether there is telephone directory data (duplicate data) including partially overlapping items (step 3). Details of step 3 will be described separately with reference to FIGS.

  If there is duplicate data already registered in the phone book 21 as a result of the search process in step 3, the detailed data items in the duplicate data are compared to check the difference, and if there is a difference, the difference display process After that, if there is a user instruction, overwrite registration (step 4) is performed. If there is no difference, it is the same person who has been registered, so the next phone book data is read and the same processing is repeated. If there is no duplicate data in the phone book 21 in step 3, since it is new personal information, all data is displayed as it is, and if there is an instruction from the user, phone book registration is performed (step 4). Processing ends.

  Next, details of the telephone directory search and the difference display process (step 3) in FIG. 3 will be described.

  First, as shown in FIG. 4, the first “name” information read from the telephone directory 21 in step 11 and the “name” information acquired in step 2 of FIG. 3 are inquired. In step 12, it is checked whether all the phone book data has been searched. Therefore, if nothing is registered in the telephone book 21, the first "name" information is not read out, so the name duplication check after step 13 is omitted, and the new telephone book registration process from step 17 (step 19). Migrate as it is. If all of the phone book data has not been searched (“name” information that has not been checked for duplication exists in the phone book), it is equal to the first “name” information read from the phone book 21 (duplication). Is checked (step 13).

  If the “name” information of the phone book data read from the phone book 21 does not overlap with the “name” information extracted from the photographed image, the next “name” information is read from the phone book 21 (step 14). Step 12 and subsequent steps are repeated. If the “name” information read from the phone book 21 and the “name” information extracted from the photographed image are duplicated, the duplicate (same name and same name) is marked in the phone book data (step 15), and the phone of the same name After the number of book information is incremented by 1 (step 16), the next “name” information is read from the telephone book 21 (step 14), and the steps 12 and after are repeated.

  If all the phone book data has been searched (step 12, Yes), the process proceeds to step 17, and it is checked whether the number of duplicate data in the phone book is 0 with reference to the count value in step 16. If there is no duplicate data (the number of duplicate data is 0), the personal information data extracted from the image data acquired from the camera is unregistered, and the process proceeds to a process of registering as new phone book data (step 19). ). If duplicate data exists, duplicate data processing (explained in detail with reference to FIG. 5) such as differential data extraction and narrowing search in the telephone directory 21 is further performed in step 18.

  In the duplicate data processing (step 18) in FIG. 4, as shown in FIG. 5, the duplicate data narrowing process (for details, see if the duplicate data contains phone book data of the same person as the image recognition data). (Described in FIG. 6) to clarify the duplicate data (step 21). If the duplicate data includes telephone book data of the same person as the image recognition data, difference information extraction processing (details will be described with reference to FIG. 7) is performed at step 22.

  Thereafter, in step 23, the difference information portion extracted in step 22 is colored and displayed, and in step 24, it is determined whether there is an update instruction from the user. If it is determined in step 24 that there is an update instruction from the user, only the difference information is overwritten and registered in the registered telephone directory (step 25). If not, the process ends without updating. If the duplicate data narrowing process in step 21 does not include the same person's phone book data as the image recognition data in the duplicate data, all the image recognition data is displayed as it is (step 26). If it is determined that there is a new registration instruction from the user, registration is made in the telephone book 21 (step 28). Otherwise, the process is terminated without registration.

  Next, as shown in FIG. 6, in the duplication data narrowing process (step 21) in FIG. 5, in step 31, “detailed data items of personal information extracted from the photographed image” as “information 1” (FIG. 9B). Reference 32) is read out, and in step 32, as “information 2”, the first phone book duplication data among the phone book duplication data marked in step 15 is “registered in the phone book”. “Detailed data item of personal information” (see FIG. 9A) is read. In step 33, it is checked whether there is an equal information item in items other than the name information (detail items 1 and 2 in FIG. 9) in the “information 1” and “information 2” (for details see FIG. 8). To explain).

  If there is an information item equal to the information other than the name information in “information 1” and “information 2” in step 33, it is determined that “information 1” and “information 2” are data of the same person. Thus, the current telephone directory data is set as update data, and the process proceeds to step 22 and subsequent steps in FIG. If there is no information item other than name information in “information 1” and “information 2” in step 33, next, in step 34, all the phone book data having the same name with the mark is searched. It is checked by the duplicate data count value whether it is finished. If not completed, the detailed data item of the personal information in the telephone book, which will be the next duplicate data, is read as “information 2” (step 36), and step 33 and subsequent steps are repeated. If all the phone books having the same name have been searched in step 34, the information extracted from the photographed image is determined as another person with the same name, and the image recognition result data is set as newly registered data (step 35). ), The process proceeds to step 26 and subsequent steps in FIG.

  Next, in the difference information extraction process (step 22) in FIG. 5, as shown in FIG. 7, first, the second item of the detailed data item “personal information extracted from the photographed image” shown in FIG. 9B is read. (Step 41) In Step 42, the second item of the detailed data item of “personal information registered in the telephone book” shown in FIG. In step 43, it is checked whether the second item of “information 1” and “information 2” is the same information item. The “personal information registered in the phone book” to be read out in step 42 is duplicate data of “personal information extracted from the photographed image”, and the items 1 of the respective information are equal. Is excluded. Item 2 may also be excluded.

  If there is a difference in step 43 (if "information 1" and "information 2" are not equal), the content of information 1 is set in the difference information table shown in FIG. Information registered in the difference information table is set at the same data item position as the data item confirmed in step 43 on the difference table. If there is no difference in step 43, the third item of the detailed data item “personal information extracted from the photographed image” is read (step 45). In step 46, “information 2” is recorded as “person registered in the phone book”. The third item (step 46) of the detailed data item of information is read out.

  When the setting of data items in “information 1” and “information 2” is completed, it is checked in step 47 whether there are unconfirmed items. If there is an unconfirmed item, the process returns to step 43 to check the difference. If there are no unconfirmed items, the process ends. The items are confirmed from the second item to the last of the detailed data items of the phone book with respect to the items 1 to 11 of the detailed data items of the phone book and the items 1 to 11 of the confirmation result of the image recognition engine shown in FIG. Confirmation is made up to items (detailed item 11 corresponds in FIG. 9).

  Next, as shown in FIG. 8, the item check process (step 33) of “information 1” and “information 2” in FIG. 6 is first performed in step 51, as shown in FIG. The third item of the detailed data item of “personal information extracted from the photographed image” shown in FIG. 9 is read, and in step 52, the details of “personal information registered in the telephone book” shown in FIG. Read the third data item. In step 3, it is checked whether the third item of “information 1” and “information 2” is the same information item.

  If there is no difference at step 53 (if "information 1" and "information 2" are equal), at step 54, the value as the return value of the processing of FIG. finish. If there is a difference in step 53, the fourth item of the detailed data item of “personal information extracted from the photographed image” is read (step 55), and “details of personal information registered in the telephone book” is recorded as “information 2”. Read the fourth data item (step 56).

  When the setting of data items in information 1 and information 2 is completed, it is checked in step 57 whether there are unconfirmed items. If there is an unconfirmed item, the process returns to step 53 to check the difference. Confirm all items, and if there are no items that match (equal) in 3-11 items for the detailed data items in the phone book and the image recognition engine confirmation result, set no item equal to the return value (step) 58), the process is terminated.

  The items are confirmed in order from the third item to the items 1 to 11 of the detailed data items of the phone book and the items 1 to 11 of the confirmation result of the image recognition engine shown in FIG. Confirmation is made up to the last item (detail item 11 corresponds in FIG. 9). The “personal information registered in the phone book” to be read out in step 52 is duplicate data of “personal information extracted from the photographed image”, and item 1 (name) of each information is the same, and item 2 ( Are also considered equal, so they are excluded from the check targets in this item check process.

  In the above description, the case where camera photography is performed with a mobile phone has been described. However, the camera module is mounted on another mobile terminal device such as a PHS, and the personal information such as a business card is managed. If so, the present invention can be used.

It is a principal part block diagram of the mobile telephone as a portable terminal device which shows embodiment of this invention. It is a block diagram which shows the software structure of this embodiment. It is a flowchart which shows the operation | movement in this embodiment. It is a flowchart which shows the operation | movement in this embodiment. It is a flowchart which shows the operation | movement in this embodiment. It is a flowchart which shows the operation | movement in this embodiment. It is a flowchart which shows the operation | movement in this embodiment. It is a flowchart which shows the operation | movement in this embodiment. It is explanatory drawing which shows the example of the difference information extraction operation | movement in this embodiment.

Explanation of symbols

1 Mobile phone 2 CPU
3 bus 4 ROM
5 RAM
6 Input unit 7 Enter key 8 Up / down / left / right key 9 Wireless unit 10 Control unit 11 LCD
12 Speaker / Microphone 13 Camera Module 21 Phone Book 22 Recognition Engine 23 Recognition Engine Control Unit 24 Recognition Engine I / F
25 Image Recognition Control Unit 26 Camera Control Unit 27 Camera I / F
28 Difference extraction processing unit

Claims (6)

A mobile terminal device with a camera having a mobile phone function, a camera function, and a phone book function for registering personal information consisting of a plurality of items for specifying an email or a phone caller with the mobile phone function as phone book data in,
Image data recognition means for extracting character information related to the personal information from the image information photographed by the camera function ;
The phone book data registered in the phone book is searched based on the character information extracted by the image data recognition means to determine whether or not the personal information is registered. The corresponding phone book data and the extracted character information are displayed so that the difference can be identified, and if not registered, the duplicate data determination means for displaying the extracted character information as it is ,
When the personal information is registered in the phone book data in response to a registration instruction from the user to the phone book of the character information displayed by the duplicate data determination means , the difference identification display is displayed. When the duplicate data registered in the phone book is overwritten with character information and the personal information is not registered in the phone book data, all the extracted character information is newly stored in the phone book. Phone book registration means to register ;
A portable terminal device with a camera, comprising:   The camera-equipped mobile terminal device according to claim 1, wherein the plurality of items include a name, an address, a telephone number, a mail address, and a URL.   The duplicate data determination means compares the “name” item in the phone book data registered in the phone book with the “name” item in the extracted character information, and matches when both match. A first comparison / determination unit that outputs a determination; an item other than the “name” item in the phonebook data determined to match by the first comparison / determination unit; and the “name” in the extracted character information And a second comparison / determination unit that determines that the extracted character information is personal information already registered as the telephone directory data when items other than the item are compared and a matching item exists. The camera-equipped mobile terminal device according to claim 1, wherein the mobile terminal device has a camera. A mobile terminal device with a camera having a mobile phone function, a camera function, and a phone book function for registering personal information consisting of a plurality of items for specifying an email or a phone caller with the mobile phone function as phone book data The computer (CPU) built into the
A process of extracting character information related to the personal information from image information captured by the camera function ;
A process of determining whether or not the personal information is registered by searching the telephone book data registered in the telephone book by the extracted character information ;
When it is determined that the personal information is registered, the corresponding phone book data in the phone book is compared with the extracted character information and the difference is displayed in an identifiable manner ;
Processing to display the extracted character information as it is when it is determined that the personal information is not registered ;
When receiving a registration instruction from the user to the phone book when the difference between the corresponding phone book data in the phone book and the extracted character information is displayed in an identifiable manner, the personal information is When registered in the phone book data, a process of overwriting the corresponding phone book data registered in the phone book with the character information displayed in the difference identification ,
If the personal information is not registered in the phone book data, the extracted character information is displayed on the phone when receiving the instruction to register in the phone book while the extracted character information is displayed as it is. New registration to the book ,
A program for running   The process of searching the phone book data registered in the phone book by the extracted character information to determine whether the personal information is registered is performed in the phone book data registered in the phone book. By comparing the “name” item with the “name” item in the extracted character information and outputting a match determination when they match, the first comparison determination process An item other than the “name” item in the phone book data determined to match is compared with an item other than the “name” item in the extracted character information, and when there is a matching item, The program according to claim 4, wherein the extracted character information includes a second comparison and determination process for determining that the telephone book data is personal information already registered.   The program according to claim 5, wherein an item other than the “name” item compared in the second comparison determination process includes an address, a telephone number, an e-mail address, and a URL.

Images