JP2010267174A - Terminal device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a terminal device, which facilitates identifying a certain object from objects included in an image. <P>SOLUTION: A control unit 100 performs image recognition to identify characters included in an image; stores the location and display position of each of the identified characters in the image; combines each of the stored characters to search a dictionary database for a significant character string (word); extracts and temporarily stores corresponding character strings (words) as candidates; determines, when a desired character string is included in the candidates (words), whether the corresponding character string is displayed on a display screen of a display unit 105; and performs identification display, when it is displayed, and also notifies detection of the corresponding character string by sound, light or vibration. According to this, a certain object can be easily identified from objects included in the image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a terminal device and a program for easily identifying a certain object from among objects included in an image.

  There is known a technique for recognizing an object such as a character (character string) or a human face included in an image, searching for an object designated by the user, and identifying and displaying the object. As this type of technology, for example, in Patent Document 1, each character code and its position information are recognized from an image captured by a scanner, stored in a memory, and a character string that matches an input search word is stored. A technique is disclosed in which position information is retrieved from a memory and a corresponding character string is highlighted in an image.

JP-A-8-180068

  By the way, in the technique disclosed in Patent Document 1, the user must input an object to be searched (object to be identified and displayed) in advance, which is complicated. For example, when the object is character information, the user needs to input the character information to be searched. In addition, when the user cannot input an object to be searched, the corresponding object cannot be searched. That is, if the object is text information, even if the text information that the user wants to search is text information that is likely to be on the newspaper, it is forgotten or ambiguously stored. The user cannot input character information to be searched, and as a result cannot be identified and displayed. This is a situation that can occur frequently not only in newspaper pages but also in character information contained in a specific document.

  Furthermore, the technique disclosed in Patent Document 1 requires many operations such as image capture (input), image recording, display of recorded images, and input of an object to be searched, and these operations are complicated. Invite For example, if a user wants to know whether or not a certain character information exists in a newspaper or magazine page, and if it exists, if the user wants to know, Many operations are required to capture (input) a recorded image, record it, display the recorded image on a screen, and input character information to be searched. In other words, in other words, there is a problem in that a certain object cannot be easily identified from among objects included in an image.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide a terminal device and a program that can easily identify a certain object from among objects included in an image.

In order to achieve the above object, according to the first aspect of the present invention, an image acquisition means for acquiring an image, an image display means for displaying at least a part of the image acquired by the image acquisition means on a screen, and the image Specify one of specifying means for specifying an object included in the image acquired by the acquiring means, extracting means for extracting the object specified by the specifying means as candidate information, and candidate information extracted by the extracting means An identification display for identifying and displaying on the screen the display position of the object corresponding to the candidate information designated by the designation means when at least part of the image is displayed on the screen by the designation means Means.

  The invention according to claim 2 subordinate to claim 1 further includes an image pickup means for picking up an image, and the image acquisition means acquires an image picked up by the image pickup means.

  In the invention according to claim 3 subordinate to claim 1, there is provided simultaneous display means for simultaneously displaying on a screen at least part of the image acquired by the image acquisition means and candidate information specified by the specification means. It is further provided with the feature.

  In the invention according to claim 4 subordinate to claim 1, the object corresponding to the candidate information designated by the designation means is displayed when at least a part of the image is displayed on the screen by the image display means. Determination means for determining whether or not the object is displayed, and position information display means for displaying position information indicating the position of the corresponding object when the determination means determines that the corresponding object is not displayed. It is characterized by providing.

  In the invention according to claim 5 that is dependent on claim 1, the object corresponding to the candidate information designated by the designation means is displayed when at least a part of the image is displayed on the screen by the image display means. And a display for changing the display position of the image so that the corresponding object is displayed on the screen when it is determined by the determining means that the corresponding object is not displayed. And a position changing means.

  In the invention according to claim 6 that is dependent on claim 1, the object corresponding to the candidate information designated by the designation means is displayed when at least a part of the image is displayed on the screen by the image display means. It is further characterized by further comprising: a discriminating unit for discriminating whether or not the target object is displayed; and a notifying unit for informing the fact that the discriminating unit determines that the corresponding object is displayed.

  The invention according to claim 7 that depends on claim 1 is characterized in that the specifying means specifies a person as an object included in the image acquired by the image acquiring means.

  In the invention according to claim 8, which is dependent on claim 1, storage control means for storing an object corresponding to the candidate information designated by the designation means in association with information stored in the storage means is further provided. It is characterized by providing.

  The invention according to claim 9 that is dependent on claim 1 is characterized in that the specifying means specifies character information as an object included in the image acquired by the image acquiring means.

  The invention according to claim 10, which is dependent on claim 1, further comprises setting means for setting an object corresponding to the candidate information specified by the specifying means as character information input by the character input means. Features.

In the invention described in claim 11, an image display unit that displays at least a part of an image captured by the imaging unit on a screen, an image update unit that updates display of an image by the image display unit at a predetermined timing, Among the specifying means for specifying the first character information included in the image captured by the imaging means, the specifying means for specifying the second character information, and the first character information specified by the specifying means, A determining unit that determines whether or not the corresponding character information that matches the second character information specified by the specifying unit exists; and if the determining unit determines that the corresponding character information exists, the corresponding character Identification display means for identifying and displaying the display position of information on the display screen, and identification display update for updating the identification display by the identification display means when the image updating means updates the image display Characterized by comprising a stage.

  In the invention according to claim 12, which is dependent on the above-mentioned claim 11, simultaneous display means for simultaneously displaying on the screen at least a part of the image picked up by the image pickup means and the second character information designated by the designation means. Is further provided.

  In the invention according to claim 13, which is dependent on claim 11, the image display means is designated by the designation means in a state where at least a part of the image taken by the image pickup means is displayed on the screen. Characteristic information changing means for changing the second character information is further provided.

  In the invention according to claim 14, which is dependent on the above-described claim 11, the image display means displays at least a part of the image picked up by the image pickup means on the screen. Display determining means for determining whether or not the corresponding character information matching the character information of 2 is displayed on the screen, and when the display determining means determines that the corresponding character information is not displayed on the screen, It further comprises position information display means for displaying position information indicating the position of the corresponding character information.

  In the invention according to claim 15, which is dependent on the above-mentioned claim 14, when the image update means updates the display of the image, the position information update means for updating the display of the position information displayed by the position information display means is provided. It is further provided with the feature.

  In the invention according to claim 16 that is dependent on the above-mentioned claim 11, the image display means displays at least a part of the image picked up by the image pickup means on the screen. Display determining means for determining whether or not the corresponding character information matching the character information of 2 is displayed on the screen, and when the display determining means determines that the corresponding character information is not displayed on the screen, It further comprises display position changing means for changing the display position of the image so that the corresponding character information is displayed on the screen.

  In the invention according to claim 17 that is dependent on the above-mentioned claim 11, the image display means displays at least a part of the image picked up by the image pickup means on the screen. Display determination means for determining whether or not the corresponding character information matching the character information of 2 is displayed on the screen, and if the display determination means determines that the corresponding character information is displayed on the screen It is further provided with an informing means for informing.

In the invention according to claim 18, an image acquisition step of acquiring an image on a computer, an image display step of displaying at least a part of the image acquired in the image acquisition step on a screen, and an acquisition in the image acquisition step A specifying step for specifying an object included in the extracted image, an extracting step for extracting the object specified in the specifying step as candidate information, and a specifying step for specifying any of the candidate information extracted in the extracting step And an identification display step for identifying and displaying on the screen the display position of the object corresponding to the candidate information specified in the specification step when at least a part of the image is displayed on the screen in the image display step. It is made to perform.

  In the invention according to claim 19, an image display step for displaying at least a part of the captured image on a screen, an image update step for updating display of the image by the image display step at a predetermined timing, and the captured image It is specified by the specifying step in the specifying step for specifying the first character information included in the image, the specifying step for specifying the second character information, and the first character information specified in the specifying step. A determination step for determining whether or not there is corresponding character information matching the second character information, and if it is determined in the determination step that the corresponding character information exists, the display position of the corresponding character information is An identification display step for identifying and displaying on the display screen, and when the image update step updates the display of the image, the identification display by the identification display step is updated. Characterized in that to execute the identification display update step.

  In the present invention, an object can be easily identified from among objects included in an image.

1 is a system configuration diagram for explaining an overview of functions of a mobile phone 10 according to an embodiment. FIG. 2 is a block diagram showing an electrical configuration of the mobile phone 10. FIG. It is a figure which shows an example of the control information memorize | stored in the control information memory | storage part 101c of the memory | storage part 101. FIG. 6 is a diagram for explaining a correspondence relationship between a shooting range of the imaging unit 104 and a display range displayed on the screen of the display unit 105. FIG. 6 is a diagram showing an example of a screen field IMF and an input field INF provided in the display unit 105. FIG. It is a flowchart which shows operation | movement of a main routine. It is a flowchart which shows the operation | movement of a function starting process. It is a flowchart which shows operation | movement of imaging | photography function processing. It is a flowchart which shows the operation | movement of a real-time character search process. It is a flowchart which shows the operation | movement of a real-time character search process. It is a flowchart which shows the operation | movement of a recorded image character search process. It is a flowchart which shows the operation | movement of a recorded image character search process. It is a flowchart which shows the operation | movement of a real-time person search process. It is a flowchart which shows operation | movement of a recorded image person search process. It is a flowchart which shows the operation | movement of a character specific process. It is a flowchart which shows operation | movement of a person specific process. It is a flowchart which shows the operation | movement of a character identification display update process. It is a flowchart which shows the operation | movement of a person identification display update process. It is a flowchart which shows the operation | movement of a character identification display process. It is a flowchart which shows the operation | movement of a person identification display process. It is a figure which shows an example of operation | movement. It is a figure which shows an example of operation | movement. It is a figure which shows an example of operation | movement. It is a figure which shows an example of operation | movement.

Embodiments of the present invention will be described below with reference to the drawings.
A. Constitution
(1) System Configuration FIG. 1 is a system configuration diagram for explaining an overview of functions provided in the mobile phone 10 according to an embodiment. The mobile phone 10 has a mobile radio communication function and a mail transmission / reception function. When a call is made using the mobile radio communication function, the mobile phone 10 makes a call to the originating exchange SW via the registered base station BS. In response to the outgoing call, the originating side exchange SW switches the terminating side exchange SW based on the called party number (subscriber number) and location registration information inquired from a subscriber registration server (not shown) provided in the wireless communication network RN. Call. The called-side exchange SW calls the called-side mobile phone 10 (not shown) via the called-side base station BS, and when the called-side side responds to the incoming call, the calling-side exchange SW and the called-side exchange are exchanged. A link with the SW is established and communication becomes possible.

  Mail transmitted from the mobile phone 10 using the mail transmission / reception function is sent to a mail server (not shown) on the Internet ITN via a base station BS, an exchange SW, and a gateway server (not shown) provided in the wireless communication network RN. Is transmitted. The mail server is configured to send mail to the mobile phone 10 having the destination mail address by following the reverse route. In addition, the mobile phone 10 uses the mobile radio communication function to access the information distribution apparatus 20 on the Internet ITN via the base station BS, the exchange SW, and the radio communication network RN to obtain image data (still image / moving image). Download and get.

(2) Electrical configuration of mobile phone 10 FIG. 2 is a block diagram showing an electrical configuration of the mobile phone 10. In this figure, a control unit 100 includes a CPU, an input / output circuit, and the like, and controls the operation of each unit of the mobile phone 10 in accordance with various operation events supplied from an operation unit 106 described later. A characteristic processing operation of the control unit 100 according to the gist of the present invention will be described in detail later.

  The memory | storage part 101 is comprised from ROM and RAM, and has each memory | storage part 101a-101e mentioned later. The program storage unit 101a stores various program data executed by the control unit 100, control data referred to by a predetermined program, and screen data forming various display screens such as a standby screen. The various programs stored in the program storage unit 101a are a main routine, a function activation process, a shooting function process, a real-time character search process, a recorded image character search process, a real-time person search process, a recorded image person search, which will be described in detail later. Processing, character identification processing, person identification processing, character identification display update processing, person identification display update processing, character identification display processing, and person identification display processing.

  The various information temporary storage unit 101b includes a work area and an image data area. In the work area of the various information temporary storage unit 101b, various register / flag data necessary for the operation of the mobile phone 10, screen information for forming the display screen of the display unit 105, and the like are stored. On the other hand, in the image data area, an image (still image data / moving image data) photographed and recorded by the imaging unit 104, or an image downloaded from the image distribution device 20 on the Internet ITN (still image data / moving image data). Is saved.

As shown in FIG. 3, the control information storage unit 101c is designated by a setting flag that is set in accordance with a user operation out of two items (“change display position” or “display position”). Control information of items to be stored is stored. This item designates what control is to be performed when the corresponding object exists in the image but is not displayed on the screen. For example, when the setting flag “change display position” is set to “1”, the control information means that the display position of the corresponding object existing in the image is changed. Further, when the “display position” setting flag is set to “1”, the control information means that the position of the corresponding object existing in the image is displayed.

  In the dictionary database storage unit 101d, dictionary data for kana-kanji conversion, that is, dictionary data representing “conversion candidates (words)” associated with “reading” inputted as characters is stored and managed. In the address book storage unit 101e, information representing personal attributes associated with each registration number, for example, an address composed of "name", "telephone number", "mail address", "image", etc. Book data is stored.

The wireless communication transmitting / receiving unit 102 exchanges data with the base station BS through the antenna ANT1 under the control of the control unit 100 during data communication, and the control unit 100 receives and demodulates the voice data received and demodulated through the antenna ANT1 during voice communication. On the other hand, the transmission signal obtained by modulating the audio data supplied from the control unit 100 is amplified by high frequency and transmitted from the antenna ANT1. The audio signal processing unit 103 includes a speaker SP and a microphone MIC. The audio data supplied from the control unit 100 is D / A converted into an audio signal to generate sound from the speaker SP, or an audio signal output from the microphone MIC is output. The audio data is A / D converted and supplied to the control unit 100.

The imaging unit 104 outputs an image captured according to the imaging parameters (ISO sensitivity and shutter speed) specified by the control unit 100 according to the brightness of the subject. As shown in the example shown in FIG. 4, the shooting range of the imaging unit 104 is such that the display range is within the shooting range (the range that can be shot at once) when the newspaper is shot from a distance that allows text interpretation. The display range can be changed by a zoom operation, and the entire shooting range can be set as the display range.

The display unit 105 includes a color liquid crystal panel and a display driver. Under the control of the control unit 100, various operation screens such as a standby screen, images captured by the imaging unit 104, and the wireless communication unit 102 are used. The image acquired from the image distribution apparatus 20 on the Internet ITN is displayed on the screen. Further, as shown in FIG. 5, the display screen of the display unit 105 includes an image field IMF that is an area for displaying an image, and an input field INF that is mainly used as a character input area.

The operation unit 106 has various operation keys and buttons such as a power switch for power on / off, an off-hook / on-hook switch operated at the start / end of a call, a character input switch that also serves as a dial switch, and a shutter button. An event corresponding to the button operation is generated and output to the control unit 100. The RTC 107 generates a real-time clock used for timekeeping and timer interruption and supplies it to the control unit 100. The notification unit 108 performs a notification operation (sounding of the sound speaker 108a, light emission of the LED 108b, and driving of the vibration motor 108c) according to the notification signal supplied from the control unit 100. The battery unit 109 generates a driving voltage that drives each unit of the mobile phone 10.

B. Action
Next, the operation of the mobile phone 10 having the above configuration will be described with reference to FIGS. In the following, a main routine executed by the control unit 100, a function activation process, a photographing function process, a real-time character search process, a recorded image character search process, a real-time person search process, a recorded image person search process, a character specification process, a person specification process, Each operation of the character identification display update process, the person identification display update process, the character identification display process, and the person identification display process will be described.

(1) Operation of Main Routine FIG. 6 is a flowchart showing the operation of the main routine executed by the control unit 100. The control unit 100 stands by until a power-on operation is performed in step SA1, and when the user performs a power-on operation, the determination result is “YES”, and the process proceeds to step SA2. In step SA2, each part of the mobile phone 10 is initialized in response to power-on, and then a standby process for entering a standby state by performing location registration in the wireless communication network RN is executed. In the standby state, a standby screen (not shown) is displayed on the screen of the display unit 105.

  Subsequently, in steps SA3 to SA7, the presence / absence of each event of “setting operation”, “function activation operation”, “incoming call detection”, “power-off operation”, and “other operation” is determined. If these events do not occur, the determination results of steps SA3 to SA7 are all “NO”, and the apparatus waits in the standby state of step SA2. When any of the above events occurs in this standby state, processing corresponding to the generated event is executed. Hereinafter, the operation will be described separately for each event.

<When a setting operation is performed>
When a setting operation event occurs, the determination result in step SA3 is “YES”, the process proceeds to step SA8, and a setting process corresponding to the setting operation performed by the user is executed. Specifically, after performing processing such as designating control information (item) by setting flag operation in the control information storage unit 101c described above or recording a downloaded image, the process returns to the standby state in step SA2. .

<When the function is activated>
When a function activation operation event occurs, the determination result in step SA4 is “YES”, and the process proceeds to step SA9. After function activation processing (described later) is executed, the process returns to the standby state in step SA2.

<When an incoming call is detected>
When an incoming call detection event occurs, the determination result in step SA5 is “YES”, and the flow proceeds to step SA10. In step SA10, the incoming call notification is performed, and if the off-hook operation is performed during the incoming call notification, the line is connected to the calling side to start the call, and the call is disconnected when the on-hook operation is performed during the call. The process returns to the standby state in step SA2.

<When the power is turned off>
When a power-off operation event occurs, the determination result in step SA6 is “YES”, and the process proceeds to step SA11 to execute the power-off process, and then this process is completed.

<For other operations>
When other operations (for example, a transmission operation or a mail reception operation) are performed, the determination result in step SA7 is “YES”, and the process proceeds to step SA12. After executing processing corresponding to the operation, the process waits for step SA2. Return to the state.

(2) Operation of Function Activation Process Next, the operation of the function activation process will be described with reference to FIG. When this process is executed via step SA9 (see FIG. 6) of the main routine described above, the control unit 100 determines that the function instructed by the function activation operation is “imaging” in steps SB1 to SB4 illustrated in FIG. It is determined whether the function is “real-time character search function”, “recorded image character search function”, “real-time person search function”, or “recorded image person search function”. Hereinafter, the description of the operation will be made separately when each of these functions is activated.

<When the shooting function is activated>
In this case, the determination result in step SB1 is “YES”, and after returning to the standby state after performing the photographing process via step SB5. In the imaging process (see FIG. 8), as will be described later, an image captured and output from the imaging unit 104 is displayed on the display unit 105 and updated for display every predetermined time (every fixed period). When “recording operation (shutter button operation)” is performed, an image output from the imaging unit 104 is recorded in an image data area (not shown) of the various information temporary storage unit 101b, and when “end operation” is performed. When this process ends and “other operation” is performed, the other process is executed.

<When the real-time character search function is activated>
In this case, the determination result in step SB2 is “YES”, and after executing the real-time character search process via step SB6, the process returns to the standby state. In the real-time character search process (see FIGS. 9 to 10), as will be described later, image recognition is performed on an image taken every predetermined time to identify characters included in the image, and each identified character The position and display position in the image are stored, a significant character string (word) is searched from the dictionary database by combining the stored characters, and the corresponding character string (word) is extracted as a candidate and temporarily stored. Remember.

Then, the search character string “identified”, the search character string “positioned”, or the “input” search character string is searched from the temporarily stored candidates (words) to obtain candidates (words). If there is a search character string that matches, it is determined whether or not the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105. When the identification is already displayed, the display is updated), and the detection of the corresponding search character string is notified by sound, light, and vibration.

<When the recorded image text search function is activated>
In this case, the determination result in step SB3 is “YES”, and after executing the recorded image character search process via step SB7, the process returns to the standby state. In the recorded image character search process (see FIGS. 11 to 12), as will be described later, an image selected for character search from images recorded in an image data area (not shown) of the various information storage unit 101b is selected. The image recognition is performed to identify the character included in the image, the position of each identified character in the image and the display position are stored, and a significant character string (word) is combined by combining the stored characters. A search is made from the dictionary database, and the corresponding character string (word) is extracted as a candidate and temporarily stored.

Then, the search character string “identified”, the search character string “positioned”, or the “input” search character string is searched from the temporarily stored candidates (words) to obtain candidates (words). If there is a search character string that matches, it is determined whether or not the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105. When the identification is already displayed, the display is updated), and the detection of the corresponding search character string is notified by sound, light, and vibration.

<When the real-time person search function is activated>
In this case, the determination result in step SB4 is “YES”, the real-time person search process is executed via step SB8, and then the standby state is restored. In the real-time person search process (see FIG. 13), as will be described later, face recognition is performed on an image taken every predetermined time, and a person appearing in the image (a person image showing a face) is specified. The position in the image of the specified person and the display position on the display screen of the display unit 105 are temporarily stored in the various information temporary storage unit 101b, and the specified person (person image with a face) is selected as a candidate. Extracted as a temporary storage.

Then, from the temporarily stored candidates (person images), a “identified display” person, a “position display” person, or a “selected candidate (person image)” is searched, and the corresponding person is an image. If it is present, it is determined whether or not the corresponding person is displayed on the display screen (image field IMF) of the display unit 105. Display update), and the detection of the person is notified by sound, light, and vibration.

<When the recorded image person search function is activated>
In this case, the determination result in step SB4 is “NO”, and after executing the recorded image person search process via step SB9, the process returns to the standby state. In the recorded image person search processing (see FIG. 14), as will be described later, face recognition is performed on an image selected for person search from images recorded in an image data area (not shown) of the various information storage unit 101b. To identify a person (person image with a face) in the image, and store the position of the specified person in the image and the display position on the display screen of the display unit 105 in the various information temporary storage unit 101b. In addition to temporary storage, the specified person (person image with a face) is extracted as a candidate and temporarily stored in the various information temporary storage unit 101b.

When the temporarily stored person image is displayed as a thumbnail and a candidate (person image) to be searched for is designated from among the thumbnails, if a person matching the designated candidate (person image) does not exist in the image, the corresponding It is determined whether or not the person to be displayed is displayed on the display screen (image field IMF) of the display unit 105. If it is displayed, the person is identified and displayed, and the detection of the person is notified by sound, light, and vibration. To do.

(3) Operation of Shooting Function Process Next, the operation of the shooting function process will be described with reference to FIG. When this process is executed via step SB1 (see FIG. 7) of the function activation process described above, control unit 100 proceeds to step SC1 shown in FIG. In step SC1, an image output by the imaging unit 104 is captured in the work area of the various information temporary storage unit 101b based on the shooting parameters (ISO sensitivity, white balance, etc.) corresponding to the subject, and in the subsequent step SC2, the captured image is captured. Is displayed on the screen of the display unit 105. In step SC2, if an image has already been displayed on the screen, the newly captured image is displayed on the screen and updated.

  Next, in step SC3, it is determined whether or not a predetermined time (for example, 0.5 seconds) has elapsed since the start of imaging. If the predetermined time has elapsed, the determination result is “YES”, and the process returns to step SC1 to update the display of the newly captured and captured image. On the other hand, if the predetermined time has not elapsed, the determination result in step SC3 is “NO”, and the flow proceeds to step SC4.

  In steps SC4 to SC6, it is determined whether or not “recording operation”, “end operation” and “other operation” are performed during shooting. If the “recording operation (shutter button operation)” is performed, the determination result in step SC4 is “YES”, the process proceeds to step SC7, and the image captured in the work area of the various information temporary storage unit 101b. Is recorded and saved in the image data area of the various information storage unit 101b, and the process returns to step SC3.

When the “end operation” is performed, the determination result in step SC5 is “YES”, and this process is finished. When “other operation” is performed, for example, when an operation for instructing the change of the shooting parameter is performed, the determination result in step SC6 becomes “YES”, and the process proceeds to step SC8, and the setting of the shooting parameter is changed according to the operation. After performing “other processing” such as performing, the processing returns to step SC3.

  As described above, in the shooting function processing, the image output from the imaging unit 104 is displayed on the display unit 105 and displayed and updated at regular intervals (every predetermined time), and “recording operation” is performed in this process. In this case, the captured image is recorded in the image data area of the various information storage unit 101b, and when “end operation” is performed, this processing is ended. When “other operation” is performed, other information is stored. Processing is to be executed.

(4) Operation of Real-Time Character Search Process Next, the operation of the real-time character search process will be described with reference to FIG. When this process is executed through step SB6 (see FIG. 7) of the function activation process described above, the control unit 100 proceeds to step SD1 shown in FIG. 9, and takes shooting parameters (ISO sensitivity and white balance) corresponding to the subject. Etc.), the image output by the imaging unit 104 is captured in the work area of the various information temporary storage unit 101b. In the subsequent step SD2, the captured image is displayed on the display unit 105 on the screen. In step SD2, if an image has already been displayed on the screen, the newly captured image is displayed on the screen and updated.

  Next, in step SD3, a character specifying process is executed. In the character specifying process (see FIG. 15), as will be described later, image recognition is performed on the image captured in step SD1 to specify characters included in the image, and the position of each specified character in the image and The display position is stored, and a significant character string (word) is searched from the dictionary database by combining the stored characters, and the corresponding character string (word) is extracted as a candidate and temporarily stored.

  And a character identification display update process is performed via step SD4. In the character identification display update process (see FIG. 17), as will be described later, a search character string “identified and displayed” from the candidates (words) extracted from the image and temporarily stored in the character identification process in step SD3. When the search character string “position display” or the search character string “input” is searched and a search character string that matches a candidate (word) exists, the corresponding search character string is displayed on the display screen of the display unit 105. It is determined whether or not it is displayed in the (image field IMF). If it is displayed, it is displayed for identification (display update when it is already displayed for identification), and the detection of the corresponding search character string is sounded.・ Notify by light and vibration.

On the other hand, when the corresponding search character string is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag “change display position” of the control information stored in the control information storage unit 101c is “ 1 ”, the display position is changed so that the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105, and the corresponding search character string is identified and displayed / updated. The search character string to be detected is notified by sound, light, and vibration.

  Next, in step SD5, it is determined whether a predetermined time (for example, 0.5 seconds) has elapsed since the start of imaging. When the predetermined time has elapsed, the determination result is “YES”, the process returns to step SD1, and a newly captured image is captured. On the other hand, if the predetermined time has not elapsed, the determination result in step SD5 is “NO”, and the process proceeds to step SD6.

In steps SD6 to SD7 and steps SD8 to SD10 shown in FIG. 10, “character input / change operation”, “candidate display operation”, “input candidate setting operation”, “end operation”, and “other operations” are performed. Determine whether any operation has been performed. If neither of these operations is performed, each determination result is “NO”, and the process returns to step SD5.

On the other hand, if “character input / change operation” is performed, the determination result in step SD6 is “YES”, and the process proceeds to step SD11, where the search character string is entered in the input field INF in accordance with the character input / change operation. Enter or change the search string already entered in the input field INF. And after performing a character identification display process (refer FIG. 19) via step SD12, a process is returned to above-mentioned step SD5.

In the character identification display process, as will be described later, in step SD11, the search character string input / changed in the input field INF is extracted from the image by the character identification process (see FIG. 15) and temporarily stored ( Word) and if there is a corresponding search character string, it is determined whether or not the search character string is displayed on the display screen (image field IMF) of the display unit 105. In addition to identifying and displaying, the detection of the corresponding search character string is notified by sound, light, and vibration.

On the other hand, when the corresponding search character string is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag “change display position” of the control information stored in the control information storage unit 101c is “ 1 ”, the display position is changed so that the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105, and the corresponding search character string is identified and displayed / updated. The search character string to be detected is notified by sound, light, and vibration.

When the “candidate display operation” is performed, the determination result in step SD7 is “YES”, the process proceeds to step SD13, and the candidate (word) extracted from the image and temporarily stored in the character specifying process in step SD3 is stored. A list is displayed as an example shown in FIG. Then, the process proceeds to step SD14, and when a candidate designating operation for selecting a desired candidate from the list of candidates displayed is performed, the process proceeds to step SD15, and the candidate selected by the candidate designating operation is input as a search character string. Input to INF or change a search character string already input to a selected candidate search character string. Thereafter, the process proceeds to step SD12, and after executing a character identification display process (see FIG. 19) based on the updated candidate (searched character string), the process returns to step SD5 described above.

When the “input candidate setting operation” is performed, the determination result in step SD8 shown in FIG. 10 is “YES”, the process proceeds to step SD16, and in the character identification display update process (see FIG. 17) in step SD4 described above. The search character string identified and displayed is registered in the dictionary database as a candidate (word) associated with “reading” input by the user so as to be used for character input from the next time onward, and the process returns to step SD5 described above. .

When the “end operation” is performed, the determination result in step SD9 is “YES”, and this process is finished. If "other operation" is performed, the determination result in step SD10 is "YES", the process proceeds to step SD17, other processes are executed, and the process returns to step SD5 described above. The other process is, for example, a character specification executed through the above-described step SD3 if an operation for changing the display position of the image such as a zoom-up / zoom-down operation of the displayed image is performed. The display position of the character temporarily stored in the process (see FIG. 15) is updated, or the identification display and the position display are updated in the character identification display update process (see FIG. 17) executed through the above-described step SD4. Do.

  As described above, in the real-time character search process, image recognition is performed on an image captured every predetermined time to identify characters included in the image, and the position and display position of each identified character in the image Is stored, and the stored characters are combined to retrieve a significant character string (word) from the dictionary database, and the corresponding character string (word) is extracted as a candidate and temporarily stored. Search characters that match "candidate (word)" by searching for "identified display" search character string, "position display" search character string, or "input" search character string If a column exists, it is determined whether or not the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105. If it is displayed, an identification display (already identified and displayed) is displayed. And together with the display update) the time is, to inform the detection of the appropriate search string in the sound, light and vibration.

(5) Operation of Recorded Image Character Search Process Next, the operation of the recorded image character search process will be described with reference to FIG. When this process is executed via step SB7 (see FIG. 7) of the function activation process described above, control unit 100 proceeds to step SE1 shown in FIG. In step SE1, when an image selection operation is performed to select an image that is a character search target from images recorded in the image data area of the various information storage unit 101b, the process proceeds to step SE2, and the selected image is displayed. The data is read and displayed on the display unit 105.

  Subsequently, in step SE3, image recognition is performed on the image selected in step SE1 to specify characters included in the image, and the position and display position of each specified character in the image are stored. A character specifying process (see FIG. 15) is performed in which a significant character string (word) is searched from the dictionary database by combining the stored characters, and the corresponding character string (word) is extracted as a candidate and temporarily stored. .

Then, in steps SE4 to SE5 and steps SE6 to SE8 shown in FIG. 12, “character input / change operation”, “candidate display operation”, “input candidate setting operation”, “end operation” and “other operations” are selected. Determine whether any operation has been performed. If these operations are not performed, each determination result is “NO”, and the process returns to step SE4.

On the other hand, if “character input / change operation” is performed, the determination result in step SE4 is “YES”, and the process proceeds to step SE9, where the search character string is stored in the input field INF in accordance with the character input / change operation. Enter or change the search string already entered in the input field INF. And after performing a character identification display process (refer FIG. 19) via step SD12, a process is returned to above-mentioned step SD5.

In the character identification display process, the search character string input / changed in the input field INF in step SE9 is extracted from the images (see FIG. 15) extracted from the image and temporarily stored (words). If the search character string is searched and the corresponding search character string is present, it is determined whether or not the search character string is displayed on the display screen (image field IMF) of the display unit 105. The detection of the corresponding search character string is notified by sound, light and vibration.

On the other hand, when the corresponding search character string is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag “change display position” of the control information stored in the control information storage unit 101c is “ 1 ”, the display position is changed so that the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105, and the corresponding search character string is identified and displayed / updated. The search character string to be detected is notified by sound, light, and vibration.

When the “candidate display operation” is performed, the determination result in step SE5 is “YES”, the process proceeds to step SE11, and the candidate (word) extracted from the image and temporarily stored in the character identification process in step SE3 is stored. A list is displayed as an example shown in FIG. Then, the process proceeds to step SE12, and when a candidate designating operation for selecting a desired candidate from the list of candidates displayed is performed, the process proceeds to step SE13, and the candidate selected by the candidate designating operation is input as a search character string. Input to INF or change a search character string already input to a selected candidate search character string. Thereafter, the process proceeds to step SE10 described above, and after executing a character identification display process (see FIG. 19) based on the updated candidate (searched character string), the process returns to step SE4.

When the “input candidate setting operation” is performed, the determination result in step SE6 shown in FIG. 12 is “YES”, the process proceeds to step SE14, and the search identified and displayed in step SE10 (character identification display processing) described above. In order to use the character string for subsequent character input, the character string is registered in the dictionary database as a candidate (word) associated with “reading” input by the user, and then the process returns to step SE4.

When the “end operation” is performed, the determination result in step SE7 is “YES”, and this process is finished. If “other operation” is performed, the determination result in step SE8 is “YES”, the process proceeds to step SE15, other processes are executed, and the process returns to step SE4. The other process is, for example, a character temporarily stored in step SE3 (character specifying process) if an operation for changing the display position of the image such as a zoom-up / zoom-down operation of the displayed image is performed. The display position is updated, and the identification display and position display made in step SE10 (character identification display processing) are updated.

  As described above, in the recorded image character search process, the image selected for character search from the images recorded in the image data area of the various information storage unit 101b is subjected to image recognition, and the characters included in the image are included. Is stored, the position in the image and the display position of each specified character are stored, and a significant character string (word) is searched from the dictionary database by combining the stored characters, and the corresponding character string (word ) As a candidate and temporarily stored.

Then, the search character string “identified”, the search character string “positioned”, or the “input” search character string is searched from the temporarily stored candidates (words) to obtain candidates (words). If there is a search character string that matches, it is determined whether or not the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105. When the identification is already displayed, the display is updated), and the detection of the corresponding search character string is notified by sound, light, and vibration.

(6) Operation of Real-Time Person Search Process Next, the operation of the real-time person search process will be described with reference to FIG. When this process is executed via step SB8 (see FIG. 7) of the function activation process described above, the control unit 100 proceeds to step SF1 shown in FIG. 13 and takes shooting parameters (ISO sensitivity and white balance) corresponding to the subject. And the like, the image output by the imaging unit 104 is captured in the work area of the various information temporary storage unit 101b. In the subsequent step SF2, the captured image is displayed on the display unit 105 on the screen. In step SF2, if an image has already been displayed on the screen, the newly captured image is displayed on the screen and updated.

Next, in step SF3, a person specifying process is executed. In the person identification process (see FIG. 16), as will be described later, face recognition is performed on the captured image to identify a person (person image with a face) in the image, The temporary position and the display position on the display screen of the display unit 105 are temporarily stored in the various information temporary storage unit 101b, and all the specified persons (person images in which the face is reflected) are extracted as candidates for various information. Temporary storage in the temporary storage unit 101b.

  And a person identification display update process is performed via step SF4. In the person identification display update process (see FIG. 18), as will be described later, the person “identified and displayed” from the candidates (person images) extracted from the image and temporarily stored in the person specifying process in step SF3, When a person who has been “position-displayed” or “selected candidate (person image)” is searched and a person that matches the candidate (person image) exists in the image, that person is displayed on the display screen 105 ( It is determined whether or not it is displayed in the image field (IMF). If it is displayed, it is displayed for identification (or updated when it is already displayed for identification), and the detection of the person is detected by sound, light and vibration. To inform.

On the other hand, when the corresponding person is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag of “change display position” of the control information stored in the control information storage unit 101c is “1”. If the display position is changed so that the corresponding person is displayed on the display screen (image field IMF) of the display unit 105, and the corresponding person is identified and displayed / updated, the detection of the corresponding person is sounded. Notification by light and vibration.

  Next, in step SF5, it is determined whether a predetermined time (for example, 0.5 seconds) has elapsed since the start of imaging. When the predetermined time has elapsed, the determination result is “YES”, and the process returns to step SF1 to capture a newly captured image. On the other hand, if the predetermined time has not elapsed, the determination result in step SF5 is “NO”, and the process proceeds to step SF6. Then, in steps SF6 to SF9, it is determined whether any one of “candidate designation operation”, “address book registration operation”, “end operation”, and “other operation” has been performed. If these operations are not performed, the determination results are all “NO”, and the process returns to step SF5.

On the other hand, when the “candidate designation operation” is performed, the determination result in step SF6 is “YES”, the process proceeds to step SF10, and the person is extracted from the image by the person identification process (see FIG. 16) and temporarily stored. Candidates (person images) are displayed as thumbnails in the input field INF as in the example shown in FIG. The candidates (person images) displayed as thumbnails are displayed by moving the display positions of the candidates (person images) in the left-right direction in accordance with left / right key operations (not shown). When a candidate (person image) to be searched for is specified from the thumbnail display candidates (person images), the process proceeds to step SF11, and after executing the person identification display process, the process returns to step SF5 described above.

In the person identification display process (see FIG. 20), as will be described later, when a person that matches a designated candidate (person image) exists in the image, the person is displayed on the display screen (image field IMF) of the display unit 105. ) Is displayed, and if it is displayed, identification is displayed and the detection of the corresponding person is notified by sound / light / vibration.

On the other hand, when the corresponding person is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag of “change display position” of the control information stored in the control information storage unit 101c is “1”. , The display position is changed so that the corresponding person is displayed on the display screen (image field IMF) of the display unit 105, the corresponding person is identified and displayed, and then the detection of the corresponding person is performed by sound, light, Notify by vibration.

When the “address book registration operation” is performed, the determination result in step SF7 is “YES”, and the process proceeds to step SF12, where the image of the corresponding person is cut out within a predetermined range, and the address book data ( “Name”, “phone number”, “mail address”, etc.) are associated with “image” and registered in the address book storage unit 101e, and the process returns to step SF5 described above.

When the “end operation” is performed, the determination result in step SF8 is “YES”, and the present process ends. When “other operation” is performed, the determination result in step SF9 is “YES”, the process proceeds to step SF13, and other processes are executed, and the process returns to the above-described step SF5. The other process is, for example, when the operation for changing the display position of the image is performed, such as a zoom-up / zoom-down operation of the displayed image, and person identification executed through the above-described step SF3 The display position of the person (person image) temporarily stored in the process is updated, or the identification display and the position display updated in the person identification display update process executed through the above-described step SF4 are performed.

  As described above, in the real-time person search process, face recognition is performed on an image captured every predetermined time to identify a person (person image with a face) in the image, and the identified person Are temporarily stored in the various information temporary storage unit 101b, and all specified persons (person images with faces) are extracted as candidates. And store it temporarily.

Then, from the temporarily stored candidates (person images), a “identified display” person, a “position display” person, or a “selected candidate (person image)” is searched, and the corresponding person is an image. If it is present, it is determined whether or not the corresponding person is displayed on the display screen (image field IMF) of the display unit 105. Display update), and the detection of the person is notified by sound, light, and vibration.

(7) Operation of Recorded Image Person Search Process Next, the operation of the recorded image person search process will be described with reference to FIG. When this process is executed via step SB9 (see FIG. 7) of the function activation process described above, control unit 100 proceeds to step SG1 shown in FIG. In step SG1, when an image selection operation is performed to select an image for searching for a person from images recorded in the image data area of the various information storage unit 101b, the process proceeds to step SG2, and the selected image is read and displayed. The screen is displayed on the unit 105.

  Subsequently, in step SG3, face recognition is performed on the image selected in step SG1 to identify a person (a person image in which the face is reflected) in the image. The position and the display position of the display unit 105 on the display screen are temporarily stored in the various information temporary storage unit 101b, and all the specified persons (person images in which the face is reflected) are extracted as candidates and various types of information are temporarily stored. The person specifying process (see FIG. 16) temporarily stored in the unit 101b is executed.

In steps SG4 to SG7, it is determined whether any of the “candidate designation operation”, “address book registration operation”, “end operation”, and “other operation” has been performed. If these operations are not performed, the determination results are all “NO”, and the process returns to step SG4.

On the other hand, when the “candidate designation operation” is performed, the determination result in step SG4 is “YES”, the process proceeds to step SG8, and the candidate extracted from the image and temporarily stored in the person identification process in step SG3. (Portrait) is displayed as a thumbnail in the input field INF. The candidates (person images) displayed as thumbnails are displayed by moving the display positions of the candidates (person images) in the left-right direction in accordance with left / right key operations (not shown). When a candidate (person image) to be searched for is specified from the candidates (person images) displayed as thumbnails, the person identification display process is executed via step SG9, and then the process returns to step SF5 described above.

In the person identification display process (see FIG. 20), as will be described later, when a person that matches a designated candidate (person image) exists in the image, the person is displayed on the display screen (image field IMF) of the display unit 105. ) Is displayed, and if it is displayed, identification is displayed and the detection of the corresponding person is notified by sound / light / vibration.

On the other hand, when the corresponding person is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag of “change display position” of the control information stored in the control information storage unit 101c is “1”. , The display position is changed so that the corresponding person is displayed on the display screen (image field IMF) of the display unit 105, the corresponding person is identified and displayed, and then the detection of the corresponding person is performed by sound, light, Notify by vibration.

When the “address book registration operation” is performed, the determination result in step SG5 is “YES”, and the process proceeds to step SG10, where the image of the corresponding person is cut out within a predetermined range, and is added to the address book data selected by the user. After associating it with “image” and registering it in the address book storage unit 101e, the process returns to step SG4.

When the “end operation” is performed, the determination result in step SG6 is “YES”, and this process is finished. If “other operation” is performed, the determination result in step SG7 is “YES”, the process proceeds to step SG11, other processes are executed, and the process returns to step SG4. The other process is, for example, when the operation for changing the display position of the image is performed, such as a zoom-up / zoom-down operation of the displayed image, and person identification executed through the above-described step SF3 The display position of the person (person image) temporarily stored in the process is updated, or the identification display and the position display updated in the person identification display process in step SG9 are performed.

As described above, in the recorded image person search process, a person (a person appearing in the image by performing face recognition on the image selected for person search from the images recorded in the image data area of the various information storage unit 101b). A person image with a face) is specified, and the position of the specified person in the image and the display position of the display unit 105 on the display screen are temporarily stored in the various information temporary storage unit 101b and the specified person (A person image showing a face) is extracted as a candidate and temporarily stored in the various information temporary storage unit 101b.

When the temporarily stored person image is displayed as a thumbnail and a candidate (person image) to be searched for is designated from among the thumbnails, if a person matching the designated candidate (person image) does not exist in the image, the corresponding It is determined whether or not the person to be displayed is displayed on the display screen (image field IMF) of the display unit 105. If it is displayed, the person is identified and displayed, and the detection of the person is notified by sound, light, and vibration. To do.

(8) Operation of Character Specifying Process Next, the operation of the character specifying process will be described with reference to FIG. When this process is executed through either the above-described step SD3 of the real-time character search process (see FIG. 9) or the above-mentioned step SE3 of the recorded image character search process (see FIG. 11), the control unit 100 executes FIG. Then, the process proceeds to step SH1 shown in FIG. 2 to perform well-known image recognition on the photographed image and specify characters included in the image. Subsequently, in step SH2, the character specified in step SH1, the position of the character in the image, and the display position on the display screen of the display unit 105 are temporarily stored in the various information temporary storage unit 101b. If the character specified in the image is out of the display range, the display position is treated as a blank (null data).

  Next, in steps SH3 to SH4, a significant character string (word) is searched by combining the characters temporarily stored in step SH2 with reference to the dictionary database stored in the dictionary database storage unit 101d. A column (word) is extracted as a candidate. In step SH5, the extracted candidates are temporarily stored in the various information temporary storage unit 101b.

  As described above, in the character string specifying process, the captured image is subjected to image recognition to specify the character included in the image, and the position and display position of the specified character in the image are stored. A significant character string (word) is searched from the dictionary database by combining the stored characters, and the corresponding character string (word) is extracted as a candidate and temporarily stored.

(9) Operation of Person Identification Process Next, the operation of the person identification process will be described with reference to FIG. When this process is executed through either the above-described step SF3 of the real-time person search process (see FIG. 13) or the above-described step SG3 of the recorded image person search process (see FIG. 14), the control unit 100 executes FIG. The process proceeds to step SJ1 shown in FIG. 1 to perform known face recognition on the photographed image, and based on the recognition result, a person (person image with a face) shown in the image is specified.

Subsequently, in step SJ2, the person specified in step SJ1, the position of the specified person in the image, and the display position on the display screen of the display unit 105 are temporarily stored in the various information temporary storage unit 101b. . If the person specified in the image is out of the display range, the display position is treated as a blank (null data). In steps SJ3 to SJ4, all the persons specified in the image in step SJ1 (the person image showing the face) are extracted as candidates and temporarily stored in the various information temporary storage unit 101b.

As described above, in the person specifying process, face recognition is performed on the captured image to specify a person (person image showing a face) in the image, and the position of the specified person in the image and the display unit The display position on the display screen 105 is temporarily stored in the various information temporary storage unit 101b, and all the specified persons (person images with faces) are extracted as candidates and temporarily stored in the various information temporary storage unit 101b. Remember.

(10) Operation of Character Identification Display Update Process Next, the operation of the character identification display update process will be described with reference to FIG. When this process is executed through step SD4 (see FIG. 9) of the real-time character search process described above, control unit 100 proceeds to step SK1 shown in FIG. In steps SK1 to SK3, it is determined whether the state is “identified display”, “position display”, or “character input”.

  Note that the identification display state is present in the photographed image of the input search character when “Forex” is input as the search character string in the input field INF as in the example shown in FIG. And the search character string “foreign exchange” is surrounded by a frame and displayed when it is within the display range of the image field IMF. This display mode is not limited to this, and may be a mode in which the corresponding search character string is displayed in bold type, etc., or may be distinguished from other characters by reverse display or display color change.

  Also, the position displayed state is present in the photographed image when the input search character string “Forex” is input as the search character string in the input field INF, as in the example shown in FIG. However, when it is outside the display range of the image field IMF, it indicates a display mode in which the display position (direction / distance) of the search character string “foreign exchange” is indicated by an arrow mark. In the case of the mode indicated by the arrow mark, the distance from the display range may be expressed by the thickness or length of the arrow, or the distance may be expressed by a numerical value. In addition, when there are multiple character strings corresponding to the search character string in the captured image and they are outside the display range, an arrow indicating the display position (direction) of each corresponding character string is displayed. You may make it.

  Now, in the state where the search character string is not input in the input field INF, the determination result in step SK3 is “NO”, and this processing ends. However, when the search character string has already been input, the above step SK3 is completed. The determination result is “YES”, and the process proceeds to Step SK4. Further, if the search character string is “identified and displayed” or “position displayed” by a character identification display process (see FIG. 19) described later, the determination result of step SK1 or step SK2 is “YES”. The process proceeds to step SK4.

  In step SK4, a character search for searching for a search character string that matches a candidate (word) extracted from the image by the character specifying process (see FIG. 15) and temporarily stored is executed. In this character search, an arbitrary search method such as complete match, partial match or fuzzy search is used. Next, in step SK5, it is determined whether there is a search character string that matches the candidate (word). If the corresponding search character string does not exist, the determination result is “NO”, and the present process ends.

  On the other hand, if there is a search character string that matches the candidate (word), the determination result in step SK5 is “YES”, and the flow proceeds to step SK6. In step SK6, it is determined whether or not the corresponding search character string is displayed on the display screen (image field IMF) of display unit 105. For this determination, the display position of the character stored in the character specifying process (see FIG. 15) is referred to. If displayed, the determination result is “YES”, and the flow proceeds to Step SK7.

In step SK7, the corresponding search character string is identified and displayed as in the example shown in FIG. If the identification display is already performed, the identification display is updated. Thereafter, the process proceeds to step SK8, the sound speaker 108a, the LED 108b, and the vibration motor 108c of the notification unit 108 are driven to notify that the corresponding search character string has been detected by sound / light / vibration, and the process ends. .

  On the other hand, if the corresponding search character string is not displayed on the display screen (image field IMF) of the display unit 105, the determination result in Step SK6 is “NO”, and the flow proceeds to Step SK9. In step SK9, it is determined whether or not the control information stored in the control information storage unit 101c is “change display position”, that is, whether or not the setting flag of “change display position” is “1”. To do.

When the setting flag for “change display position” is “0”, that is, when the setting flag for “display position” is “1”, the determination result in step SK9 is “NO”, and the process returns to step SK10. Then, as in the example illustrated in FIG. 21, the display position (direction / distance) of the corresponding search character string is displayed with an arrow mark, and the present process ends. If the position is already displayed, the position display is updated and the process is terminated.

On the other hand, if the setting flag of “change display position” is “1”, the determination result in step SK9 is “YES”, and the process proceeds to step SK11, where the corresponding search character string is displayed on the display screen (image After the display position is changed so as to be displayed in the field IMF), the process proceeds to step SK7 described above, and the corresponding search character string is identified and displayed / updated as shown in the example shown in FIG. The search character string to be detected is notified by sound / light / vibration, and the process is terminated.

  In this way, in the character identification display update process, the search character string “position display” “identified and displayed” from the candidates (words) extracted from the image and temporarily stored by the character identification process (see FIG. 15). If a search character string that matches a candidate (word) exists, the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105. If it is displayed, it will be displayed with identification (if it is already displayed, the display will be updated) and the search string will be detected by sound, light, and vibration. Inform.

On the other hand, when the corresponding search character string is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag “change display position” of the control information stored in the control information storage unit 101c is “ 1 ”, the display position is changed so that the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105, and the corresponding search character string is identified and displayed / updated. The search character string to be detected is notified by sound, light, and vibration.

(11) Operation of Person Identification Display Update Process Next, the operation of the person identification display update process will be described with reference to FIG. When this process is executed via step SF4 (see FIG. 13) of the real-time person search process described above, the control unit 100 proceeds to step SL1 shown in FIG. In steps SL1 to SL3, it is determined whether the state is “identified display”, “position display”, or “candidate is selected”.

  Note that the identified and displayed state means that the candidate (person image) selected in the input field INF exists in the captured image and falls within the display range of the image field IMF, as in the example illustrated in FIG. When it exists, it refers to a mode in which the image area of the corresponding person is displayed surrounded by a frame line. This display mode is not limited to this, and any mode can be used as long as it can be distinguished from other people, for example, by periodically changing the brightness or brightness of the image area of the corresponding person.

  Now, in the state where “candidate is not selected” in the input field INF, the determination result in the above-mentioned step SL3 is “NO”, and this processing is finished. For example, as shown in FIG. If the candidate is displayed in gray and the currently selected candidate (person image) is present among the plurality of candidates (person images) displayed on the INF, the determination result in step SL3 is “YES”. Proceed to SL4. If the candidate (person image) is “identified and displayed” or “positioned” by a person identification display process (see FIG. 20) described later, the determination result of step SL1 or step SL2 is as follows. It becomes "YES" and progresses to step SL4.

  In step SL4, a person search for searching for a person matching the candidate (person image) extracted from the image by the person specifying process (see FIG. 16) and temporarily stored is executed. In this person search, an arbitrary search method such as complete match, partial match or fuzzy search is used. Next, in step SL5, it is determined whether there is a person that matches the candidate (person image). If the corresponding person does not exist, the determination result is “NO”, and the present process ends.

  On the other hand, if there is a person who matches the candidate (person image), the determination result in step SL5 is “YES”, and the process proceeds to step SL6. In step SL6, it is determined whether or not the corresponding person is displayed on the display screen (image field IMF) of display unit 105. For this determination, the display position stored in the person specifying process (see FIG. 16) is referred to. If displayed, the determination result is “YES”, and the flow proceeds to step SL7.

In step SL7, the corresponding person is identified and displayed as in the example shown in FIG. If the identification display is already performed, the identification display is updated. Thereafter, the process proceeds to step SL8, where the sound speaker 108a, the LED 108b, and the vibration motor 108c of the notification unit 108 are driven to notify that the corresponding person has been detected by sound / light / vibration, and the process ends.

  On the other hand, if the corresponding person is not displayed on the display screen (image field IMF) of the display unit 105, the determination result in step SL6 is “NO”, and the flow advances to step SL9. In step SL9, it is determined whether or not the control information stored in the control information storage unit 101c is “change display position”, that is, whether or not the setting flag of “change display position” is “1”. To do.

If the setting flag for “change display position” is “0”, that is, if the setting flag for “display position” is “1”, the determination result in step SL9 is “NO”, and step SL10 is entered. Then, the display position (direction / distance) of the corresponding person is displayed with an arrow mark, and the process is completed. If the position is already displayed, the position display is updated and the process is terminated.

On the other hand, if the setting flag of “change display position” is “1”, the determination result in step SL9 is “YES”, the process proceeds to step SL11, and the corresponding person moves to the display screen (image field IMF) of the display unit 105. ), The display position is changed so that the process proceeds to the above-described step SL7, where the corresponding person is identified and displayed / updated. In the subsequent step SL8, the fact that the corresponding person has been detected is detected by sound / light / vibration. Informs and finishes this processing.

  As described above, in the person identification display update process, the person who is “identified and displayed” from the candidates (person images) extracted from the image and temporarily stored by the person specifying process (see FIG. 16) is “position displayed”. If a person matching the candidate (person image) exists in the image, the corresponding person is displayed on the display screen (image field IMF) of the display unit 105. It is determined whether or not it is displayed. If it is displayed, it is identified and displayed (when it is already identified and displayed), and the detection of the person is notified by sound, light, and vibration.

On the other hand, when the corresponding person is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag of “change display position” of the control information stored in the control information storage unit 101c is “1”. If the display position is changed so that the corresponding person is displayed on the display screen (image field IMF) of the display unit 105, and the corresponding person is identified and displayed / updated, the detection of the corresponding person is sounded. Notification by light and vibration.

(12) Operation of Character Identification Display Process Next, the operation of the character identification display process will be described with reference to FIG. When this process is executed through either the above-described step SD12 of the real-time character search process (see FIG. 9) or the above-described step SE10 of the recorded image character search process (see FIG. 11), the control unit 100 executes FIG. The process proceeds to step SM1 shown in FIG. In step SM1, the search character string input / changed in the input field INF in step SD11 of the real-time character search process described above (or step SE9 of the recorded image character search process described above) is performed by the character specifying process (see FIG. 15). Search from candidates (words) extracted from the image and temporarily stored.

  Subsequently, in step SM2, it is determined whether there is a search character string that matches the candidate (word). If the corresponding search character string does not exist, the determination result is “NO”, and the present process ends. However, if the search character string that matches the candidate (word) exists, the determination result is “YES”, and the process returns to step SM3. move on. In Step SM3, it is determined whether or not the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105. For this determination, the display position of the character stored in the character specifying process (see FIG. 15) is referred to. If it is displayed, the determination result is “YES”, the process proceeds to step SM4, and the corresponding search character string is identified and displayed as in the example shown in FIG. In step SM5, the sound speaker 108a, the LED 108b, and the vibration motor 108c of the notification unit 108 are driven to notify that the corresponding search character string has been detected by sound / light / vibration, and the process ends.

On the other hand, when the corresponding search character string is not displayed on the display screen (image field IMF) of the display unit 105, that is, when it exists outside the display range, the determination result of step SM3 becomes “NO”, Proceed to step SM6. In step SM6, it is determined whether or not the control information stored in the control information storage unit 101c is “change display position”, that is, whether or not the setting flag of “change display position” is “1”. To do.

If the “change display position” setting flag is “0”, that is, if the “display position” setting flag is “1”, the determination result in step SM6 is “NO”, and the process proceeds to step SM7. As in the example shown in FIG. 21, the display position (direction / distance) of the corresponding search character string is displayed with an arrow mark, and this processing is completed.

On the other hand, if the setting flag of “change display position” is “1”, the determination result in step SM6 is “YES”, and the process proceeds to step SM8, where the corresponding search character string is displayed on the display unit 105. After the display position is changed to be displayed on the screen (image field IMF), the process proceeds to step SM4 described above, and the corresponding search character string is identified and displayed as in the example shown in FIG. The fact that the corresponding search character string has been detected is notified by sound, light, and vibration, and this processing is completed.

  As described above, in the character identification display process, the search character string input / changed in the input field INF is searched from the candidates (words) extracted from the image by the character specifying process (see FIG. 15) and temporarily stored. If the corresponding search character string exists, it is determined whether or not the search character string is displayed on the display screen (image field IMF) of the display unit 105. The search character string to be detected is notified by sound, light, and vibration.

On the other hand, when the corresponding search character string is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag “change display position” of the control information stored in the control information storage unit 101c is “ 1 ”, the display position is changed so that the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105, and the corresponding search character string is identified and displayed / updated. The search character string to be detected is notified by sound, light, and vibration.

(13) Operation of Person Identification Display Process Next, the operation of the person identification display process will be described with reference to FIG. When this process is executed through either the above-described step SF11 of the real-time person search process (see FIG. 13) or the above-described step SG9 of the recorded image person search process (see FIG. 14), the control unit 100 executes FIG. The process proceeds to step SN1 shown in FIG.

In step SN1, the candidate (person image) specified in step SF10 of the real-time person search process described above (or step SG8 of the recorded image person search process described above) is extracted from the image by the person specifying process (see FIG. 16). A person search for searching for a person (person image) temporarily stored is executed. In this person search, an arbitrary search method such as complete match, partial match or fuzzy search is used. Next, in step SN2, it is determined whether there is a person that matches the candidate (person image). If the corresponding person does not exist, the determination result is “NO”, and the present process ends.

  On the other hand, if there is a person who matches the candidate (person image), the determination result in step SN2 is “YES”, and the process proceeds to step SN3. In Step SN3, it is determined whether or not the corresponding person is displayed on the display screen (image field IMF) of the display unit 105. For this determination, the display position stored in the person specifying process (see FIG. 16) is referred to. If displayed, the determination result is “YES”, and the flow proceeds to step SN4. In step SN4, the corresponding person is identified and displayed as in the example shown in FIG. Thereafter, the process proceeds to step SN5, the sound speaker 108a, the LED 108b, and the vibration motor 108c of the notification unit 108 are driven to notify that the corresponding person has been detected by sound / light / vibration, and the process ends.

  On the other hand, if the corresponding person is not displayed on the display screen (image field IMF) of the display unit 105, the determination result in step SN3 is “NO”, and the process proceeds to step SN6. In step SN6, it is determined whether or not the control information stored in the control information storage unit 101c is “change display position”, that is, whether or not the setting flag of “change display position” is “1”. To do.

If the setting flag for “change display position” is “0”, that is, if the setting flag for “display position” is “1”, the determination result in step SN6 is “NO”, and step SN7 is entered. Then, the display position (direction / distance) of the corresponding person is displayed with an arrow mark, and the process is completed. If the position is already displayed, the position display is updated and the process is terminated.

On the other hand, if the setting flag of “change display position” is “1”, the determination result in step SN6 is “YES”, and the process proceeds to step SN8, where the corresponding person displays the display screen (image field IMF) of the display unit 105. ), The display position is changed so that the process proceeds to step SN4 described above, the corresponding person is identified and displayed, and in the subsequent step SN5, the sound, light, and vibration are notified that the corresponding person has been detected. To finish this process.

  As described above, in the person identification display process, if a person matching the designated candidate (person image) exists in the image, is the corresponding person displayed on the display screen (image field IMF) of the display unit 105? In the case of being displayed, identification is displayed, and the detection of the corresponding person is notified by sound, light and vibration.

On the other hand, when the corresponding person is not displayed on the display screen (image field IMF) of the display unit 105, the setting flag of “change display position” of the control information stored in the control information storage unit 101c is “1”. If the display position is changed so that the corresponding person is displayed on the display screen (image field IMF) of the display unit 105, the corresponding person is identified and displayed, and then the detection of the corresponding person is detected by sound / light.・ Notify by vibration.

As described above, in the present embodiment, image recognition is performed on a captured (or recorded) image to specify characters included in the image, and the position and display position of each specified character in the image are determined. In addition to storing, the stored characters are combined to retrieve a significant character string (word) from the dictionary database, the corresponding character string (word) is extracted as a candidate, temporarily stored, and temporarily stored. Search character string that matches the candidate (word) by searching the search character string “identified”, the search character string “positioned”, or the “input” search character string Is present, it is determined whether or not the corresponding search character string is displayed on the display screen (image field IMF) of the display unit 105. Time is With shows updated) for so informing the detection of the corresponding search string sound, light and vibration, among characters included in the image can easily identify the desired character string. Therefore, an object can be easily identified from among the objects included in the image.

Further, face recognition is performed on the captured (or recorded) image to identify a person (person image with a face) in the image, and the position of the identified person in the image and the display unit 105. Is temporarily stored in the various information temporary storage unit 101b, and a specified person (a person image showing a face) is extracted and temporarily stored as a candidate. Search for “identified display” person, “position display” person or “selected candidate (person image)” from candidates (person image), and if the corresponding person exists in the image, It is determined whether or not a person is displayed on the display screen (image field IMF) of the display unit 105. If the person is displayed, identification display (display update when already displayed) is performed, and Detection of people by sound, light, vibration In so informing, from the person (person image that is reflected face) included in the image can easily identify a person. Therefore, an object can be easily identified from among the objects included in the image.

Further, other more specific effects obtained in the above-described embodiment are as described in the following items (a) to (p).
(A) A user can easily identify a certain object from among objects included in an image without the user newly inputting an object to be identified. For example, when the object is text information, the text information that the user wants to identify is text information that is forgotten or vaguely memorized even though it is known that the text information is likely to be on the newspaper. Even so, it is not necessary for the user to input the character information, and the character information can be identified and displayed.
(B) A user can easily identify a certain object from among objects included in a captured image without the user newly inputting an object to be identified.
(C) The user can confirm the image while confirming the object to be identified.
(D) When an object to be identified is included in an image but is not displayed, the user can easily confirm at which position (direction and distance) the current display is located.
(E) Even if an object to be identified is included in the image but not displayed, the user can easily identify the object by changing the display position of the image.
(F) The user can easily confirm that the object to be identified is displayed.
(G) A user can be easily identified from among persons included in the image without the user newly inputting a person to be identified. For example, in the case of an image that contains a large number of people, such as a group photo, the user exists in the group photo or where it exists if there is a specific person It is cumbersome to search for it with one's own eyes, especially when there are many people, but in the above embodiment, the person included in the image is extracted as candidate information. Identification can be easily performed by simply designating a person to be identified from the extracted candidate information.
(H) The identified person can be stored in association with any stored information such as an address book, schedule, or mail.
(I) The user can easily identify certain character information from the character information included in the image without the user newly inputting character information to be identified.
(J) The identified character information can be used for character input.
(K) The user can easily identify the designated character information without performing an operation of recording an image or an operation of displaying the recorded image. For example, if a user wants to know whether or not certain character information exists in a newspaper page, and if it exists, it should include all the target pages. After shooting as a simple image, the specific character information can be identified and displayed without the operation of recording the image and the operation of displaying the recorded image. Even if the range of the image to be taken is moved, it is displayed correctly.
(L) The user can confirm the image while confirming the character information to be identified.
(M) The user can easily change the character information to be identified.
(N) When the character information to be identified is included in the image but is not displayed, the user can easily confirm the position (direction or distance) from the current display.
(O) The position from the current display can be correctly confirmed by moving the range of the captured image.
(P) Character information to be identified is included in the image, but even if it is not displayed, the user can easily identify the character information by changing the display position of the image.
(Q) The user can easily confirm that the object to be identified is displayed. For example, the user is moving the range to be imaged in an attempt to know whether or not certain character information exists in the newspaper, and if so, where it exists. In this case, it is difficult to always check the display unit according to the movement. However, in the above-described embodiment, the user can confirm that character information desired to be identified by sound, vibration, light, or the like is displayed, so there is no need to constantly check the display unit.

  In the above-described embodiment, in the real-time character search process (or real-time person search process), a character specifying process (or person specifying process) is executed each time a new image is taken (every predetermined time elapses). Instead of specifying (or specifying a person), instead of specifying a character (or specifying a person) every time a predetermined time elapses, a change in the difference between the previously taken image and the currently taken image If the number of characters is small, the processing may be simplified by updating only the display position of the specified character (or person).

The above embodiment assumes the mobile phone 10, but the gist of the present invention is not limited to this, and a function of capturing and displaying an image such as a personal computer, a PDA, a digital camera, or a music player. Needless to say, the present invention can be applied to all devices.

In the present embodiment, the folding open / close type mobile phone 10 is taken as an example. However, the present invention is not limited to this, and an arbitrary housing structure such as a straight type, a slide type, or a spin top type may be used. The storage unit 101 is not limited to a memory provided in the mobile phone 10 and may be an arbitrary external storage device such as a memory card or an external hard disk. Furthermore, the display unit 105 is not limited to the one provided in the mobile phone 10 and may be in a form using an arbitrary external display device such as an external monitor.

The image to be stored may be any type of image such as a still image or a moving image, and is not limited to an image captured by the image capturing unit, an image received by the broadcast receiving unit, a downloaded image, or an image received by data communication, and any method. It becomes possible to apply to the image acquired by. In this embodiment, a character or a person is used as an example of an object specified from an image.

Furthermore, in the present embodiment, an example is given in which a person is specified by face recognition. However, instead of this, a person may be specified from a shape included in an arbitrary range such as the upper body or the whole body. In addition, in the present embodiment, the specified object is stored in association with the address book data. However, the present invention is not limited to this, and may be associated with schedule data or mail data.

DESCRIPTION OF SYMBOLS 10 Mobile phone 100 Control part 101 Storage part 101a Program storage part 101b Various information temporary storage part 101c Control information storage part 101d Dictionary database storage part 101e Address book storage part 102 Wireless communication part 103 Voice signal processing part 104 Imaging part 105 Display part 106 Operation unit 107 RTC
108 Notification unit 108a Sound speaker 108b LED
108c Vibration motor 109 Battery part

Claims (19)

Image acquisition means for acquiring images;
Image display means for displaying on the screen at least a part of the image acquired by the image acquisition means;
Specifying means for specifying an object included in the image acquired by the image acquiring means;
Extracting means for extracting the object specified by the specifying means as candidate information;
Designation means for designating any of the candidate information extracted by the extraction means;
And an identification display means for identifying and displaying on the screen the display position of the object corresponding to the candidate information designated by the designation means when at least a part of the image is displayed on the screen by the image display means. A terminal device characterized by that. It further comprises imaging means for taking an image,
The terminal device according to claim 1, wherein the image acquisition unit acquires an image captured by the imaging unit.   The terminal device according to claim 1, further comprising a simultaneous display unit that simultaneously displays on a screen at least a part of the image acquired by the image acquisition unit and the candidate information specified by the specifying unit. Determining means for determining whether or not an object corresponding to the candidate information specified by the specifying means is displayed when at least a part of the image is displayed on the screen by the image display means;
2. The terminal according to claim 1, further comprising position information display means for displaying position information indicating a position of the corresponding object when the determining means determines that the corresponding object is not displayed. apparatus. Determining means for determining whether or not an object corresponding to the candidate information specified by the specifying means is displayed when at least a part of the image is displayed on the screen by the image display means;
And a display position changing unit for changing the display position of the image so that the corresponding object is displayed on the screen when the determining unit determines that the corresponding object is not displayed. The terminal device according to claim 1. Determining means for determining whether or not an object corresponding to the candidate information specified by the specifying means is displayed when at least a part of the image is displayed on the screen by the image display means;
The terminal device according to claim 1, further comprising notification means for notifying the fact that the determination means determines that the corresponding object is displayed.   The terminal device according to claim 1, wherein the specifying unit specifies a person as an object included in the image acquired by the image acquiring unit. The terminal device according to claim 1, further comprising a storage control unit that stores an object corresponding to the candidate information specified by the specifying unit in association with information stored in the storage unit. The terminal device according to claim 1, wherein the specifying unit specifies character information as an object included in the image acquired by the image acquisition unit. The terminal device according to claim 1, further comprising a setting unit that sets an object corresponding to the candidate information specified by the specifying unit as character information input by the character input unit. Image display means for displaying on the screen at least a part of the image picked up by the image pickup means;
Image updating means for updating display of the image by the image display means at a predetermined timing;
Specifying means for specifying first character information included in an image picked up by the image pickup means;
A designation means for designating the second character information;
A discriminating means for discriminating whether or not there is character information matching the second character information designated by the designation means in the first character information identified by the identifying means;
An identification display means for identifying and displaying the display position of the corresponding character information on the display screen when it is determined by the determination means that the corresponding character information exists;
An identification display update means for updating an identification display by the identification display means when the image update means updates an image display. 12. The terminal device according to claim 11, further comprising a simultaneous display unit that simultaneously displays on a screen at least a part of an image captured by the imaging unit and the second character information designated by the designation unit. The image display means further includes character information changing means for changing the second character information specified by the specifying means in a state where at least a part of the image picked up by the image pickup means is displayed on the screen. The terminal device according to claim 11. In the state where the image display means displays at least a part of the image picked up by the image pickup means on the screen, the corresponding character information matching the second character information specified by the specifying means is displayed on the screen. Display determination means for determining whether or not,
A position information display unit that displays position information indicating a position of the corresponding character information when the display determining unit determines that the corresponding character information is not displayed on the screen. 11. The terminal device according to 11. 15. The terminal device according to claim 14, further comprising position information update means for updating display of position information displayed by the position information display means when the image update means updates display of an image. In the state where the image display means displays at least a part of the image picked up by the image pickup means on the screen, the corresponding character information matching the second character information specified by the specifying means is displayed on the screen. Display determination means for determining whether or not,
Display position changing means for changing the display position of the image so that the corresponding character information is displayed on the screen when the display determining means determines that the corresponding character information is not displayed on the screen. The terminal device according to claim 11. In the state where the image display means displays at least a part of the image picked up by the image pickup means on the screen, the corresponding character information matching the second character information specified by the specifying means is displayed on the screen. Display determination means for determining whether or not,
The terminal device according to claim 11, further comprising notification means for notifying that the character information is displayed on the screen by the display determination means. On the computer,
An image acquisition step of acquiring an image;
An image display step of displaying on the screen at least a part of the image acquired in the image acquisition step;
A specifying step of specifying an object included in the image acquired in the image acquiring step;
An extraction step of extracting the object specified in the specifying step as candidate information;
A designation step for designating any of the candidate information extracted in the extraction step;
When at least a part of the image is displayed on the screen in the image display step, an identification display step for identifying and displaying on the screen the display position of the object corresponding to the candidate information specified in the specification step is executed. A program characterized by that. An image display step for displaying on the screen at least a part of the captured image;
An image update step for updating display of the image in the image display step at a predetermined timing;
A specifying step of specifying first character information included in the captured image;
A designation step for designating second character information;
A determining step of determining whether or not there is character information that matches the second character information specified in the specifying step among the first character information specified in the specifying step;
An identification display step for identifying and displaying the display position of the corresponding character information on the display screen when it is determined that the corresponding character information exists in the determination step;
An identification display update step for updating the identification display in the identification display step when the image update step updates the display of the image.

Images