JP5029986B2 - Information processing apparatus and program - Google Patents

Description

  The present invention relates to an information processing apparatus that processes sound and images and a program that causes a computer to function as the information processing apparatus.

  There is a sound source separation technique capable of separating and extracting only a target sound (hereinafter referred to as “target sound”) from sounds acquired using a plurality of microphones. The sound source separation technology sets a specific area including the direction in which the target sound arrives in advance, separates and extracts only the target sound that arrives from this specific area, and removes sound arriving from areas other than the specific area, etc. The method is used.

  For example, Japanese Patent Laid-Open No. 2001-84713 discloses a video camera-integrated sound source separation / sound collection microphone system that separates and extracts sounds generated at a focused position from ambient noise.

There is also a sound source localization technique that can specify the position (direction) of a sound source that generates a target sound. For example, Non-Patent Document 1 discriminates the position of a sound source that generates sound using a plurality of microphones, then determines the face of a person existing at that position, and determines the target sound generated by that person. A robot capable of separation and extraction is disclosed.
JP 2001-84713 A Masato Togami, Akio Amano, Hiroshi Shinjo, Ryota Kamoshida, “Hearing Function of Human Symbiotic Robot EMIEW”, Japanese Society for Artificial Intelligence, pp.59-64, 2005/10/14

  The video camera-integrated sound source separation / collection microphone system disclosed in Patent Document 1 cannot separate and extract a target sound when the sound source generating the target sound deviates from the focus position of the video camera.

In addition, the robot disclosed in Non-Patent Document 1 cannot separate and extract the target sound when the person generating the target sound deviates from the position determined using a plurality of microphone elements.
That is, in the conventional technique, once the sound source moves from the specified position, it becomes impossible to extract the sound from the sound source thereafter.

The present invention has been made in view of the above problems, and an object of the present invention is to enable separation and extraction of sound from a sound source even when the sound source is out of the region for specifying the direction of arrival of the sound to be extracted. And
Another object of the present invention is to make it possible to separate and extract sound from a sound source even when the sound source moves.

In order to solve the above problems, an information processing apparatus of the present invention provides:
An audio input means for inputting sound, an imaging unit for capturing an image, and display means for displaying the data, the captured image of the imaging means has captured, for specifying the direction of arrival of sound and the separation extraction target A display control unit that displays the specific region on the display unit, a sound source localization unit that identifies the direction of the sound source, and a direction of the sound source identified by the sound source localization unit in the specific region displayed by the display unit. Sound source presence / absence detecting means for detecting whether or not the sound source is detected; and when the sound source presence / absence detecting means detects that the direction of the sound source does not correspond to the specific area, an area position changing means for changing the position, the imaging means and the captured image captured is pre SL region display control hand to display a specific area, on the display means superimposed on the position change area position changing unit If the sound which the region display control unit comes from a direction in which the specific region to identify which is displayed on the display unit, and a sound source separating means for separating and extracting from the sound the sound input means inputs, the display control Means for determining whether or not a person is included in the captured image of the captured image; and when the person presence / absence determining means in the image determines that a person is included in the captured image In addition, when it is determined that the number of persons included in the captured image is plural and the number of persons determination means is not plural, the position of the entire area of the captured image of the imaging means is determined. Initial area setting means for setting the specific area as the position and size of the specific area, the captured image captured by the imaging means, and the specific area set by the initial area setting means are superimposed on the display means. And means for, and is characterized in that.

  Furthermore, a program for causing a computer to execute the main functions of the present invention is provided.

  According to the information processing apparatus of the present invention, when it is detected that there is no sound source generating the target sound in the currently set specific area, the position of the specific area is changed, and the changed specific area is changed. The sound from the sound source located at is separated and extracted. Thereby, for example, even if the sound source deviates from the current specific region, it is possible to change the position of the specific region so as to include the sound source and to separate and extract the sound from the sound source.

  Hereinafter, an information processing apparatus according to an embodiment of the present invention will be described with reference to FIGS. Below, the example which applies information processing apparatus to the portable terminal 1 provided with a communication function is demonstrated.

The mobile terminal 1 is a mobile phone device having a TV (TV) telephone function, and is, for example, a foldable type as shown in FIG. 1, and includes a keyboard 2, a camera 3, a display panel 4, a plurality of microphones 6, and a notification. LED8 etc. are provided.
The keyboard 2 is operated by the user in order to instruct various data and instructions, for example, start and end of a videophone, change of the size and position of a target sound range for specifying a voice to be extracted, and the like. used.

The camera 3 includes a CCD (Charge Coupled Devices) camera, a CMOS camera, and the like, and takes an image (still image or moving image), for example, an image for a videophone. The camera 3 has a zoom-in / zoom-out function, and can change the imaging range.
The display panel 4 includes an LCD (liquid crystal display) panel, a driver circuit, and the like, and displays an arbitrary image, for example, an image of a videophone speaker captured by the camera 3 or an image of a target sound range.

The call microphone 5 inputs call voice. Similarly to the microphone 6, the call microphone 5 may be used to specify the direction in which the sound source is located.
A plurality of microphones 6 are arranged, and each output a sound signal corresponding to the sound level of the collected sound. The number and position of the microphones 6 may be any number and position as long as the direction and direction of the sound source can be specified from the phase difference of the sound input from the microphone 6.
The speaker 7 outputs a received voice or the like.
The notification LED 8 notifies the user of various information by lighting / flashing.

  As shown in FIG. 2, the mobile terminal 1 includes a control unit 11, a wireless communication unit 12, a storage unit 13, an operation unit 14, an imaging unit 15, a display unit 16, a voice input unit 17, a voice output unit 18, and a notification unit 19. , Bus 20 and the like.

  The control unit 11 includes a CPU (Central Processing Unit) and the like, and controls the entire mobile terminal 1 according to an operation program stored in the storage unit 13. The control unit 11 includes a sound source localization separation unit 111, an image processing unit 112, a detection unit 113, a notification control unit 114, a target sound range manual change unit 115, and a target sound range automatic change unit 116. Prepare.

  The sound source localization separation unit 111 identifies the direction in which the sound source is located from the phase difference between the sounds input by the plurality of microphones 6 included in the sound input unit 17. Further, the sound source localization separation unit 111 separates and extracts only sounds that have arrived from a preset target sound range. Furthermore, the sound source localization separation unit 111 identifies each sound source of the sound (digital sound signal) input by the sound input unit 17, determines the sound level for each sound source, and identifies the sound with the highest sound level.

  The image processing unit 112 performs a process of superimposing the target sound range on the captured image at an arbitrary position and size. In addition, the image processing unit 112 identifies a person included in the captured image and identifies an area occupied by the identified person on the captured image. Furthermore, the image processing unit 112 generates an image in which the sound source specified by the sound source localization separation unit 111 is associated with the sound level of sound coming from the sound source.

  The detection unit 113 acquires the sound level of sound coming from within the target sound range from the sound source localization separation unit 111, and determines that the target sound source is within the target range if the sound level is equal to or higher than a predetermined level. Is determined. Further, the detection unit 113 acquires the feature points determined by the image processing unit 112 and detects the presence or absence of the target sound source within the target sound range. The notification control unit 114 controls the notification unit 19 or the audio output unit 18 to notify the user that there is no target sound source in the target sound range.

  The target sound range manual changing unit 115 sets the position and size of the target sound range in accordance with an instruction from the operation unit 14 in response to a user operation when target sound range change mode information 131 described later is set to “manual”. change.

  When the target sound range change mode information 131 described later is set to “automatic”, the target sound range automatic changing unit 116 responds to the detection unit 113 detecting that there is no target sound source in the target sound range. The target sound range is changed to include the target sound source. At this time, if the target sound source is not included in the captured image, the imaging range (for example, the angle of view) of the imaging unit 15 is changed to include the target sound source.

  The wireless communication unit 12 transmits call voice, captured images, and the like to the call destination via the base station. Further, the wireless communication unit 12 receives audio signals and image data from the communication partner via the base station.

  The storage unit 13 stores a program of the control unit 11, audio data, video data, and the like. For example, the storage unit 13 stores a control program for causing the control unit 11 to execute processing described later with reference to FIGS. 8, 9, and 11 to 16, a videophone application program, and the like. The storage unit 13 may be composed of either an internal memory or an external memory.

  The storage unit 13 also stores target sound range change mode information 131, target sound source presence / absence information 132, notification information 133, target sound range information 134, and target sound angle information 135.

The target sound range change mode information 131 indicates whether the target sound range is changed manually or automatically, as shown in FIG.
As shown in FIG. 4, the target sound source presence / absence information 132 indicates whether or not the target sound source exists in the target sound range during execution of a videophone or the like.

As shown in FIG. 5, the notification information 133 includes items of a notification mode, a notification method, and a notification operation.
“Notification mode” is an item indicating whether or not to notify the user that the target sound source does not exist in the target sound range. “Notification method” is an item indicating a notification method (any one of light-up, vibration, and audio output) used when notifying the user. “Notification operation” is an item indicating whether a user notification is being executed (execution or stoppage).

  As shown in FIG. 6, the target sound range information 134 is information that defines a rectangular target sound range set on the captured image, and includes four vertex coordinates (x, y). In the example of FIG. 17A, the target sound range is a region OS1 of x1 ≦ x ≦ x2 and y1 ≦ y ≦ y2 on the captured image PT1, and the target sound range information 134 is as shown in FIG. The coordinates of the four vertices are (x1, y2), (x1, y1), (x2, y2), (x2, y1).

  The target sound angle information 135 is information indicating an angle (direction) when the target sound source is viewed from the mobile terminal 1, and for example, as shown in FIG. 7, a predetermined reference ( Indicates the angle to the (0 degree) direction.

  The operation unit 14 shown in FIG. 2 includes a keyboard 2 and the like, inputs data and instructions, and gives them to the control unit 11.

  The imaging unit 15 includes the camera 3, captures an image (still image or moving image), and transmits the image to the control unit 11.

  The display unit 16 includes the display panel 4 and a driver circuit, and displays an image on the display panel 4 under the control of the control unit 11.

The audio input unit 17 includes microphones 5 and 6, and supplies the audio signal input by the microphones 5 and 6 to the control unit 11.
The sound output unit 18 outputs sound from the speaker 7 under the control of the control unit 11.

The notification unit 19 includes a notification LED, a vibration generation mechanism, and the like. Under the control of the control unit 11, the notification unit 19 notifies the user of various types of information through the lighting operation of the notification LED 8 and the vibration operation of the vibration generation mechanism.
The bus 20 transmits data between the units.

  Next, with reference to FIG. 8, FIG. 9 and FIGS. 11 to 16, the operation of the videophone of the portable terminal 1 having the above configuration will be described. Note that the communication operation itself of this mobile terminal is the same as that of a normal mobile phone, and the characteristic videophone operation of this embodiment will be described below.

When the user operates the operation unit 14 to instruct the start of the videophone, in response to the instruction, the control unit 11 starts the processing of the videophone application shown in FIG. First, the control unit 11 activates the imaging unit 15, acquires a captured image of the camera 3, supplies the image to the display unit 16, displays the image on the display panel 4, and starts processing (step S1).
Then, the control part 11 performs the target sound range determination process (step S2).

  The videophone function of the mobile terminal 1 separates the voice (target sound) of a specific speaker (sound source) when there are a plurality of persons (persons who may become speakers) in front of the mobile terminal 1・ Has a function to extract and send to the callee. The target sound range determination process (step S2) is a process of setting a target sound range, which is an image area for specifying a target sound (speaker, sound source) to be separated, on the display panel 4.

As shown in FIG. 9, in the target sound range determination process, the detection unit 113 determines whether or not there is a person (a person who may become a speaker) in the captured image. Is requested to the image processing unit 112 (step S21). The image processing unit 112 analyzes the captured image and determines whether or not there is a person in the captured image by determining the presence / absence / number of images similar to a human face by pattern matching, for example. It is determined whether the number of persons is plural, and the detection unit 113 is notified of the determination result.
Based on the notification from the image processing unit 112, the detection unit 113 determines whether the number of persons in the captured image is plural (two or more) (step S22).

If there are a plurality of images (step S22; Yes), the image processing unit 112 identifies the person with the largest face image. In the case of a videophone, the main speaker is located in front of the camera 3 and the face image of this speaker tends to be the largest. Therefore, in order to use the voice from this person as the target sound, an area occupied by this person in the captured image is specified, and the target sound range is overlaid on the area.
The control unit 11 causes the display panel 4 to display an image obtained by superimposing the captured image and the target sound range (step S24).

  Specifically, when the captured image is the captured image PT1 shown in FIG. 10A, it is determined that a plurality of human images P1, P2, and P3 exist in the captured image PT1, and the largest facial image is obtained. The person P1 is specified, the position occupied by the face image of the person P1 is specified, and as shown in FIG. 10B, the target sound range OS1 is displayed over the face image of the person P1.

At this stage, the user can change (edit) the size and position of the target sound range OS1 by operating keys on the keyboard 2.
For example, the user can instruct to widen the target sound range OS1 on the person P1 as shown in FIG. The image processing unit 112 changes the size of the target sound range OS1 in accordance with the instruction and displays it on the display panel 4 via the display unit 16. Further, for example, when the voice of the person P2 is set as the target sound and the user wants to selectively transmit the call to the destination, the user operates the operation unit 14 to move the target sound range OS1 to the area of the person P2.

  When the user determines the target sound range, as shown in FIG. 10B, it comes from the position (direction) of the sound source (persons P1 to P3, etc.) on the captured image PT1 and the direction corresponding to the position. A sound level display image VD1 that associates the sound level of the sound to be played may be displayed on the display unit 16.

  In this case, the sound source localization separation unit 111 identifies the direction in which the sound arrives and the sound level of the sound. The image processing unit 112 determines a position on the captured image PT1 based on the direction specified by the sound source localization separation unit 111, and generates the sound level display image VD1 by associating the position with the sound level. When step S24 ends, the process proceeds to step S25.

  On the other hand, if the number of persons is not plural (one or less) (step S22; No), the image processing unit 112 has at most one videophone speaker, and the voice of this speaker and the voices of other persons. There is no need to set the target sound range so as to be distinguished from each other. For this reason, the target sound range is defined as the entire region of the captured image so that the speaker is unlikely to deviate from the target sound range, and the control unit 11 displays an image obtained by superimposing the captured image and the target sound range via the display unit 16. It is displayed on the panel 4 (step S23).

  For example, when the captured image is the captured image PT2 shown in FIG. 10D, there is one person in the captured image PT2, and the image processing unit 112 sets the entire area of the captured image PT2 to the size of the target sound range OS2. Then, as shown in FIG. 10E, the image is displayed on the display panel 4 via the display unit 16. Also in step S13, an audio level display image VD2 substantially the same as the audio level display image VD1 may be generated and displayed on the display panel 4 via the display unit 16. When step S23 ends, the process proceeds to step S25.

When the operation unit 14 is operated by the user to instruct the determination of the target sound range, the control unit 11 generates target sound range information 134 indicating the target sound range set at that time, and as shown in FIG. It memorize | stores in the memory | storage part 13 (step S25). This is the initial value of the target sound range.
Subsequently, the control unit 11 obtains a target sound angle based on the set target sound range, and stores target sound angle information 135 indicating the target sound angle in the storage unit 13 as shown in FIG. A method for obtaining the target sound angle based on the target sound range will be described later. Thus, the target sound range determination process (FIG. 8, step S2) is completed, and the process proceeds to step S3 in FIG.

  Subsequently, the control unit 11 performs initial setting of the target sound source detection operation and the notification operation (step S3). Specifically, the control unit 11 sets the initial value of the target sound source presence / absence information 132 illustrated in FIG. 4 to “present” and the initial value of the notification method of the notification information 133 illustrated in FIG. 5 to “light-up”.

  In addition, the control unit 11 sets the operation mode (“manual” or “automatic”) selected by the operation unit 14 in accordance with the user operation in the target sound range change mode information 131 illustrated in FIG. Note that the notification control unit 114 sets the initial value of the notification mode of the notification information 133 illustrated in FIG. Thus, the initial setting (step S3) is completed, and the process proceeds to step S4.

  In step S4, the sound source localization separation unit 111 separates only the sound arriving from the target sound angle indicated by the target sound angle information 135 among the sounds collected by the microphone 6 as the target sound that is the voice of the videophone speaker. Extract. The control unit 11 transmits the extracted target sound data, that is, the voice of the videophone speaker along with the captured image of the camera 3 to the other party of the videophone via the wireless communication unit 12.

  The control unit 11 also receives audio data and image data from the other party of the videophone via the wireless communication unit 12, provides the audio data to the audio output unit 18, emits sound from the speaker 7, and outputs the image data. The image is displayed on the display panel 4 via the display unit 16.

The control unit 11 determines whether or not there is an instruction to end the videophone (step S5). When it is determined that the end is instructed (step S5; Yes), the control unit 11 causes the camera 3 to end the imaging operation via the imaging unit 15, and ends the display operation to the display panel 4 via the display unit 16 ( Step S7), the videophone call is terminated. In the case of the end of movie shooting, the image data captured from the imaging unit 15 from the start time and the audio data separated and extracted from the sound source localization separation unit 111 are stored in the storage unit 13.
When it is determined that there is no end instruction (step S5; No), the control unit 11 executes a target sound range changing process (step S6) corresponding to the current operation mode ("manual" or "automatic"). The process in step S6 is periodically executed during the videophone call.

  As shown in FIG. 11, in the target sound range changing process, first, a detection process is executed in order to determine whether or not the videophone speaker has moved from the set target sound range (step S1). S61). In the detection process, as shown in FIG. 12, the detection unit 113 requests the sound source localization separation unit 111 to determine the sound level of the sound coming from the sound source specified in the currently set target sound range. The sound source localization separation unit 111 determines the direction of sound arrival and the sound level of the sound from the correlation and phase difference of the sound signals input by the plurality of microphones 6 and notifies the detection unit 113 of the result.

  Based on the notification, the detection unit 113 determines whether or not the sound from the direction corresponding to the target sound range, that is, the target sound to be separated has arrived (step S611). The detection unit 113 determines that the target sound has arrived if the sound level of the sound from the direction corresponding to the target sound range is higher than a predetermined level, and determines that it has not arrived if the sound level is low. If it is determined that the target sound has arrived (step S611; Yes), the detection unit 113 sets the target sound source presence / absence information 132 to “present” (step S615), and the process returns to FIG. 11 and proceeds to step S62. .

  When the detection unit 113 determines that the target sound has not arrived (step S611; No), the speaker, that is, the target sound source does not exist (moves) in the target sound range, or the target sound source falls within the target sound range. It is determined whether or not sound is generated based on whether or not there is an image of the target sound source in the target sound range on the captured image. Therefore, the process proceeds to step S612.

In the present embodiment, the sound source is a person, and whether or not a sound source exists is determined based on whether or not there is an image of a person's face in the target sound range.
First, the detection unit 113 requests the image processing unit 112 to determine the presence or absence of a face within the target sound range (step S612). At this time, the detection unit 113 instructs the feature point (face (eye movement)) to be discriminated to the image processing unit 112, and the feature point discrimination result (face (eye movement) in the target sound range) from the image processing unit 112. Wait for notification.

Here, the presence / absence of eye movement is determined because there is a possibility that the image processing unit 112 may detect a person photograph instead of the person of the target sound source when the photographed image includes a person photograph. The image processing unit 112 detects whether or not the eye candidate detected as the feature point has a motion, and determines that the eye has been detected when the motion is detected.
Note that the image processing unit 112, for example, labels a black pixel group on a binarized captured image, identifies a black pixel region that is a candidate for constituting the right eye and the left eye, and centroid of the identified pixel region It is determined that there is a human eye movement from the movement mode.

  If there is a human face (eye movement) (step S612; Yes), the detection unit 113 sets the target sound source presence / absence information 132 to “present” (step S615), and the process returns to FIG. 11 and step S62. Proceed to If there is no face (eye movement) (step S612; No), the detection unit 113 requests the image processing unit 112 to determine whether there is a body (body movement) within the target sound range (step S613).

Here, the movement of the body is determined, for example, when the scene captured by the imaging unit 15 is a scene in which the board BD is arranged in the background of the person P1 shown in FIG. This is because there may be a case where the person's face (eyes) is not facing the imaging unit 15 and the eyes cannot be detected as in the state of writing the minutes toward the board BD. After the request for determination, the detection unit 113 waits until the determination result is notified from the image processing unit 112 in response to the request.
The image processing unit 112 extracts, for example, an edge line that is a candidate for a contour of a human body from a captured image that has been subjected to Sobel filter processing, obtains a centroid position of the edge line, and moves the obtained centroid position It is determined that there is a person's body movement.

  When the image processing unit 112 notifies that there is a body (movement of the body) (step 613; Yes), the detection unit 113 sets the target sound source presence / absence information 132 to “present” (step S615), and notifies that it does not exist. (Step S613; No), “None” is set (Step S614). Thus, the detection process (FIG. 11, step S61) ends, and the process proceeds to step S62.

  The notification control unit 114 refers to the target sound source presence / absence information 132 and determines whether or not the detection unit 113 has detected that the target sound source is within the currently set target sound range (step S62). If the target sound source is not in the target sound range (step S62; No), the notification control unit 114 determines whether or not to notify the user to that effect in the notification mode included in the notification information 133 shown in FIG. It is determined whether or not the setting is “ON” (step S63). If it is not on (step S63; No), the process proceeds to step S65. If it is ON (step S63; Yes), the notification control unit 114 executes notification processing in order to perform user notification according to the notification method set in the notification information 133 shown in FIG. 5 (step S64).

  As illustrated in FIG. 13, in the notification process, the notification control unit 114 determines whether the notification operation setting of the notification information 133 illustrated in FIG. 5 is “notifying” (step S641). If notification is in progress (step S641; Yes), there is no need to notify the user again, so the control unit 11 ends the notification process, the process returns to FIG. 11, and the process proceeds to step S65.

  If notification is not in progress (step S641; No), the notification control unit 114 sets the notification operation to “notifying” (step S642), and the notification method of the notification information 133 shown in FIG. It is determined whether or not there is (step S643). If it is light-up (step S643; Yes), the notification control unit 114 turns on the LED in the notification unit 19 (step S644), and if not (step S643; No), whether or not the notification method is “vibration”. A determination is made (step S645).

  If it is a vibration (step S645; Yes), the notification control unit 114 causes the notification unit 19 to perform a vibration operation (step S646). If not (step S645; No), whether or not the notification method is “voice output”. Is determined (step S647). If it is an audio output (step S647; Yes), the notification control unit 114 causes the audio output unit 18 to output a melody sound (step S648). Thus, the notification process (FIG. 11, step S64) ends, and the process proceeds to step S65 in FIG.

  The control unit 11 refers to the target sound range change mode information 131 and determines whether or not the operation mode is set to “manual” (step S65). If the operation mode is “manual” (step S65; Yes), the user is notified that the target sound source does not exist within the target sound range, and the target sound range designated by the user is newly set as the target sound range. In order to do this, the target sound range manual changing unit 115 executes a target sound range manual changing process (step S66).

  As illustrated in FIG. 14, in the target sound range manual change process, the target sound range manual change unit 115 determines whether the user has operated the operation unit 14 to change the currently set target sound range. Is determined (step S661). When the target sound range manual changing unit 115 determines that the change is not instructed (step S661; No), the process returns to FIG. 8 and returns to step S4.

  When the user operates the operation unit 14 to instruct a change in the position and size of the currently set target sound range (step S661; Yes), the target sound range manual change unit 115 is changed according to the instruction. The target sound range is set in the target sound range information 134 (step S662).

  Subsequently, the target sound range manual changing unit 115 obtains a target sound angle from the target sound range based on Equations 1 to 4 described later (step S663), and obtains target sound angle information 135 indicating the obtained target sound angle. It is generated and stored in the storage unit 13.

  A method for obtaining the target sound angle from the target sound range in step S663 will be described. As a specific example, the captured image when the target sound angle is obtained from the target sound range is set to the captured image PT1 shown in FIG. 17A, that is, the center of the angle of view θ3 of the imaging unit 15 shown in FIG. An image taken in the 90 degree direction of the reference (0 degree) will be described as an example.

The horizontal widths x1 and x2 and the horizontal width x3 of the captured image PT1 are determined from the x coordinate value corresponding to the other end when one end of each width corresponds to the origin (0) of the x coordinate on the captured image PT1. I want. In addition, the angle of view θ3 of the imaging unit 15 is an angle corresponding to the horizontal width x3 of the captured image PT1, and is a value determined in advance by the zoom function of the imaging unit 15. By substituting these into the following formulas 1 and 2, θ1 and θ2 shown in FIG.
(Equation 1) θ1 = (x1 / x3) × θ3
(Equation 2) θ2 = (x2 / x3) × θ3

Moreover, the target sound angles m degrees and n degrees shown in FIG. 17B can be expressed by the following formulas 3 and 4, respectively. Here, the target sound angle m degrees corresponding to the left end of the target sound range is obtained by substituting θ1 and the angle of view θ3 obtained from Expression 1 into Expression 3. Further, by substituting θ2 and the angle of view θ3 obtained from Equation 2 into Equation 4, the target sound angle n degrees corresponding to the right end of the target sound range is obtained.
(Equation 3) m = 90 + (θ3 / 2) −θ1
(Equation 4) n = 90 + (θ3 / 2) −θ2

  Equations 1 to 4 can be similarly applied even when the target sound angle is obtained in the target sound range automatic changing process (FIG. 11, step S67). Thus, the target sound range manual change processing (FIG. 11, step S66) is completed, the processing returns to FIG. 8, and returns to step S4.

  On the other hand, when the current operation mode determined in step S65 is not “manual” (FIG. 11, step S65; No), the target sound range automatic changing unit 116 has the target sound source within the target sound range. In response to detecting that the target sound source has not been detected, a target sound range automatic change process is executed in order to automatically change the target sound range to include the target sound source (step S67).

  In videophones, the voices of the main speakers tend to be loudest among the sounds arriving at the microphone 6 from around the mobile terminal 1. Therefore, as shown in FIG. 15, in the target sound range automatic change processing, first, the target sound range automatic change unit 116 determines the direction of arrival of the sound having the highest sound level among the sounds collected by the plurality of microphones 6. Search is performed (step S671).

At this time, the target sound range automatic changing unit 116 requests the sound source localization separating unit 111 to determine the direction and the sound level of the sound coming from the direction. The sound source localization separation unit 111 notifies the target sound range automatic change unit 116 of the determined direction and sound level in response to a request.
To explain with a specific example, in FIG. 18A, when the sound having the highest sound level among the sounds collected by the person P1 with the microphone 6 is generated, the sound source localization separation unit 111 determines, as a determination result, Information indicating the angle p degree shown in FIG. 18B and the sound level of the sound generated by the person P1 are notified to the target sound range automatic changing unit 116.

  The target sound range automatic changing unit 116 determines the presence or absence of the target sound source based on whether or not the sound level notified from the sound source localization separating unit 111 is equal to or higher than a predetermined level (step S672). This predetermined level is a reference for determining whether or not the sound with the highest sound level collected by the microphone 6 is the voice of the speaker of the videophone. For example, the predetermined level is an average sound level when a person talks. It is set.

  If it is smaller than the predetermined level, the target sound range automatic changing unit 116 determines that there is no target sound source (step S672; No), and determines that it is equal to or higher than the predetermined level (step S672; Yes). If it is determined that there is no target sound source, the process returns to FIG. 8 and proceeds to step S4. If it is determined that there is, the target sound range automatic changing unit 116 newly determines the target sound angle based on the angle (direction) at which the sound having the highest sound level acquired from the sound source localization separating unit 111 arrives (step S673). ).

  More specifically, the target sound range automatic changing unit 116 has a predetermined angle Ap in the circumferential direction from the direction notified by the sound source localization separation unit 111 (p degrees shown in FIGS. 18B and 19B). The (p + Ap) degree and (p-Ap) degree shifted by degrees are set as new target sound angles (FIG. 19B), target sound angle information 135 indicating the target sound angles is generated, and stored in the storage unit 13. Remember me.

  Next, the target sound range automatic changing unit 116 causes the image processing unit 112 to change the position of the target sound range on the captured image so as to correspond to the newly determined target sound angle (step S674). The image processing unit 112 causes the display unit 16 to display an image superimposed with the target sound range whose position has been changed.

Here, a method for obtaining the position of the target sound range on the captured image corresponding to the newly determined target sound angle will be described. Specifically, the target sound range automatic changing unit 116 performs imaging shown in FIG. 19A from the newly determined target sound angle (p + Ap) degree shown in FIG. An angle θ4 shown in FIG. 19B corresponding to the width x4 of the image PT1 is determined. Subsequently, the target sound range automatic changing unit 116 causes the imaging unit 15 to change the angle of view of the imaging unit 15 to θ4 degrees.
(Equation 5) θ4 = (p + Ap−90) × 2

Also, the angle θ2 corresponding to the horizontal width x2 of the target sound range is obtained by substituting newly defined target sound angles (p + Ap) degrees and (p−Ap) degrees into the following Equation 6. The horizontal position x2 of the target sound range OS1 is obtained by substituting θ4 obtained from Equation 5, θ2 obtained from Equation 6, and the lateral width x4 of the captured image PT1 into Equation 7 below. Thereby, the target sound range corresponding to the target sound angle after the change can be displayed on the display unit 16.
(Equation 6) θ2 = (p + Ap) − (p−Ap) = 2Ap
(Expression 7) x2 = (θ2 / θ4) × x4

  Equation 7 is also used to specify the position (direction) of the sound source on the captured image when the image processing unit 112 generates the sound level display image. More specifically, when the image processing unit 112 generates the sound level display image VD1 shown in FIG. 10B, the angle of view θ4 of the image pickup unit 15 specified from the zoom function of the image pickup unit 15 and FIG. Substituting the horizontal width x4 of the captured image PT1 and θ2 obtained from Equation 6 into Equation 7 to obtain the x coordinate value x2 corresponding to the direction of the sound source in the captured image. Thus, the target sound range automatic changing process (FIG. 11, step S67) is completed, the process returns to FIG. 8, and returns to step S4.

  On the other hand, when it is determined that the target sound source is in the target sound range (FIG. 11, step S62; Yes), the notification control unit 114 determines whether the notification mode is on (step S68). If it is not on (step S68; No), the process returns to step S4 in FIG. If it is on (step S62; No), the notification control unit 114 determines whether or not the notification operation of the notification information 133 indicates “notifying” in order to determine whether or not the user notification has already been executed. (Step S69). If the notification is not in progress (step S69; No), the process returns to step S4 in FIG.

  If notification is in progress (step S69; Yes), the notification control unit 114 executes notification cancellation processing (step S610). As shown in FIG. 16, in the notification release process, the notification control unit 114 sets the notification operation of the notification information 133 to “stopped” (step S6101), and whether or not the notification method is “light-up”. Is discriminated (step S6102). If it is light-up (step S6102; Yes), the notification control unit 114 causes the notification unit 19 to turn off the notification LED (step S6103). If not (step S6102; No), the notification control unit 114 determines whether the notification method is vibration. (Step S6104).

  If it is vibration (step S6104; Yes), the notification control unit 114 causes the notification unit 19 to cancel the vibration operation (step S6105). If not (step S6104; No), the notification control unit 114 determines whether the notification method is audio output (step S6104; No). Step S6106). If it is an audio output (step S6106; Yes), the notification control unit 114 causes the audio output unit 18 to cancel the output of the melody sound (step S6107). This completes the notification cancellation process (FIG. 11, step S610).

Thus, the target sound range changing process (step S6) is completed, the process returns to FIG. 8 and proceeds to step S4, where the sound source localization separation unit 111 separates and extracts sounds coming from the currently set target sound range. To do. The control unit 11 transmits the extracted target sound data together with the captured image of the camera 3 to the other party of the videophone via the wireless communication unit 12.
Thereafter, the portable terminal 1 repeatedly executes the processes of steps S4 to S6 until the videophone application is terminated.

As described above, according to the present embodiment, when the arrival direction of the target sound specified by the sound source localization separation unit 111 is outside the target sound range that specifies the arrival direction of the sound to be separated and extracted, The position of the target sound range is changed to correspond to the direction of arrival of the sound, and the sound source localization separation unit 111 separates and extracts the incoming sound from the target sound range after the position is changed.
As a result, even when the target sound range deviates from the target sound arrival direction, the position of the target sound range can be changed, and the target sound can be separated and extracted from the changed target sound range.

According to the present embodiment, when the setting of the operation mode when changing the target sound range is “automatic”, the target sound range automatic changing unit 116 does not involve the user's operation of changing the target sound range, and the sound source The position of the target sound range is changed so as to correspond to the arrival direction of the target sound specified by the localization separation unit 111. The sound source localization separation unit 111 separates and extracts only sounds coming from the changed target sound range.
Thus, even when there is no target sound source in the target sound range, the target sound range can be automatically changed to correspond to the arrival direction of the target sound, and only the target sound can be separated and extracted from the changed target sound range.

In addition, according to the present embodiment, when the operation mode is set to “automatic”, the target sound range automatic changing unit 116 determines that the sound source localization separating unit 111 does the localization if the target sound source image does not exist in the captured image. The imaging unit 15 is caused to change the angle of view of the camera 3 so that the direction of arrival of the target sound to be included (the direction of the sound source generating the target sound) is included.
Thus, even when the current direction of the target sound source does not correspond to the range currently captured by the camera 3, the angle of view of the camera 3 is changed, and the target sound source positioned in the direction of arrival of the target sound is captured. Can be included.

Furthermore, according to the present embodiment, when the operation mode is set to “automatic”, the target sound range automatic changing unit 116 can collect the sound of the sound collected by the microphone 6 even when the target sound source does not exist in the target sound range. If the largest level is equal to or higher than the predetermined level, the target sound range is changed to include the direction in which the sound indicating the sound level arrives, and the sound source localization separation unit 111 comes from the changed target sound range. Separate and extract only sound.
As a result, even when there is no target sound source in the target sound range, the sound with the highest sound level among the sounds collected by the microphone 6 is regarded as the target sound, and the direction in which the sound arrives is handled. To change the target sound range and separate and extract the sound. In this case, if the sound level of the target sound is maximum as compared with the sound levels of other sounds collected by the microphone 6, convenience for the user is improved.

According to the present embodiment, when the notification mode is set to “ON”, the notification control unit 114 notifies the user when the detection unit 113 detects that there is no target sound source in the target sound range. Notice.
Thereby, the user can grasp that the target sound source is out of the target sound range.

In addition, the notification method for making this notification can be changed by the user, and in this embodiment, it can be selected from among LED blinking, vibration, and audio output.
Thereby, the user can select an arbitrary notification method according to the current situation. For example, during a conference using a videophone, a notification method other than audio output can be selected in consideration of manners to the surroundings.

According to the present embodiment, when the operation mode for changing the target sound range is set to “manual”, the target sound range manual changing unit 115 allows the user to operate the operation unit 14 to change the position and size. The target sound range information 134 indicating the changed target sound range is generated and stored in the storage unit 13.
Thus, for example, when the target sound source is out of the target sound range, the user changes the position and size of the target sound range so that the target sound source is included, and the target sound is separated and extracted from the changed target sound range. Can be adjusted.

According to the present embodiment, the user can arbitrarily change (edit) the position and size of the target sound range by operating the operation unit 14, and the image processing unit 112 can respond to the position and size changed by the user. The sound source localization separation unit 111 separates and extracts sound coming from the direction corresponding to the target sound range.
Thus, the user can change the target sound range and adjust the target sound separated and extracted from the changed target sound range according to his / her preference.

At this time, the control unit 11 causes the display panel 4 to display an image in which the target sound range is superimposed on the captured image via the display unit 16.
Thereby, the user can visually confirm the target sound source within the target sound range. For example, the user can grasp at a glance that the target sound source with the wrong target sound range is specified, and the operability is improved.

Further, according to the present embodiment, when the user changes (edits) the size or position of the target sound range, the image processing unit 112 performs sound input by the plurality of microphones 6 specified by the sound source localization separation unit 111. A position corresponding to the arrival direction on the captured image PT1 is determined, and the position and the sound level are associated with each other to generate a sound level display image VD1, and the control unit 11 causes the display panel 16 to display the sound level display image VD1.
Thereby, the user can change the target sound range after grasping the direction in which the sound level of the target sound is maximum on the captured image.

According to the present embodiment, when the initial value of the target sound range is set, if a plurality of persons are included in the captured image of the camera 3, the image processing unit 112 has the largest face image among them. Is specified, the area occupied by the person in the captured image is specified, and the target sound range is overlaid on the area.
Thus, for example, when a main speaker is located in front of the camera 3 during a videophone call, the target sound range is displayed so that the voice of the speaker is set as the target sound, so the user sets the target sound range. Convenience when doing is increased.

Further, according to the present embodiment, when setting the initial value of the target sound range, if the number of persons included in the captured image of the camera 3 is one or less, the image processing unit 112 covers the entire area of the captured image. Overlap the target sound range.
Accordingly, when it is not necessary to distinguish between the voice of a person included in the captured image and the voice of another person, it is possible to make it difficult for the person to be out of the target sound range.

According to the present embodiment, the detection unit 113 obtains the sound level of sound arriving from the target sound range set to the initial value from the sound source localization separation unit 111, determines whether or not it is equal to or higher than the predetermined level, and determines the predetermined level. If it is above, it will discriminate | determine that there exists a sound source which generate | occur | produces a target sound in the target sound range.
This increases the possibility that the target sound source that generates the target sound is present in the target sound range set to the initial value.

  Further, according to the present embodiment, the detection unit 113 detects the person's face on the captured image within the target sound range if the sound level of the sound coming from the target sound range set to the initial value is lower than the predetermined level. It is determined whether there is a target sound source. Thus, when the speaker is the target sound source, it can be determined whether the speaker has moved out of the target sound range or whether the speaker is within the target sound range but not speaking.

According to the present embodiment, when determining whether or not there is a human face image on the captured image within the target sound range, the image processing unit 112 has an eye movement on the captured image within the target sound range. The determination is made based on whether or not.
As a result, even when a captured image includes a speaker and a person photograph, it is possible to detect the movement of the speaker's eyes and separate the speaker from the person photograph.

  Further, when the detection unit 113 determines that no face exists in the target sound range, the detection unit 113 determines the presence or absence of a human body, and if there is, determines that the target sound source is within the target sound range. Thereby, even when a person's face is not facing the camera 3, it can be detected that a speaker which is the target sound source of the speaker is within the target range.

Hereinafter, modified examples and application examples of the present invention will be described.
The shape of the target sound range is not limited to a rectangle, and may be, for example, a shape that matches the contour of a person image on the captured image. In this case, the image processing unit 112 may analyze the captured image, determine a region similar to the whole image of the human face by pattern matching, for example, and determine the target sound range.

  In order to make it difficult to deviate the position of the target sound source from the target sound range, the target sound range may be made as large as possible within a limit that allows the target sound source to be identified from other sound sources. Therefore, a message prompting the user to set the target sound range as large as possible may be displayed on the display unit 16.

The detection process (FIG. 11, step S61) is not limited to detecting the body (body movement) after detecting the face (eye movement). For example, when a plurality of persons are included in the captured image, the step of detecting the body (body movement) may be omitted if it is difficult to identify the person of the target sound source even if the body is detected. . Further, for example, when the face of the person of the target sound source does not face the imaging unit 15, the body (body movement) is detected without determining that the target sound source is within the target range by detecting the face (eye movement). The detection may be performed continuously.
These combinations may be arbitrarily determined according to the performance of the image processing unit 112 and the usage status of the user.

  The notification method for notifying the user that there is no target sound source within the target sound range is not limited to LED lighting, vibration operation, and sound output, for example, a message indicating that there is no target sound source is displayed on the display panel 4 You may make it do.

  In addition, the concept of the present invention is not limited to a dedicated computer system, and can be applied to any portable electronic device such as a mobile phone, a PDA, an electronic camera, and the like that includes an imaging unit and a plurality of audio input units. . That is, a computer program for causing a computer to function and operate as the mobile terminal 1 is created, distributed, and lent, and this is installed in the computer so that the mobile terminal 1 can be used, transferred, or lent. Good.

It is an external view when it sees from the front in the state which opened the portable terminal. It is a block diagram which shows the structure of a portable terminal. It is a figure which shows the example of the target sound range change mode information. It is a figure which shows the example of target sound source presence / absence information. It is a figure which shows the example of notification information. It is a figure which shows the example of target sound range information. It is a figure which shows the example of target sound angle information. It is a flowchart which shows the process in which a portable terminal isolate | separates and extracts a target sound from the starting to the end of a videophone function. It is a flowchart which shows the target sound range determination process. (A) is a figure which shows the captured image containing a some person. (B) is a figure which shows the image and audio | voice level display image which were superimposed on the face of the person with the largest target sound range of a face image on the captured image. (C) is a figure which shows the image on which the target sound range was superimposed on the whole body of the person with the largest face image on the captured image. (D) is a figure which shows the captured image in which only one person is included. (E) is a figure which shows the image and audio | voice level display image in which the target sound range pointed out the whole area | region of the captured image on the captured image. It is a flowchart which shows the change process of the target sound range corresponding to an operation mode. It is a flowchart which shows a detection process. It is a flowchart which shows a notification process. It is a flowchart which shows the target sound range manual change process. It is a flowchart which shows the target sound range automatic change process. It is a flowchart which shows a notification cancellation | release process. (A) is a figure which shows the image on which the captured image and the target sound range were overlapped. (B) is a figure which shows the angle corresponding to the angle of view and target sound range of an imaging part. (A) is a figure which shows the captured image before the target sound range change, and the position of the target sound source. (B) is a figure which shows the angle of view of the imaging part before the target sound range change. (A) is a figure which shows the captured image after the target sound range change, and the position of the target sound source. (B) is a figure which shows the angle of view of the imaging part after the target sound range change.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Portable terminal, 11 ... Control part, 12 ... Wireless communication part, 13 ... Memory | storage part, 14 ... Operation part, 15 ... Imaging part, 16 ... Display part, 17 ... Audio | voice input part, 18 ... Audio | voice output part, 19 ... Notification unit, 20 ... bus, 111 ... sound source localization separation unit, 112 ... image processing unit, 113 ... detection unit, 114 ... notification control unit, 115 ... target sound range manual change unit, 116 ... target sound range automatic change unit, 131 ... target sound range change mode information, 132 ... target sound source presence / absence information, 133 ... notification information, 134 ... target sound range information, 135 ... target sound angle information, PT1 ... captured image, PT2 ... captured image, OS1 ... target sound range, OS2 ... target sound range, P1, P2, P3, P4 ... person, VD1, VD2 ... sound level display image, BD ... board, x1, x2 ... horizontal width on the picked-up image, x3 ... horizontal width of the picked-up image, x4 ... Width of captured image y1, y2: vertical height on the captured image, y3: height of the captured image, m: target sound angle corresponding to the left end of the target sound range, n: target sound angle corresponding to the right end of the target sound range, p: angle of the sound source, A1: angle at the left end of the captured image, A2: angle at the right end of the captured image, Ap: predetermined angle, θ1, θ2, angle, θ3, θ4: angle of view of the imaging unit

Claims (12)

Voice input means for inputting sound;
An imaging means for capturing an image;
Display means for displaying data;
Display control means for displaying the picked-up image picked up by the image pick-up means and a specific area for specifying the direction of arrival of the sound to be separated and extracted on the display means;
Sound source localization means for specifying the direction of the sound source;
Sound source presence / absence detecting means for detecting whether or not the direction of the sound source specified by the sound source localization means corresponds to the specific area displayed by the display means;
Area position changing means for changing the position of the specific area so as to coincide with the direction of the sound source when the sound source presence / absence detecting means detects that the direction of the sound source does not correspond to the specific area;
A captured image in which the imaging means has captured a certain area at the position changed before Symbol area position changing unit, and a region display control means for displaying on the display means repeatedly,
Sound source separation means for separating and extracting sound coming from the direction specified by the specific area displayed on the display means by the area display control means from the sound input by the voice input means ;
The display control means includes
In-image person presence / absence determining means for determining whether or not a person is included in the captured image of the captured image;
A number-of-persons determination unit that determines whether or not the number of persons included in the captured image is plural when the person presence / absence determination unit in the image determines that a person is included in the captured image;
An initial region setting unit that sets the position and size of the entire region of the captured image of the image capturing unit as the position and size of the specific region when the number of persons determination unit is determined to be not plural;
Means for superimposing the captured image captured by the imaging unit and the specific region set by the initial region setting unit on the display unit;
Equipped with a,
An information processing apparatus characterized by that. The region position changing means is
When the sound source presence / absence detecting unit detects that the direction of the sound source does not correspond to the specific region, the sound source localization unit searches for a direction in which sound arrives, and
An image position specifying means for specifying a position corresponding to the direction searched by the direction searching means on the captured image picked up by the image pickup means;
Changing means for changing the position of the specific area to the position specified by the image position specifying means;
The information processing apparatus according to claim 1, comprising: The area display control means includes
A sound source image presence / absence determining means for determining whether or not a sound source located in the direction searched by the direction searching means exists on a captured image captured by the imaging means;
If the sound source in the sound source image presence determining means is determined to not present on the captured image, the image pickup means, to change the angle of view to include the image of the sound source to the shooting Zoga image, the field angle The information processing apparatus according to claim 2, further comprising: a display unit configured to superimpose the captured image captured by the imaging unit that has changed the image and the specific region on the display unit. The area display control means includes
When the sound source image presence / absence determining means determines that a sound source is present on the captured image, the sound source image specifies the size of the sound source image located in the direction searched by the direction search means in the captured image captured by the image capturing means Size identification means;
The size of the specific area is changed to the size specified by the sound source image size specifying means, and the specific area size is displayed on the display means by superimposing the captured image picked up by the imaging means and the specific area having the changed size. The information processing apparatus according to claim 3, further comprising a changing unit. The direction searching means includes
When the sound source presence / absence detecting means detects that the direction of the sound source does not correspond to the specific area, the maximum level direction search for searching for the direction of arrival of the sound having the highest level among the sounds input by the sound input means Means,
The sound source image size specifying means determines whether or not the level of sound coming from the direction searched by the maximum level direction searching means is equal to or higher than a predetermined level. The information processing apparatus according to claim 4, wherein the size is specified. The sound source presence / absence detecting means includes
An incoming sound level discriminating means for discriminating whether or not the level of sound coming from the direction specified by the specific region is a predetermined level or higher;
Feature point presence / absence discrimination that discriminates whether or not a feature point of a person exists in an image within the specific region of the captured image of the imaging unit when the incoming sound level determination unit determines that the level is not equal to or higher than a predetermined level. Means,
Means for discriminating that there is a sound source for generating a sound to be separated and extracted in a direction corresponding to the specific area when the characteristic point presence / absence determining means determines that the feature point of the person is in the image in the specific area; When,
The information processing apparatus according to claim 1, further comprising: The region position changing means is
When the sound source presence / absence detection unit detects that the direction of the sound source does not correspond to the specific area, a notification unit that notifies the fact,
Changing means for changing at least one of the position and size of the specific area in response to a user operation;
The information processing apparatus according to claim 1, further comprising: The display control means includes
In-image person presence / absence determining means for determining whether or not a person is included in the captured image of the captured image;
If the image a person existence determination means has determined that a person is included in the captured image, and number of people s discrimination means the number of persons included in the captured image to determine more or not,
A face maximum person specifying means for specifying a person with the largest face image in the captured image of the imaging means when the number of persons determining means determines that the number of persons is plural;
Person area specifying means for specifying the image area of the person specified by the maximum face person specifying means in the captured image of the image pickup means;
Second initial area setting means for presetting the specific area so as to include at least a part of the image area of the person specified by the person area specifying means;
Means for displaying the picked-up image picked up by the image pickup means and the specific area set by the second initial area setting means on the display means in an overlapping manner;
The information processing apparatus according to any one of claims 1 to 7 , further comprising: A direction-of-arrival image specifying means for specifying the direction in which the sound input by the sound input means arrives in the captured image captured by the imaging means;
Image generation means for generating a direction level-corresponding image in which the direction specified by the direction-of-arrival image specifying means and the level of sound coming from the direction are associated with each other;
A corresponding image display control unit that causes the display unit to display a direction level corresponding image generated by the image generation unit by superimposing the captured image captured by the imaging unit and the specific region;
While the associated image display control means displays the direction level associated image on the display means, in response to a user operation, at least one of the position and size of the specific area displayed by the display means is displayed. Level visible area changing means to be changed;
The information processing apparatus according to any one of claims 1 to 8, further comprising a. Communication means for communicating with other devices;
Application execution means for executing an application that uses the communication means,
The communication unit transmits at least one of the sound data separated and extracted by the sound source separation unit and the captured image data captured by the imaging unit to the other device while the application execution unit is executing the application. to, the information processing apparatus according to any one of claims 1 to 9, characterized in that. Storage means for storing data;
Means for storing in the storage means at least one of the sound separated and extracted by the sound source separation means and the captured image captured by the imaging means;
The information processing apparatus according to any one of claims 1 to 10, characterized in that. The computer,
A captured image pickup means has captured the display control means and the specific region, Ru is displayed on the display unit by overlapping to identify the direction of arrival of the sound with the separation target,
Sound source localization means for specifying the direction of the sound source ,
The sound source localization means the direction of the sound source identified, the sound source detecting means, wherein the display control means detects whether or not corresponding to the specified area is displayed on the display unit,
Area position changing means for changing the position of the specific area so as to coincide with the direction of the sound source when the sound source presence / absence detecting means detects that the direction of the sound source does not correspond to the specific area;
Wherein the captured image pickup means has captured the area position changing unit and a specific area in the position has changed, the region display control means for displaying on the front Symbol display section superimposed,
A sound coming from direction in which the region display control means for specifying a specific area is displayed on the display unit, the sound source separating means for separating and extracting from the input sound, to function as,
The display control means includes
In-image person presence / absence determining means for determining whether or not a person is included in the captured image of the captured image;
A single person determination unit for determining whether or not the number of persons included in the captured image is plural when the person presence / absence determination means in the image determines that a person is included in the captured image;
An initial region setting unit that sets the position and size of the entire region of the captured image of the imaging unit as the position and size of the specific region when the number of persons determination unit is determined to be not plural;
Causing the captured image captured by the imaging unit and the specific region set by the initial region setting unit to overlap and display on the display unit;
A program characterized by that .

Images