JP2012252524A - Display device, display method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily display a desired image at a desired position in an image.SOLUTION: The display device includes: an image acquisition unit 6a for acquiring an image displayed in a display area of a display unit 8; a sound collection unit 3a for collecting sound generated from a sound source; a sound source direction identification unit 3d for identifying a sound source direction of the sound collected by the sound collection unit 3a; a position determination unit 6f for, on the basis of the sound source direction of the sound identified by the sound source direction identification unit 3d, determining a display position in a display area of sound related image associated with the sound; and a display control unit 7 for displaying the sound related image on the display position determined by the position determination unit 6f so as to be superimposed on a predetermined image displayed in the display area.

Description

  The present invention relates to a display device, a display method, and a program.

  2. Description of the Related Art Conventionally, a photo creation apparatus that combines a stamp image and a display image is known (see, for example, Patent Document 1).

JP 2004-159158 A

  However, in the case of Patent Document 1, the user has to operate the key on which the stamp image is displayed or to specify the position where the stamp image is to be combined.

  An object of the present invention is to provide a display device, a display method, and a program that can easily display a desired image at a desired position of the image.

In order to solve the above problems, the display device of the present invention provides:
In a display device comprising display means,
Obtaining means for obtaining an image displayed in a display area of the display means;
Sound collecting means for collecting sounds emitted from the sound source;
Direction specifying means for specifying the sound source direction of the sound collected by the sound collecting means with reference to the position of the apparatus body;
Position determining means for determining a display position in the display area of the sound-related image related to the sound based on the sound source direction of the sound specified by the direction specifying means;
Display control means for displaying the sound-related image at the display position determined by the position determination means so as to overlap the image displayed in the display area;
It is characterized by having.

Moreover, the display method of the present invention includes:
A display method using a display device comprising display means and sound collection means for collecting sound emitted from a sound source,
Processing for displaying an image in a display area of the display means;
A process of identifying the sound source direction of the sound collected by the sound collecting means with reference to the position of the display device body;
A process of determining a display position in the display area of a sound-related image related to the sound based on the sound source direction of the specified sound;
Processing to display the sound-related image at the determined display position so as to overlap the image displayed in the display area;
It is characterized by performing.

The program of the present invention is
A computer of a display device comprising display means and sound collection means for collecting sounds emitted from a sound source,
Obtaining means for obtaining an image displayed in a display area of the display means on the basis of the position of the display device body;
Direction specifying means for specifying the sound source direction of the sound collected by the sound collecting means;
Position determining means for determining a display position in the display area of the sound-related image related to the sound based on the sound source direction of the sound specified by the direction specifying means;
Display control means for displaying the sound-related image at the display position determined by the position determination means so as to overlap the image displayed in the display area;
It is characterized by making it function as.

  According to the present invention, a desired image can be easily displayed at a desired position of the image.

It is a block diagram which shows schematic structure of the display apparatus of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to sound-related image display processing by the display device of FIG. It is a figure which shows typically an example of the image which concerns on the sound related image display process of FIG. It is a figure which shows typically an example of the image which concerns on the sound related image display process of FIG. It is a figure which shows typically an example of the image which concerns on the sound related image display process of FIG. It is a figure which shows typically an example of the image which concerns on the sound related image display process of FIG.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram showing a schematic configuration of a display device 100 according to an embodiment to which the present invention is applied.

The display device 100 according to the present embodiment displays a sound-related image related to the sound collected by the sound collection unit 3a so as to overlap a predetermined image displayed in the display area R of the display unit 8. At that time, the sound source direction of the sound collected by the sound collecting unit 3a is specified, the display position in the display area R of the sound related image is determined based on the specified sound source direction, and the sound related image is displayed at the determined display position. indicate.
Specifically, the display device 100 is a digital photo frame or the like installed on a table, for example. As shown in FIG. 1, the central control unit 1, the operation input unit 2, the sound processing unit 3, and the memory 4. A recording medium control unit 5, an image processing unit 6, a display control unit 7, and a display unit 8.

  The central control unit 1 controls each unit of the display device 100. Specifically, although not shown, the central control unit 1 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory), and various processing programs (illustrated) for the display device 100. Various control operations are performed in accordance with operation signals input by the operation input unit 2.

  The operation input unit 2 is for inputting various instructions to the display device 100. Specifically, an operation unit including a data input key for inputting characters and the like, an up / down / left / right movement key for selecting or specifying data, various function keys, and the like is provided. The central control unit 1 causes each unit to execute a predetermined operation according to the operation signal output from the operation input unit 2 and input. In addition, the operation input unit 2 may include other input devices such as a remote controller and a touch panel.

  The sound processing unit 3 includes a sound collecting unit 3a, a recording unit 3b, a volume specifying unit 3c, a sound source direction specifying unit 3d, a sound recognizing unit 3e, an individual identifying unit 3f, and a voice information table T1. ing.

The sound collection unit 3a is a microphone or the like and collects sound emitted from a sound source S such as a human or an animal.
Specifically, for example, the sound collection unit 3a receives vibrations of sound emitted from the sound source S at two locations on the upper right side and lower left side of the display area R of the display unit 8 when the display device 100 is viewed from the front. Input sections a1 and a1 are provided (see FIG. 3 and the like), and sound data is generated by subjecting the vibration of the sound input to the input section a1 to A / D conversion or the like.
Note that the number and arrangement position of the input unit a1 of the sound collecting unit 3a included in the display device 100 can be arbitrarily changed as long as the sound source direction of the sound collected by the sound collecting unit 3a can be specified. .

The recording unit 3b starts recording the sound data of the sound collected by the sound collection unit 3a at a predetermined recording start timing, and ends the recording at a predetermined recording end timing.
Here, the recording start timing and the recording end timing may be any timing according to an instruction from the user.
Specifically, the recording unit 3b performs recording based on a predetermined recording instruction signal output from the central control unit 1 in response to a predetermined operation of a recording instruction key (not shown) of the operation input unit 2 by the user. Specify the start timing and recording end timing. For example, the recording unit 3b determines that the recording start timing is reached when the recording instruction key is pressed (for example, pressed) by the user, and determines that the recording end timing is reached when the predetermined operation is released. Also good. The recording unit 3b may determine that the recording start timing is reached when the recording instruction key is operated by the user and that the recording end timing is reached when the recording instruction key is operated again.

  The recording unit 3b specifies the recording start timing and the recording end timing based on the input of a predetermined recording instruction signal output from the central control unit 1 in response to a predetermined gesture made by the user to the imaging unit (not shown). You may do it. That is, for example, the recording unit 3b determines that the recording start timing is reached when a predetermined gesture by the user is imaged by the imaging unit, and determines that the recording end timing is reached after a predetermined time has elapsed from the recording start timing. May be. Further, the recording unit 3b determines that the recording start timing has come when the predetermined recording start instruction gesture by the user is imaged by the imaging unit, and the predetermined recording end instruction gesture by the user is imaged by the imaging unit. It may be determined that the recording end timing has come.

The volume specifying unit 3c specifies the volume of the sound collected by the sound collecting unit 3a as a volume specifying unit.
Specifically, the sound volume specifying unit 3c, for example, based on the sound data of the sound collected by the sound collecting unit 3a from the recording start timing to the recording end timing and recorded in the recording unit 3b (for example, , Voice, etc.).
Note that the volume of the sound collected by the sound collection unit 3a can be specified using a known method, and thus detailed description thereof is omitted here.

The sound source direction specifying unit 3d specifies the sound source direction of the sound collected by the sound collecting unit 3a as direction specifying means.
Specifically, the sound source direction specifying unit 3d, for example, based on the sound data of the sound collected by the sound collecting unit 3a and recorded in the recording unit 3b from the recording start timing to the recording end timing, The position of the sound source S (individuals such as humans and animals) with respect to the display device 100 is specified. That is, the sound source direction specifying unit 3d, for example, based on the difference between the time when the sound from the sound source S reaches one input unit a1 of the sound collection unit 3a and the time when the sound reaches the other input unit a1. The position (for example, the left side of the display device 100 in FIG. 3B, the right side in FIG. 4B, etc.) with respect to the display device 100 is specified. Then, the sound source direction specifying unit 3d sets the direction toward the position of the sound source S relative to the display device 100 (for example, the left direction in FIG. 3B, the right direction in FIG. 4B, etc.) as the sound source direction.

Note that the position and the sound source direction of the sound source S with respect to the display device 100 may be specified with reference to a two-dimensional space substantially parallel to a plane including the display region R of the display device 100, and the display of the display device 100 You may specify on the basis of the three-dimensional space which added the front-back direction substantially orthogonal to the area | region R. FIG.
Further, the sound source direction specifying unit 3d, for example, with respect to the display device 100 of the sound source S from a plurality of predetermined directions (for example, four directions such as an upward direction, a downward direction, a left direction, and a right direction). The direction closest to the direction toward the position may be the sound source direction. Here, the direction closest to the direction toward the position of the sound source S relative to the display device 100 is a straight line extending in each direction and the position of the sound source S relative to the display device 100 among a plurality of predetermined directions. This is the direction in which the angle formed with the straight line extending in the direction toward the direction is the smallest.

The sound recognition unit 3e recognizes the sound collected by the sound collection unit 3a as sound recognition means.
Specifically, the sound recognizing unit 3e is based on sound data of sound (particularly, sound) collected by the sound collecting unit 3a and recorded in the recording unit 3b from the recording start timing to the recording end timing, for example. The character data for recognizing the sound and expressing it with the corresponding character is generated.
Note that the sound collected by the sound collection unit 3a can be recognized using a known method such as recognition using a predetermined speech recognition dictionary, and thus detailed description thereof is omitted here.

The sound recognition unit 3e may recognize the meaning content of the sound collected by the sound collection unit 3a.
Here, since the meaning content of the sound collected by the sound collection unit 3a can be recognized using a known method such as recognition using a predetermined semantic analysis dictionary, detailed description thereof is omitted here. To do.

The individual identifying unit 3f identifies an individual that has emitted the sound collected by the sound collecting unit 3a.
Specifically, the individual identifying unit 3f, for example, matches the sound based on the sound data of the sound collected by the sound collecting unit 3a and recorded in the recording unit 3b from the recording start timing to the recording end timing. It is determined whether the audio information to be stored is stored in the audio information table T1. When it is determined that the voice information that matches the voice is stored in the voice information table T1, the individual identification unit 3f acquires the identification information corresponding to the voice information that matches the voice from the voice information table T1. The individual that has produced the sound collected by the sound collection unit 3a is identified.

In addition, the audio | voice information table T1 memorize | stores the audio | voice information used for identification of each individual | organism | solid of at least one among a human and an animal as a 2nd memory | storage means.
Specifically, the audio information table T1 includes, for example, audio information (for example, audio feature information) for identifying the audio of an individual such as a human or an animal, and identification information (for example, for identifying the individual) Name) is stored in association with each other.
Here, the voice feature information includes, for example, information such as a voice print. However, any information is not limited to this as long as it is information for identifying the voice of an individual such as a human or an animal. Also good.

  The memory 4 is composed of a ROM such as a DRAM (Dynamic Random Access Memory) or a NAND flash memory, for example, and temporarily stores data processed by the central control unit 1, the sound processing unit 3, the image processing unit 6, and the like. To remember.

The recording medium control unit 5 is configured such that the recording medium 51 is detachable, and controls reading of data from the loaded recording medium 51 and writing of data to the recording medium 51.
That is, for example, the recording medium control unit 5 controls reading of predetermined image data to be displayed from the recording medium 51 attached after being detached from an external device such as an imaging device.
The recording medium 51 is composed of, for example, a non-volatile memory (flash memory) such as an SD card or a USB memory. However, the recording medium 51 is an example and is not limited thereto, and can be arbitrarily changed as appropriate. .

  The image processing unit 6 includes an image acquisition unit 6a, a face detection unit 6b, a dimension determination unit 6c, a direction determination unit 6d, a sound related image generation unit 6e, a position determination unit 6f, and a face identification information table T2. It is equipped with.

The image acquisition unit 6a acquires image data of an image displayed in the display area R of the display unit 8, that is, an image to be displayed.
Specifically, the image acquisition unit 6a acquires image data read from the recording medium 51 based on a predetermined operation of the operation input unit 2 by the user as display target image data. In addition, the image acquisition unit 6a acquires image data designated based on a predetermined operation of the operation input unit 2 by the user among the image data stored in the recording medium 51 as image data to be displayed.

The face detection unit 6b detects a face such as a human or an animal from the display target image acquired by the image acquisition unit 6a.
Specifically, the face detection unit 6b performs a predetermined face detection process on the image data of the display target image acquired by the image acquisition unit 6a, and determines the face regions of all the faces included in the image. To detect.
Since the face detection process is a known technique, detailed description thereof is omitted here.

The dimension determination unit 6c, as a dimension determination unit, determines the dimension of the sound-related image generated by the sound-related image generation unit 6e described later based on the volume specified by the volume specification unit 3c and displayed in the display region R. To do.
Here, the sound-related image is a character image M in which a sound recognized by the sound recognition unit 3e is represented by a corresponding character.
The character image M may include a speech bubble image including a lead line L connected to a predetermined position of a frame portion surrounding the character. That is, the character image M may be an image consisting only of an image (character body image) based on the character data generated by the sound recognition unit 3e, or an image based on the character data generated by the sound recognition unit 3e ( It may be an image made up of a character body image) and a speech bubble image. Note that whether or not to include a speech bubble in the character image M depends on whether or not there is a predetermined instruction from the user, the volume specified by the volume specifying unit 3c, whether or not there is a face in the display target image, and the sound recognition unit 3e. You may select according to the meaning content etc. of the sound recognized by.

Specifically, the dimension determining unit 6c displays the characters displayed in the display region R such that the larger the volume specified by the volume specifying unit 3c, the larger the character image M displayed in the display region R. The size of the image M is determined. That is, for example, when the volume specified by the volume specifying unit 3c is less than a predetermined first volume threshold, the dimension determining unit 6c determines a predetermined “first dimension” as the size of the character image M. In addition, when the volume specified by the volume specifying unit 3c is greater than or equal to the first volume threshold and less than the predetermined second volume threshold, the dimension determining unit 6c sets a predetermined “second dimension” that is larger than the first dimension. The size of the character image M is determined. The dimension determining unit 6c determines a predetermined “third dimension” larger than the second dimension as the dimension of the character image M when the volume specified by the volume specifying unit 3c is greater than or equal to the second volume threshold.
Note that the threshold for determining the size of the character image M is not limited to two, ie, the first volume threshold and the second volume threshold, and can be arbitrarily changed as appropriate.
In addition, if the size of the sound-related image (character image M) displayed in the display region R can be determined based on the volume, the size determination unit 6c is displayed in the display region R as the volume decreases, for example. The size of the character image M displayed in the display region R may be determined so that the character image M becomes relatively large.

The orientation determining unit 6d determines the orientation of the leader line L of the speech image when the character image M includes a speech image.
Specifically, the direction determination unit 6d determines the direction of the lead-out line L of the balloon image based on the sound source direction specified by the sound source direction specifying unit 3d as direction determination means.
Here, the orientation determining unit 6d determines the orientation of the extraction line L of the speech bubble so as to extend in substantially the same direction as the sound source direction specified by the sound source direction specifying unit 3d, for example. That is, for example, as illustrated in FIG. 3B, the orientation determination unit 6 d takes out the orientation “leftward” in which the leading end of the leader line L points to the left side of the display device 100 when the sound source direction is the left direction. The direction of L is determined. For example, as illustrated in FIG. 4B, the orientation determination unit 6 d takes out the orientation “right” where the leading end of the leader line L points to the right side of the display device 100 when the sound source direction is the right direction, The direction of L is determined.

The sound related image generation unit 6e generates a sound related image related to the sound collected by the sound collection unit 3a.
Specifically, the sound-related image generation unit 6e generates, for example, an image (character body image) based on the character data generated by the sound recognition unit 3e, and the size determination unit 6c determines the size of the generated image. The character image M is generated by adjusting based on the measured dimensions.

When the character image M includes a speech bubble image, the sound-related image generation unit 6e adds a speech bubble image having a predetermined shape to an image (character body image) based on the character data generated by the sound recognition unit 3e, for example. Thus, the character image M is generated.
That is, the sound-related image generation unit 6e acquires the balloon image data recorded in a predetermined recording unit (not shown) stored in the image processing unit 6 and the like, and generates a balloon image based on the acquired balloon image data. The size of the speech balloon image is adjusted so that an image (character body image) based on the character data generated by the sound recognition unit 3e is arranged in the frame. The sound-related image generation unit 6e adjusts the direction of the extraction line L of the balloon image based on the acquired balloon image data based on the direction determined by the direction determination unit 6d. Then, the sound-related image generation unit 6e generates a composite image by combining the character main body image and the balloon image whose size and orientation are adjusted, and the size determination unit 6c determines the dimensions of the generated composite image. The character image M is generated by adjusting based on the measured dimensions.
Note that the sound-related image generation unit 6e may change the balloon image data acquired from the predetermined recording unit based on the number of characters of the image (character body image) based on the character data generated by the sound recognition unit 3e. .

  Further, when the meaning content of the sound is recognized by the sound recognition unit 3e, the sound related image generation unit 6e may change the speech image data acquired from the predetermined recording unit based on the meaning content. Specifically, the sound-related image generation unit 6e, for example, as shown in FIG. 3 (b), when the sound collection unit 3a collects the words used when shouting “Yaho” or the like as sound. Acquires speech image data of a speech bubble with a jagged shape of the frame portion surrounding the character, and as shown in FIG. When the sound is collected by 3a, the speech image data of a speech bubble image having a substantially elliptical frame portion surrounding the character may be acquired. Thereby, the display apparatus 100 can change the shape of the speech bubble image included in the character image M displayed in the display area R of the display unit 8 in accordance with the meaning content of the sound collected by the sound collection unit 3a. it can.

The position determination unit 6f determines the display position in the display region R of the sound related image (character image M) related to the sound collected by the sound collection unit 3a.
Specifically, for example, when a face is detected by the face detection unit 6b, the position determination unit 6f detects the sound source direction specified by the sound source direction specification unit 3d and a predetermined image (display target) by the face detection unit 6b. The display position of the character image M in the display area R is determined based on the face position detected from the image.
Here, for example, when the number of faces detected by the face detection unit 6b is one, the position determination unit 6f specifies the face region of the face as a face region to which the character image M is added. The position determination unit 6f then coordinates the coordinates of a predetermined position (for example, the center coordinates of the predetermined position) in the region on the opposite side of the sound source direction specified by the sound source direction specifying unit 3d among the peripheral regions of the specified face region. ) Is determined as the display position of the character image M. That is, as shown in FIG. 4B, for example, when the sound source direction is the right direction, the position determination unit 6f uses the character image M as the coordinates of a predetermined position in the left area of the peripheral area of the specified face area. The display position is determined.

For example, when there are a plurality of faces detected by the face detection unit 6b, the position determination unit 6f identifies the identification information acquired by the individual identification unit 3f (identification of the individual identified by the individual identification unit 3f). Information) is acquired from the face identification information table T2 and whether there is a face that matches the acquired face identification information among the faces detected by the face detection unit 6b. Determine.
When it is determined that there is a face that matches the acquired face identification information among the faces detected by the face detection unit 6b, the position determination unit 6f uses the face detection unit 6b to display a predetermined image (display target). The display position of the character image M in the display region R is determined based on the face position detected using the individual face identification information identified by the individual identification unit 3f. That is, the position determination unit 6f specifies the face area of the face that matches the acquired face identification information as the face area to which the character image M is added. The position determination unit 6f then coordinates the coordinates of a predetermined position (for example, the center coordinates of the predetermined position) in the region on the opposite side of the sound source direction specified by the sound source direction specifying unit 3d among the peripheral regions of the specified face region. ) Is determined as the display position of the character image M.
On the other hand, when it is determined that there is no face that matches the acquired face identification information among the faces detected by the face detection unit 6b, the position determination unit 6f detects the face detected by the face detection unit 6b. A face area (for example, a face area having the largest dimension) that satisfies a predetermined condition in the area is specified as a face area to which the character image M is applied. The position determination unit 6f then coordinates the coordinates of a predetermined position (for example, the center coordinates of the predetermined position) in the region on the opposite side of the sound source direction specified by the sound source direction specifying unit 3d among the peripheral regions of the specified face region. ) Is determined as the display position of the character image M.

At this time, for example, the position determining unit 6f specifies the predetermined position in the region on the opposite side of the sound source direction as the predetermined position in the peripheral region of the specified face region based on the dimension determined by the dimension determining unit 6c. A position where the face area and the character image M do not overlap is selected. Further, when there is no position that does not overlap, the position determination unit 6 f selects a position where the degree of overlap between the specified face area and the character image M is the smallest.
Furthermore, the position determination unit 6f sets the leading edge of the extraction line L of the speech image to the center position of the specified face area as a predetermined position in the area on the opposite side of the sound source direction among the peripheral areas of the specified face area. You may select the position which faces, or you may select the position where the tip of the lead-out line L of the extracted image faces the center position of the specific area (for example, the mouth area) in the specified face area.

In addition, for example, when the face is not detected by the face detection unit 6b, the position determination unit 6f determines the display position of the character image M in the display region R based on the sound source direction specified by the sound source direction specification unit 3d. To do.
Here, the position determination unit 6f uses the coordinates of a predetermined position (for example, the center coordinates of the predetermined position) in the region on the sound source direction side specified by the sound source direction specifying unit 3d in the display region R of the display unit 8. The display position of the character image M is determined. That is, for example, as illustrated in FIG. 3B, the position determination unit 6 f uses the coordinates of a predetermined position in the left region of the display region R as the display position of the character image M when the sound source direction is the left direction. decide.

The face identification information table T2 stores, as first storage means, face identification information for identifying the face of at least one individual among humans and animals.
Specifically, the face identification information table T2 includes, for example, face identification information (for example, facial feature information) for identifying the face of an individual such as a human or an animal, and identification for identifying the individual. Information (for example, name) is stored in advance in association with each other.
Here, the facial feature information includes, for example, facial part information corresponding to eyes, nose, mouth and the like and facial images of humans and animals photographed at predetermined angles. Any information may be used as long as it is information for identifying an individual's face without being limited thereto.

The display control unit 7 performs control to display a predetermined image in the display region R of the display unit 8 based on the image data of the display target image acquired by the image acquisition unit 6a.
Further, the display control unit 7 determines a sound-related image (character image M) by the position determination unit 6f as a display control unit so as to overlap a predetermined image (image to be displayed) displayed in the display region R. Display at the displayed position. Specifically, the display control unit 7 superimposes and displays the character image M generated by the sound-related image generation unit 6e as an OSD image so as to overlap the display target image displayed in the display region R. At this time, for example, when the coordinates determined as the display position of the character image M by the position determination unit 6f are the center coordinates of the predetermined position, the display control unit 7 determines the center coordinates and the center coordinates of the character image M. The character image M is displayed so that.
The display control unit 7 may include an external connection I / F for connecting to display means other than the display unit 8. As a result, the display control unit 7 can display an image to be displayed, a sound-related image (character image M), or the like on an external display device connected to the display device 100.

  The display unit 8 displays a display target image in the display region R as a display unit, or displays the character image M superimposed on the display target image under the control of the display control unit 7. Examples of the display unit 8 include a liquid crystal display panel and an organic EL display panel, but are not limited thereto.

Next, sound-related image display processing by the display device 100 will be described with reference to FIG.
FIG. 2 is a flowchart illustrating an example of an operation related to the sound-related image display process.

As shown in FIG. 2, the display control unit 7 first displays a predetermined image in the display region R of the display unit 8 based on the image data of the display target image acquired by the image acquisition unit 6a (step S1).
Next, the sound collection unit 3a collects sound (sound) emitted from the sound source S such as a human or an animal, and the recording unit 3b records the sound collected by the sound collection unit 3a (step S2). ).

Next, the volume specifying unit 3c specifies the volume of the sound collected and recorded in step S2, and the sound source direction specifying unit 3d specifies the sound source direction of the sound collected and recorded in step S2. (Step S3).
Specifically, the sound source direction specifying unit 3d is based on, for example, the difference between the time when the sound from the sound source S reaches one input unit a1 of the sound collection unit 3a and the time when the sound reaches the other input unit a1. The position of the sound source S (individuals such as humans and animals) with respect to the display device 100 is specified, and the direction toward the position is set as the sound source direction.

Next, the sound recognizing unit 3e recognizes the voice collected and recorded in step S2, and generates character data for representing the voice by corresponding characters (step S4).
Next, the individual identification unit 3f, based on the audio information stored in the audio information table T1, is an individual (human, animal, etc.) that has collected the sound collected by the sound collection unit 3a and recorded by the recording unit 3b. ) Is identified (step S5).

Next, the dimension determination unit 6c determines the dimension of the sound-related image (character image M) displayed in the display region R based on the sound volume specified in step S3 (step S6).
Specifically, for example, as shown in FIG. 4B, when the volume of the collected and recorded sound is relatively low (for example, less than the first volume threshold), the dimension determining unit 6c A relatively small dimension (for example, the first dimension) is determined as the dimension of the character image M displayed in the display region R. On the other hand, for example, as shown in FIG. 5, when the volume of the collected and recorded sound is relatively large (for example, when the volume is greater than or equal to the first volume threshold and less than the second volume threshold), the dimension determining unit 6 c A relatively large dimension (for example, a second dimension larger than the first dimension) is determined as the dimension of the character image M displayed in the display region R.

Next, the orientation determination unit 6d determines the orientation of the lead-out line L of the speech balloon image included in the character image M based on the sound source direction identified in step S3 (step S7).
Specifically, for example, as illustrated in FIG. 3B, when the sound source direction is the left direction, the direction determination unit 6 d extracts the direction “left direction” in which the leading end of the lead line L points to the left side of the display device 100. This is determined as the direction of the drawing line L of the image. On the other hand, for example, as illustrated in FIG. 4B, when the sound source direction is the right direction, the direction determination unit 6 d extracts the image by extracting the direction “right direction” in which the leading end of the lead line L points to the right side of the display device 100. The direction of the line L is determined.

Next, the sound-related image generation unit 6e recognizes the recognition result in step S4 (specifically, the character data generated in step S4), the dimensions determined in step S6, and the leader line determined in step S7. Based on the direction of L and the like, a sound-related image related to the sound collected and recorded in step S2 is generated (step S8).
Specifically, the sound-related image generation unit 6e acquires the balloon image data recorded in predetermined recording means (not shown) stored in the image processing unit 6 and the like, and based on the acquired balloon image data. The size of the speech bubble image is adjusted so that the image (character body image) based on the character data generated in step S4 is arranged in the frame of the speech bubble image. The sound-related image generation unit 6e adjusts the direction of the extraction line L of the balloon image based on the acquired balloon image data based on the direction determined in step S7. Then, the sound-related image generation unit 6e generates a composite image by combining the character main body image and the balloon image whose size and orientation are adjusted, and determines the dimensions of the generated composite image in the dimensions determined in step S6. The character image M is generated by adjusting based on the above.

Next, the face detection unit 6b detects a face such as a human or an animal from the display target image acquired by the image acquisition unit 6a and displayed in the display region R of the display unit 8 (step S9).
Next, the position determination unit 6f determines whether or not a face is detected in step S9 (step S10).

If it is determined in step S10 that no face has been detected in step S9 (step S10; NO), that is, the display target image displayed in the display area R of the display unit 8 is, for example, FIG. In the case of a landscape image as shown in FIG. 4, the position determination unit 6f determines the display position of the sound-related image in the display region R based on the sound source direction specified in step S3 (step S11).
Specifically, the position determination unit 6f displays the sound-related image (character image M) using the coordinates of a predetermined position in the region on the sound source direction side specified in step S3 in the display region R of the display unit 8. Determine as position. For example, as illustrated in FIG. 3B, when the sound source direction is the left direction, the position determination unit 6 f determines the coordinates of a predetermined position in the left area of the display area R as the display position of the character image M. To do.

  Next, the display control unit 7 places the sound-related image generated in step S8 at the display position determined in step S11 so as to overlap the display target image displayed in the display region R of the display unit 8. It is displayed (step S16; see FIG. 3B).

  If it is determined in step S10 that a face has been detected in step S9 (step S10; YES), that is, the display target image displayed in the display area R of the display unit 8 is, for example, FIG. ) Or FIG. 6A, the position determination unit 6f determines whether or not the number of faces detected in step S9 is plural (step S12). ).

If it is determined in step S12 that the number of faces detected in step S9 is not plural (step S12; NO), that is, the display target image displayed in the display area R of the display unit 8 is, for example, FIG. 4 (a), the position determination unit 6f displays the sound related image display region R based on the sound source direction specified in step S3 and the face position detected in step S9. The display position at is determined (step S13).
Specifically, the position determination unit 6f identifies the face region of the face detected in step S9 in the display region R of the display unit 8 as the face region to which the sound-related image (character image M) is added, and performs the identification. Among the peripheral areas of the face area, the coordinates of a predetermined position in the area opposite to the sound source direction specified in step S3 are determined as the display position of the character image M. For example, as illustrated in FIG. 4B and FIG. 5, when the sound source direction is the right direction, the position determination unit 6 f calculates the coordinates of a predetermined position in the left area of the peripheral area of the face area in the display area R. The display position of the character image M is determined.

  Next, the display control unit 7 places the sound-related image generated in step S8 at the display position determined in step S13 so as to overlap the display target image displayed in the display area R of the display unit 8. It is displayed (step S16; see FIG. 4B and FIG. 5).

  If it is determined in step S12 that the number of faces detected in step S9 is plural (step S12; YES), that is, the display target image displayed in the display area R of the display unit 8 is displayed. For example, in the case of an image or the like as shown in FIG. 6A, the position determination unit 6f determines the face detected in step S9 based on the face identification information stored in the face identification information table T2. It is determined whether or not there is a face of the individual identified in step S5 (that is, the individual who has collected and recorded the sound) (step S14).

When it is determined in step S14 that the face detected in step S9 does not include the face of the individual identified in step S5 (step S13; NO), the position determination unit 6f is specified in step S3. Based on the sound source direction and the position of the face detected in step S9, the display position of the sound-related image in the display area R is determined (step S13).
Specifically, the position determination unit 6f gives a sound-related image (character image M) to a face region that satisfies a predetermined condition among the face regions of the face detected in step S9 in the display region R of the display unit 8. The coordinates of a predetermined position in the area opposite to the sound source direction specified in step S3 among the peripheral areas of the specified face area are determined as the display position of the character image M.
It should be noted that the voice information table T1 does not store voice information that matches the voice that has been collected and recorded, or the face identification information table T2 has the sound that has been collected and recorded. Even when face identification information is not stored, the position determination unit 6f determines in step S14 that the face detected in step S9 does not include the face of the individual identified in step S5.

  Next, the display control unit 7 places the sound-related image generated in step S8 at the display position determined in step S13 so as to overlap the display target image displayed in the display area R of the display unit 8. It is displayed (step S16).

In Step S14, when it is determined that the face of the individual identified in Step S5 is included in the face detected in Step S9 (Step S14; NO), the position determination unit 6f specifies in Step S3. The display position of the sound-related image in the display region R is determined based on the sound source direction and the face position of the individual detected in step S9 and identified in step S5 (step S15).
Specifically, the position determination unit 6f applies the sound-related image (character image M) to the face region of the individual face detected in step S9 in the display region R of the display unit 8 and identified in step S5. As a region, the coordinates of a predetermined position in the region on the opposite side of the sound source direction specified in step S3 among the peripheral regions of the specified face region are determined as the display position of the character image M. For example, as shown in FIG. 6B, when the sound source direction is the right direction, the position determination unit 6f displays the face area in the display area R (specifically, the individual that has collected and recorded the sound). The coordinates of a predetermined position in the left area of the peripheral area of the face area) are determined as the display position of the character image M.

  Next, the display control unit 7 places the sound-related image generated in step S8 at the display position determined in step S15 so as to overlap the display target image displayed in the display region R of the display unit 8. It is displayed (step S16; refer to FIG. 6B).

As described above, according to the display device 100 of the present embodiment, the sound-related image related to the sound collected by the sound collecting unit 3a is displayed. Therefore, when a desired image is designated, an operation input by the user is performed. The cumbersome operation of the unit 2 is not required, and the desired image is displayed on the predetermined image displayed in the display region R by simply collecting the sound corresponding to the desired image on the sound collecting unit 3a. Can do.
Furthermore, according to the display device 100, the display position of the sound-related image related to the sound in the display region R is determined based on the sound source direction of the sound collected by the sound collecting unit 3a. When a desired position is designated as the display position, the user does not need to perform troublesome operations on the operation input unit 2, and the sound collecting unit 3a collects sound from the direction corresponding to the desired position. Can be determined as the display position of the sound-related image. That is, the user can change the position of the sound source S (for example, the user himself / herself) relative to the display device 100 according to the position where the user wants to display the sound-related image related to the sound. The user-desired image can be superimposed and displayed at the position desired by the user.
Therefore, the display device 100 can easily display the sound-related image desired by the user at the position desired by the user of the predetermined image displayed in the display area R of the display unit 8.

  Moreover, according to the display apparatus 100 of this embodiment, the character image M which represented the sound collected by the sound collection part 3a with the corresponding character as a sound related image can be displayed. Therefore, a troublesome operation such as inputting a character by an operation of the operation input unit 2 or the like by the user is not necessary, and the sound collecting unit 3a collects a sound corresponding to the user-desired character image M, and the user-desired operation is performed. The character image M can be displayed over the predetermined image displayed in the display region R.

  Further, according to the display device 100 of the present embodiment, detection is performed from the sound source direction of the sound collected by the sound collection unit 3a and a predetermined image (image to be displayed) displayed in the display region R of the display unit 8. The display position of the character image M in the display region R is determined based on the face position. Therefore, by simply collecting the sound corresponding to the user-desired character image M in the sound collecting unit 3a from the direction corresponding to the user-desired position, the user-desired position and the predetermined image displayed in the display area R The display position of the character image M desired by the user can be determined based on the face portion.

  Further, according to the display device 100 of the present embodiment, detection is performed from the sound source direction of the sound collected by the sound collection unit 3a and a predetermined image (image to be displayed) displayed in the display region R of the display unit 8. The display position of the character image M in the display area R is determined based on the position of the face of the individual that emitted the voice among the faces that have been displayed. Therefore, by simply collecting the sound corresponding to the user-desired character image M emitted by the individual from the direction corresponding to the user-desired position on the sound collection unit 3a, the sound is displayed on the user-desired position and the display region R. The display position of the character image M desired by the user can be determined based on the face portion of the individual in the predetermined image.

  Further, according to the display device 100 of the present embodiment, the direction of the extraction line L of the speech balloon included in the character image M is determined based on the sound source direction of the sound collected by the sound collection unit 3a. Therefore, even if there is no troublesome operation such as inputting the direction of the extraction line L of the speech line by the operation of the operation input unit 2 or the like by the user, an appropriate direction can be determined as the direction of the extraction line L of the speech line image. .

  Further, according to the display device 100 of the present embodiment, the dimensions of the sound-related image (character image M) displayed in the display area R of the display unit 8 based on the volume of the sound collected by the sound collection unit 3a. Can be determined. Therefore, a troublesome operation such as inputting a user-desired dimension by an operation of the operation input unit 2 or the like by the user is unnecessary, and the sound collecting unit 3a collects a sound having a volume corresponding to the user-desired dimension. A sound-related image (character image M) having a user-desired size can be superimposed on a predetermined image displayed in the region R and displayed.

  The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.

  For example, the display device 100 may store a table in which the voice information table T1 and the face identification information table T2 are integrated instead of the voice information table T1 and the face identification information table T2. That is, the display device 100 includes audio information for identifying the sound of an individual such as a human or an animal, face identification information for identifying the face of the individual, and identification information for identifying the individual. A table stored in association with each other may be stored. In this case, the table constitutes a first storage unit and a second storage unit.

  The image processing unit 6 also determines the direction of the sound-related image itself related to the sound based on the sound source direction specified by the sound source direction specifying unit 3d, that is, the sound source direction of the sound collected by the sound collecting unit 3a. May be determined. Specifically, for example, when the sound source direction is the right direction or the left direction, the image processing unit 6 directs the character image M (sound-related image) itself so that the characters of the character image M are horizontally written side by side. When the sound source direction is upward or downward, the direction of the character image M (sound-related image) itself may be determined so that the characters of the character image M are vertically written.

  Further, when the display device 100 includes an imaging unit, the individual identification unit 3f generates a sound collected by the sound collection unit 3a based on an image captured by the imaging unit instead of the sound information. Individuals such as humans and animals may be identified. That is, whether or not to use audio information for identifying each individual of at least one of humans and animals can be arbitrarily changed as appropriate.

In the above embodiment, the display control unit 7 displays the character image M so as to be superimposed on the display target image that has been displayed in the display area R. After determining the display position of the character image M in the display target image, both the display target image and the character image M may be displayed in the display region R.
In addition, the display control unit 7 displays the character image M as an OSD image so as to overlap the display target image. For example, the image processing unit 6 displays the display target image and the character image M. The combined image may be generated, and the display control unit 7 may display the combined image in the display region R.

The sound source S does not have to be an individual such as a human or an animal that emits sound, but may be any object that emits sound.
In addition, when the sound source S is a thing that emits sound other than sound such as a car, the image processing unit 6 identifies information for identifying the sound emitted by the sound source S, the outer shape of the sound source S, and the shape of the characteristic portion of the sound source S. A sound source S (for example, a car) region that emits sound collected by the sound collection unit 3a from an image to be displayed (for example, an image with a car or the like as a subject) based on information for identifying the And the display position of the sound-related image may be determined based on the detected position of the sound source S region.
Further, when the display device 100 includes an imaging unit, a display target is determined based on information for identifying the outer shape of the sound source S, the shape of the characteristic portion of the sound source S, and the like, an image captured by the imaging unit, and the like. A sound source S (for example, a car or the like) region that emits the sound collected by the sound collecting unit 3a is detected from the image (for example, an image having a car or the like as a subject), and based on the position of the detected sound source S region Thus, the display position of the sound-related image may be determined.

In the above embodiment, the sound-related image is the character image M representing the sound collected by the sound collection unit 3a with the corresponding characters. However, the sound-related image is collected by the sound collection unit 3a, for example. It may be a character image M in which sound (for example, noise, etc.) other than the generated voice is represented by corresponding characters. Specifically, for example, when the sound of a car horn called “Pupu” is collected by the sound collection unit 3a, the display device 100 displays the sound as a sound-related image with a corresponding character (“Pupu”). The character image M thus displayed may be displayed.
The sound-related image is not limited to the character image M, and can be arbitrarily changed as long as it is an image related to the sound collected by the sound collection unit 3a. Specifically, the display device 100 may display a sun image as a sound-related image, for example, when the word “taiyo” is collected as a sound in the sound collecting unit 3a. When the sound collection unit 3a collects the word “reminiscent of the morning” as sound, an image reminiscent of the morning such as the sun image may be displayed as the sound-related image. In addition, for example, when the sound of a car horn called “Pupu” is collected by the sound collection unit 3a, the display device 100 can be associated with the sound “Pupu” such as a car image as a sound-related image. May be displayed.

The display position of the sound-related image in the display region R is not limited to the above embodiment, and can be arbitrarily changed as long as it can be determined based on at least the sound source direction of the specified sound.
Further, the direction of the lead-out line L of the balloon image is not limited to the above embodiment, and can be arbitrarily changed as long as it can be determined based on the specified sound source direction. Further, the direction of the leader line L of the speech bubble may be determined based on information other than the sound source direction.
In addition, the size of the sound-related image displayed in the display region R is not limited to the above embodiment, and can be arbitrarily changed as long as it can be determined based on the specified volume. Further, the size of the sound-related image displayed in the display area R may be determined based on information other than the volume.

In the above embodiment, the position determination unit 6f is detected from a predetermined image (image to be displayed) by the face detection unit 6b when the number of faces detected by the face detection unit 6b is plural, The display position of the character image M in the display area R is determined based on the position of the face identified using the individual face identification information identified by the individual identification unit 3f, but the present invention is not limited to this. . The position determination unit 6f is, for example, an individual detected from a predetermined image (image to be displayed) by the face detection unit 6b and identified by the individual identification unit 3f regardless of the number of faces detected by the face detection unit 6b. The display position of the character image M in the display region R may be determined based on the position of the face identified using the face identification information. In this case, when the number of faces detected by the face detection unit 6b is one and the face is not the face of the individual that has collected and recorded the sound, the position determination unit 6f, for example, detects the face As in the case where no face is detected by the unit 6b, the display position of the character image M in the display region R is determined based only on the sound source direction specified by the sound source direction specifying unit 3d.
In addition, the position determination unit 6f uses the face identification image information to identify the face of the individual that has collected and recorded the sound. For example, the face detection unit 6b detects the face. At the same time, it is also possible to identify the face of an individual who has produced a sound that has been collected and recorded using face identification image information.

In addition, in the above-described embodiment, the functions of the acquisition unit, the direction specification unit, the position determination unit, and the display control unit are controlled by the central control unit 1, and the image acquisition unit 6a and the sound source direction specification unit 3d. The position determining unit 6f and the display control unit 7 are configured to be driven. However, the present invention is not limited to this, and is realized by executing a predetermined program or the like by the CPU of the central control unit 1. It is good also as a structure.
That is, a program including an acquisition process routine, a direction specifying process routine, a position determination process routine, and a display control process routine is stored in a program memory (not shown) that stores the program. Then, the CPU of the central control unit 1 may function as an acquisition unit that acquires a predetermined image displayed in the display area of the display unit by an acquisition process routine. Further, the CPU of the central control unit 1 may function as direction specifying means for specifying the sound source direction of the sound collected by the sound collecting means by the direction specifying processing routine. Further, the CPU of the central control unit 1 by the position determination processing routine determines the display position in the display area of the sound related image related to the sound based on the sound source direction of the sound specified by the direction specifying means. You may make it function as. Also, the display control processing routine causes the CPU of the central control unit 1 to function as display control means for displaying the sound-related image at the display position determined by the position determination means so as to overlap a predetermined image displayed in the display area. You may make it let it.

  Similarly, other functions such as sound recognition means, detection means, identification means, direction determination means, volume specification means, and dimension determination means are also realized by executing predetermined programs and the like by the CPU of the central control unit 1. It is good also as a structure made.

  Further, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory, a portable image recording medium such as a CD-ROM, etc. is applied in addition to a ROM and a hard disk. It is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

Although the embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, and includes the scope of the invention described in the claims and an equivalent scope thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
In a display device comprising display means,
Obtaining means for obtaining an image displayed in a display area of the display means;
Sound collecting means for collecting sounds emitted from the sound source;
Direction specifying means for specifying the sound source direction of the sound collected by the sound collecting means with reference to the position of the apparatus body;
Position determining means for determining a display position in the display area of the sound-related image related to the sound based on the sound source direction of the sound specified by the direction specifying means;
Display control means for displaying the sound-related image at the display position determined by the position determination means so as to overlap the image displayed in the display area;
A display device comprising:
<Claim 2>
It further comprises detection means for detecting a face from the image acquired by the acquisition means,
The said position determination means further determines the display position in the said display area of the said sound related image based on the position of the face detected from the said image by the said detection means. Display device.
<Claim 3>
First storage means for storing face identification information for identifying the face of each individual of at least one of human and animal;
Second storage means for storing voice information used for identification of each individual of at least one of human and animal;
Identification means for identifying an individual that has emitted the sound collected by the sound collection means based on the sound information stored in the second storage means;
The position determination unit is configured to display the sound-related image based on a position of a face detected from the image by the detection unit and identified using the face identification information of the individual identified by the identification unit. The display device according to claim 2, wherein a display position in the region is determined.
<Claim 4>
A sound recognition means for recognizing the sound collected by the sound collection means;
The said display control means displays the character image which represented the sound recognized by the said sound recognition means by the character corresponding as the said sound related image, The Claim 1 characterized by the above-mentioned. Display device.
<Claim 5>
The character image includes a speech bubble image having a lead line connected to a predetermined position of a frame portion surrounding the character,
The display device according to claim 4, further comprising a direction determining unit that determines a direction of a leader line of the balloon image based on a sound source direction specified by the direction specifying unit.
<Claim 6>
Volume specifying means for specifying the volume of the sound collected by the sound collecting means;
6. The method according to claim 1, further comprising: a dimension determining unit that determines a dimension of the sound-related image displayed in the display area based on the volume specified by the volume specifying unit. The display device according to one item.
<Claim 7>
A display method using a display device comprising display means and sound collection means for collecting sound emitted from a sound source,
Processing for displaying an image in a display area of the display means;
A process of identifying the sound source direction of the sound collected by the sound collecting means with reference to the position of the display device body;
A process of determining a display position in the display area of a sound-related image related to the sound based on the sound source direction of the specified sound;
Processing to display the sound-related image at the determined display position so as to overlap the image displayed in the display area;
A display method characterized by:
<Claim 8>
A computer of a display device comprising display means and sound collection means for collecting sounds emitted from a sound source,
Obtaining means for obtaining an image displayed in a display area of the display means on the basis of the position of the display device body;
Direction specifying means for specifying the sound source direction of the sound collected by the sound collecting means;
Position determining means for determining a display position in the display area of the sound-related image related to the sound based on the sound source direction of the sound specified by the direction specifying means;
Display control means for displaying the sound-related image at the display position determined by the position determination means so as to overlap the image displayed in the display area;
A program characterized by functioning as

DESCRIPTION OF SYMBOLS 1 Central control part 3a Sound collection part 3c Sound volume specific | specification part 3d Sound source direction specific | specification part 3e Sound recognition part 3f Individual identification part 7a Image acquisition part 7b Face detection part 7c Dimension determination part 7d Direction determination part 7f Position determination part 8 Display control part 9 Display unit 100 Display device L Lead line M Character image R Display area S Sound source T1 Audio information table T2 Face identification information table

Claims (8)

In a display device comprising display means,
Obtaining means for obtaining an image displayed in a display area of the display means;
Sound collecting means for collecting sounds emitted from the sound source;
Direction specifying means for specifying the sound source direction of the sound collected by the sound collecting means with reference to the position of the apparatus body;
Position determining means for determining a display position in the display area of the sound-related image related to the sound based on the sound source direction of the sound specified by the direction specifying means;
Display control means for displaying the sound-related image at the display position determined by the position determination means so as to overlap the image displayed in the display area;
A display device comprising: It further comprises detection means for detecting a face from the image acquired by the acquisition means,
The said position determination means further determines the display position in the said display area of the said sound related image based on the position of the face detected from the said image by the said detection means. Display device. First storage means for storing face identification information for identifying the face of each individual of at least one of human and animal;
Second storage means for storing voice information used for identification of each individual of at least one of human and animal;
Identification means for identifying an individual that has emitted the sound collected by the sound collection means based on the sound information stored in the second storage means;
The position determination unit is configured to display the sound-related image based on a position of a face detected from the image by the detection unit and identified using the face identification information of the individual identified by the identification unit. The display device according to claim 2, wherein a display position in the region is determined. A sound recognition means for recognizing the sound collected by the sound collection means;
The said display control means displays the character image which represented the sound recognized by the said sound recognition means by the character corresponding as the said sound related image, The Claim 1 characterized by the above-mentioned. Display device. The character image includes a speech bubble image having a lead line connected to a predetermined position of a frame portion surrounding the character,
The display device according to claim 4, further comprising a direction determining unit that determines a direction of a leader line of the balloon image based on a sound source direction specified by the direction specifying unit. Volume specifying means for specifying the volume of the sound collected by the sound collecting means;
6. The method according to claim 1, further comprising: a dimension determining unit that determines a dimension of the sound-related image displayed in the display area based on the volume specified by the volume specifying unit. The display device according to one item. A display method using a display device comprising display means and sound collection means for collecting sound emitted from a sound source,
Processing for displaying an image in a display area of the display means;
A process of identifying the sound source direction of the sound collected by the sound collecting means with reference to the position of the display device body;
A process of determining a display position in the display area of a sound-related image related to the sound based on the sound source direction of the specified sound;
Processing to display the sound-related image at the determined display position so as to overlap the image displayed in the display area;
A display method characterized by: A computer of a display device comprising display means and sound collection means for collecting sounds emitted from a sound source,
Obtaining means for obtaining an image displayed in a display area of the display means on the basis of the position of the display device body;
Direction specifying means for specifying the sound source direction of the sound collected by the sound collecting means;
Position determining means for determining a display position in the display area of the sound-related image related to the sound based on the sound source direction of the sound specified by the direction specifying means;
Display control means for displaying the sound-related image at the display position determined by the position determination means so as to overlap the image displayed in the display area;
A program characterized by functioning as

Images