JP2010134507A - Reproduction device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To visually confirm a person from which only voice is heard without appearance of a face in an image. <P>SOLUTION: A database 16 is registered with face image data and feature values thereof, and voice data and feature values thereof about a plurality of persons. By a person voice recognition unit 32 and a person voice verification unit 34, a feature value of person voice included in a reproduction voice signal is verified with the database to search for a vocalizing person. By a face image recognition unit 20 and a face image verification unit 22, a feature value of a face image included in a reproduction image signal is verified with the database 16 to search for a person inside a screen. An additional image display decision unit 40 excludes the searched person from the searched vocalizing persons, and specifies a speaker outside the screen. An additional image generation unit 24 reads an additional image of the specified speaker from the database 16, and supplies it to a display image generation unit 26. The display image generation unit 26 composes the additional image to the reproduction image signal. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention has a function of displaying on a reproduced image what kind of person the person who can hear only voice without appearing in the image when reproducing the recorded image signal and audio signal. The present invention relates to a playback apparatus having

2. Description of the Related Art Conventionally, there is known a system that additionally displays information about a person displayed on a screen when a recorded video or a video transmitted from a remote place is reproduced and displayed. For example, in a video conference system, Patent Document 1 discloses a technique for generating a video signal in which an image of a speaker is added with any information that the viewer is interested in regarding the speaker. Information on a person who can be photographed is registered in advance, and the person is identified and added at the time of photographing. This method is useful when it is desired to obtain information about a person shown in the video.
Japanese Patent Laid-Open No. 2-67889

  In a consumer video camera, a person to be photographed is often a known person such as a viewer's family or friends. In such a case, the necessity of additionally displaying subject person information is low.

  However, in many cases, only the voice is recorded for a person outside the field of view. In such a case, the only way to identify the person is to rely on voice storage. However, it is difficult to identify a person unless he / she is used to listening or has a feature in speaking. In addition, the work of tracing the memory for estimating the utterer reduces the degree of concentration with respect to video viewing and hinders pure enjoyment of the video.

  An object of the present invention is to provide a playback apparatus that eliminates such inconveniences.

  In order to achieve the above object, a playback apparatus according to the present invention plays back image data read from a recording medium, outputs a playback image signal, and plays back audio data read from the recording medium. Voice processing means for outputting voice signals, a database recording voice data and face image data of a plurality of people, human voice recognition means for discriminating and recognizing voices from the reproduced voice signals, and recognition by the voice recognition means A human voice collating means for collating and identifying the recorded human voice and voice data registered in the database, a face image recognizing means for recognizing and recognizing a human face from the reproduced image signal, and the face image recognizing means. Face image matching means for comparing and identifying the face of the recognized person and face image data registered in the database, and the face image from the person identified by the voice matching means in the same scene The additional image display determining means for determining a person excluding the person identified by the combining means as an additional image display target, and information indicating the target person determined by the additional image display determining means from the database And an additional image generating means for generating an additional image to be combined with the reproduced image signal, and a display image generating means for combining the additional image with the reproduced image signal.

  According to the present invention, it is possible to visually recognize the utterer by displaying the information related to the utterer of the voice that is not projected on the video in the reproduced video. It is no longer possible to reduce the degree of concentration with respect to video viewing by tracing the memory for estimating the speaker.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic block diagram of an embodiment of a playback apparatus according to the present invention. A video signal with sound is recorded on the recording medium 12 of the reproducing apparatus 10. The recording medium 12 includes an optical disk typified by a DVD (Digital Versatile Disk), a magnetic tape, a hard disk, or a memory card incorporating a flash memory. The recording medium driving device 14 is a device that drives the recording medium 12 to read / write signals from / to the recording medium 12.

  In the database 16, personal information data of a plurality of persons can be registered and are actually registered. Each personal information data includes face image data used for the face image matching function and face image feature value data indicating the feature value, voice data used for the human voice matching function and voice feature value data indicating the feature value, and Includes various other data.

  The image processing unit 18 decodes the compressed image data read from the recording medium 12 and performs various processes to generate reproduced image data. The face image recognition unit 20 determines the presence or absence of a region corresponding to a human face from the image data processed by the image processing unit 18. The face image matching unit 22 compares the area determined to have a face image by the face image recognition unit 20 with the face image data registered in the database 16. Thereby, it is possible to specify whether or not the person included in the reproduced image is registered in the database 16 and who is registered.

  The additional image generation unit 24 generates additional image data to be superimposed on the reproduced image signal reproduced from the recording medium 12. The display image generation unit 26 synthesizes the additional image data output from the additional image generation unit 24 with the reproduced image data output from the image processing unit 18. The display image generation unit 26 generates a composite image in which the additional image is superimposed on the reproduction image. However, the output image of the display image generation unit 26 may be composed of only the reproduced image or only the additional image in addition to the synthesized image.

  The display unit 28 displays the image data generated by the display image generation unit 26 as an image so that the user can visually recognize the image data. The display unit 28 is incorporated in the playback device, for example. The display unit 28 includes a display device such as a liquid crystal display (LCD) or an organic EL display and a driving circuit thereof.

  The audio processing unit 30 decodes the compressed audio data read from the recording medium 12 and performs various processes. The human voice recognition unit 32 determines whether or not there is a voice corresponding to a human voice from the reproduced voice signal obtained by the voice processing unit 30. The human voice collation unit 34 compares the portion determined as the human voice by the human voice recognition unit 32 with the voice data registered in the database 16. When the human voice recognition unit 32 determines that the voice is a human voice, the volume detector 36 detects the volume level. The audio output device 38 is a device that acoustically outputs the reproduced audio generated by the audio processing unit 30. The audio output device 38 includes, for example, a speaker incorporated in the playback device 10 and its drive circuit.

  The additional image display determination unit 40 determines the content of the additional image to be generated by the additional image generation unit 24 based on the collation result of the face image collation unit 22 and the collation result of the human voice collation unit 34.

  The external output unit 42 is a unit that outputs the image signal generated by the display image generation unit 26 and the reproduced audio signal generated by the audio processing unit 30 to the outside. For example, the external output drive circuit and the connection terminal or transmission antenna It consists of.

  The CPU 44 is a central processing unit that not only controls the additional image generation unit 24 and the display image generation unit 26 but also controls the entire playback device 10 so that image processing and audio processing are synchronized.

  The input device 46 is a device for a user to input an operation mode, an operation condition, and the like to the playback device 10, and includes various switches or buttons, operable elements displayed on a menu screen, and the like.

  In order to understand the characteristic operation of the present embodiment, the following situation is assumed. That is, as a result of directing the camera to Mr. A in the situation where Mr. A and Mr. B are having a conversation, only Mr. A falls within the shooting angle of view as shown in FIG. Suppose only was recorded.

  FIG. 3 shows a display example of a reproduction screen of a video recorded in such a situation by a conventional method. A reproduced image including the person A is displayed on the screen, and reproduced sound is output from a speaker on the side of the screen. The symbol group attached to the right corner of the screen and the speaker is an illustration of Mr. B's voice.

  On the other hand, FIG. 4 shows an example of a playback screen according to the present embodiment. The reproduced image is displayed on the screen of the display unit 28, and the reproduced sound is output from the speaker (sound output device 38) on the side of the screen. The symbol group attached to the right corner of the screen and the speaker is an illustration of Mr. B's voice. Unlike FIG. 3, an additional image showing the person B is superimposed on the screen. With the additional image showing the person B, the viewer can recognize or infer that the speaker is Mr. B.

  An operation of identifying a speaker who is not within the shooting angle of view and synthesizing an additional image indicating the speaker according to the present embodiment will be described. FIG. 5 shows a flowchart of the operation. Here, it is assumed that the recording medium 12 is a DVD.

  First, personal information data of a necessary person is registered in advance in the database 16 (S1). As described above, each person's personal information data includes face image data and face image feature amount data indicating the feature amount, voice data and voice feature amount data indicating the feature amount, and various other attributes. Consists of data. The attribute data includes, for example, name, nickname, sex ", age, date of birth, and portrait.

  The audio processing unit 30 expands the compressed audio data read from the recording medium 12 and supplies the reproduced audio signal to the human voice recognition unit 32, the audio output device 38, and the external output unit 42. The human voice recognition unit 32 extracts and analyzes a human voice from the reproduced voice signal, and determines whether or not a human voice is included (S2). If a human voice is included, the human voice recognition unit 32 supplies the extracted voice data to the human voice verification unit 34.

  The human voice collation unit 34 extracts the feature amount from the voice data from the human voice recognition unit 32, collates it with the voice feature amount data in the database 16, and selects a person having a feature amount having a certain correlation or higher. Search (S3). When there is voice feature value data having a degree of correlation equal to or greater than a certain value, the human voice matching unit 34 notifies the additional image display determination unit 40 of a person having the voice feature value data as a candidate for additional image display ( S4).

  On the other hand, the image processing unit 18 decompresses the compressed image data reproduced from the recording medium 12 to generate reproduced image data. The face image recognition unit 20 extracts a human face image from the reproduced image, and supplies the extracted face image data to the face image matching unit 22 (S5). The face image collating unit 22 extracts the feature amount from the face image data from the face image recognizing unit 20 and collates it with the face image feature amount data in the database 16 to obtain a face image feature amount having a high correlation of a certain level or more. The person who has it is searched (S6). When face image feature value data having a degree of correlation equal to or greater than a certain value exists, the face image matching unit 22 sets a person having the face image feature value data as an additional image display exclusion candidate to the additional image display determination unit 40. Notification is made (S7).

  The additional image display determination unit 40 excludes the exclusion candidate extracted from the reproduction image (exclusion candidate from the face image verification unit 22) from the candidate extracted from the reproduction sound (notification information from the human voice verification unit 34). (S8). Thereby, the person (S7) in the screen can be excluded from the person candidates (S4) having a high possibility of being a speaker. That is, it is possible to specify a person who is not in the shooting screen but has recorded voice. When there are a plurality of persons, the person with the highest correlation degree of the human voice feature is determined as the speaker. The additional image display determination unit 40 notifies the additional image generation unit 24 of the finally determined person.

  The additional image generation unit 24 reads the face image data of the person notified from the additional image display determination unit 40 from the database 16, and generates an additional image of a predetermined size including the face image (S9). The additional image generation unit 24 supplies the generated additional image to the display image generation unit 26.

  The display image generation unit 26 superimposes the additional image from the additional image generation unit 24 on the reproduced image data from the image processing unit 18 to generate composite image data (S10). The composite image data is applied to the display unit 28 and displayed as shown in FIG. Of course, the external output unit 42 may output to the outside. The display image looks like a pictograph superimposed on the image recorded on the recording medium 12. For this reason, the additional image that is superimposed on the reproduced image is hereinafter also expressed as a pictograph.

  The database 16 will be described. The database 16 may be stored in the recording medium 12 or built in the playback device 10. When the playback device 10 is compatible with a network, the database 16 may be prepared on a server connected via a network. When the recording medium 12 is detachable from the playback device 10, either the method of loading the database of the recording medium 12 into the storage area of the playback device 10 for use or the method of directly referring to the database on the recording medium 12 is used. May be. In the former case, a removable recording medium in which only the database is recorded may be prepared.

  The personal information data of each person registered in the database 16 necessarily includes face image data for use in face image matching and pictogram display, and voice data for use in voice matching. In addition, data such as name, nickname, gender, age, date of birth, and caricature are stored as additional attribute information. These additional attribute information may be used for pictogram display at the same time as or in place of the face image data.

  As the number of people registered in the database 16 increases, there is a classification of a person who wants to display as a pictograph and a person who does not want to display. In order to correspond to this classification, it is preferable to provide an item for setting whether or not to display a pictograph for each personal information data. Which is set by default when newly registering newly registered personal information data may be determined in accordance with the usage status. The CPU 44 causes the display image generation unit 26 to synthesize the additional image with the reproduced image signal when the person who should display the additional image is set in the database 16 so as to permit the combined display.

  The database 16 is preferably versatile enough to read and use data created by other devices.

  The pictograph display of the present embodiment will be described. Displaying information about a person who can only hear a voice as a pictograph is useful because it can add visibility. And the effect can be further increased by devising the display method.

  If the user can set the validity / invalidity of the pictograph display function, that is, whether or not it is necessary, it is possible to respond to a request to view only the original recorded image. This is effective when the person who is not shown in the image is not cared about, or when a known voice is heard. For example, when the pictograph display function is disabled by the user, the CPU 44 does not cause the additional image generation unit 24 to generate an additional image even if a voice of a speaker outside the screen is recorded.

  The information displayed as a pictograph is one or more items among the items of the database 16. If an item to be displayed can be selected, it can be linked to information that the user wants to know. For example, if “name” is displayed at the same time as the face image, it becomes easier to identify a person even when it is difficult to determine who the face image alone is. Of course, the same effect can be obtained by displaying only the “name”.

  If the data items to be displayed for all the pictogram display target persons are unified, there is a possibility that the corresponding item is not registered in some persons. In order to cope with such a situation, it is preferable to give priority to each item of the pictograph display data and display the next item when the upper item is not registered. For example, it is assumed that “nickname” is set as the first priority and “name” is set as the second priority. In this case, “nickname” is displayed for a person for whom the item “nickname” is registered, and “name” is displayed for a person for whom the item “nickname” is not registered. That is, when there are a plurality of items that can be used for generating an additional image among the items of each person in the database 16, a means is provided for the user to set priorities for all or some of the plurality of items.

  The task of setting such priorities becomes more complicated as the number of items of personal information data of each person that can be registered in the database 16 increases. For this, the user can set up to a finite number of higher ranks, and the playback apparatus 10 may automatically allocate items of lower ranks. This reduces the burden on the user. Even if the user wants to add any data to the items in the personal information data of the database 16, the face image data, which is a registration-required item, is assigned to any priority order, so that it is displayed on the pictograph. Data is secured.

  It is desirable that the pictograph on the screen does not block as much as possible the portion of high interest in the reproduced image. In general, the degree of attention of an image tends to be higher in the central part than in the peripheral part. Therefore, the display location of the pictograph is basically arranged in the peripheral part of the reproduced image. However, in a scene where the subject is moving, the attention degree of the peripheral portion may increase as the subject moves. At this time, it is not desirable that the place where the degree of attention is high and the place where the pictograph is displayed overlap. If the user can set the display location of the pictograph, such a situation can be avoided.

  It is convenient if the user can also set the size of the pictograph. For example, there are formats that allow you to select the size from a specified level expressed as “Large”, “Medium”, “Small” or the number of dots, or a format that allows you to select a size using a chart with similar expressions. It is done. This is because the number of pixels of the display screen is large, such as when viewing on a relatively small display screen provided in the playback apparatus 10 and when viewing on a relatively large screen such as a television via an external connection function. Useful when different. This is because there is a non-negligible difference in the size of the pictograph required to satisfy the visibility.

  Further, regarding the shape, if options such as a rectangle and a circle are provided, the user's preference can be utilized. However, it is necessary to limit the data that can be displayed according to the set shape. For example, data that can be expressed while satisfying visibility within a small external dimension is limited. The actual restriction may be provided in a manner suitable for each playback device. When such a shape that restricts is set as the shape of the pictograph, the highest priority is displayed among the displayable data candidates. For both the size and the shape, it is preferable that the size and shape of the actual pictograph change simultaneously with the setting change operation.

  When the reproduced voice signal and the voice data in the database 16 are collated with the respective feature amounts and the correct person cannot be selected, incorrect data is used for the pictograph. When the user notices an error by looking at the displayed pictograph, the present embodiment can correct the content correctly. FIG. 6 shows a flowchart of the correcting operation.

  The user looks at the reproduced display image and confirms whether the voice to be heard and the personal information displayed on the pictograph are correct or incorrect (S21). Here, when an error is found and corrected, a correction function is used (S22). While viewing the screen of the display unit 28, the input device 46 such as a switch or a touch panel is used to correct so that correct information is displayed as a histogram (S23 to S31).

  This will be specifically described. When the user selects correction (S22), the correction mode is entered and a screen for selecting a pictograph on the screen is displayed (S23). When selecting a pictograph, a pictograph indicating “no corresponding person” is displayed for a voice that has been excluded from the display target because there is no corresponding person in the database 16 so that it can be selected. It is desirable to do. As a result, it is possible to correct an error in which the human voice matching unit 34 could not identify the reproduced voice signal and the voice data in the database 16.

  The pictograph selected on the pictograph selection screen changes its shape, color, border, etc. and displays some highlights. The user selects a pictograph to be corrected by the input device 46 (S24). Immediately after the selection, a dialog for confirming the intention of correction is displayed (S25). Subsequently, data to be used after correction is searched and selected (S26).

  At the same time, the target range of the pictograph to be corrected is selected (S27). Providing such a selection step can reduce the possibility of erroneous selection. For example, the first option is limited to “only selected”. The second option is limited to “all those where the same person as the selected pictograph is recognized”. The third option is limited to “one in which the same person is recognized as the selected pictograph, and has a lower correlation rate of voice verification than the selected scene”. As a method for expressing these options, a method suitable for each reproducing apparatus is selected.

  Whether or not correction can be executed is confirmed (S28). In accordance with the user's decision to execute correction, actual pictograph display data replacement processing is performed (S29). At this time, if the range of simultaneous correction with the target pictograph is reflected in the database 16, the accuracy of human voice verification after the next time is improved.

  If there are other pictographs to be corrected (S30), correction is performed by the same procedure. If there is no other pictograph to be corrected (S30), the correction mode is exited (S231), and a series of correction operations is completed.

  With the above configuration and operation, when an image signal and an audio signal are reproduced, information related to a person who can hear only a voice without being reflected in the image can be visually recognized.

  A second embodiment of the present invention will be described. FIG. 7 shows a schematic block diagram of the second embodiment. The same components as those in the embodiment shown in FIG. The same components as those in FIG. 1 are denoted by the same reference numerals.

  In the second embodiment, a function for recording images and sounds is added to the personal information database. That is, a camera unit 50 and a microphone 52 are added to the playback device 10a. In addition to the functions of the CPU 44, the image processing unit 18, and the sound processing unit 30, the CPU 44a, the image processing unit 18a, and the sound processing unit 30a have functions of recording images and sounds in the database 16a, respectively.

  The changed part will be described in detail. The camera unit 50 includes a lens and an image sensor, and can be used to capture face image data to be registered in the database 16a. That is, the image processing unit 18a performs color balance and gamma correction on the image signal photographed by the camera unit 50, adjusts the size and the like, and registers it as face image data in the database 16a.

  The microphone 52 can be used to collect a human voice that is the basis of voice data in the database 16a. The sound processing unit 30a performs necessary processing on the sound signal collected by the microphone 52, and then registers it as sound data in the database 16a. For this processing, the audio processing unit 30a includes a preamplifier that amplifies the output of the microphone 52 and an A / D converter that digitizes the analog output of the preamplifier.

  With reference to FIG. 8, a method of registering the captured image data and audio data in the database 16a will be described.

  The user selects a database registration function from the function menu of the playback device 10a (S41), and creates new personal information in the database 16 (S42).

  First, face image data is registered (S43). Specifically, the camera unit 50 is activated and is ready for shooting (S44). A person to be registered is taken as a subject, and an image including the face is photographed (S45). At this time, an image captured by the camera unit 50 is displayed on the display unit 28. If the person who is the subject can check the positional relationship between the camera unit 50 and his / her face while viewing the display image on the display unit 28, efficient shooting can be performed. When the subject is accurately captured within the shooting angle of view, the switch of the input device 46 is pressed to perform shooting, and the face image of the subject is captured (S46). The image processing unit 18a processes the image data from the camera unit 50 into a format and size suitable for registration in the database 16a. At this time, the image processing unit 18a calculates a face image feature amount from the captured face image data (S47), and registers it in the database 16a together with the face image data (S48). In this face image feature amount calculation process, a function for calculating a face image feature amount of a person detected from a reproduced image can be used.

  Next, voice data is registered (S49). The audio processing unit 30a activates a function for processing an audio signal collected by the microphone 52 (S50). When the sound collection preparation is complete and the sound can be collected, a display prompting the user to input a voice is displayed on the display unit 28. If a phrase suitable for registration is displayed and the registrant speaks it, it is preferable because the registrant does not get lost in the phrase to be spoken. When recording a voice, the switch of the input device 46 is pressed at the start and end of the recording, or the recording period is set while the switch is being pressed.

  The voice processing unit 30a temporarily holds the voice from the microphone 52 (S51). Then, this sound is reproduced together with the display to confirm with the user whether or not registration is allowed (S52). If the user refuses to register, the voice capture is performed again. When the user approves the registration, the voice processing unit 30a processes the captured voice into a format and size suitable for registration in the database 16a, calculates voice feature quantities (S53), and obtains voice data and voice feature quantities. Register in the database 16a (S54). In this voice feature amount calculation process, a function for calculating a voice feature amount from the reproduced voice can be used.

  Subsequently, for example, other items such as names are entered in the database 16a (S55).

  The registration order shown in FIG. 8 is an example. For example, it is obvious that the face image data may be registered after first entering “name”.

  The personal information data of the database 16a registered in this way can be used when reproducing the image signal and the audio signal recorded on the recording medium 12 in the reproducing apparatus 10 of the first embodiment.

  FIG. 9 shows a schematic block diagram of the third embodiment of the present invention. A video signal with sound is recorded on the recording medium 112 of the playback device 110. The recording medium 112 is composed of an optical disk typified by a DVD, a magnetic tape, a hard disk, or a memory card incorporating a flash memory. The recording medium driving device 114 is a device that drives the recording medium 112 to read / write signals from / to the recording medium 112.

  Similar to the database 16, personal information data of a plurality of persons can be registered in the database 116, and is registered in practice. Each personal information data includes face image data used for the face image matching function and face image feature value data indicating the feature value, voice data used for the human voice matching function and voice feature value data indicating the feature value, and Includes various other data.

  The image processing unit 118 decodes the compressed image data read from the recording medium 112, performs various processes, and generates reproduced image data. The face image recognition unit 120 determines whether or not there is a region corresponding to a human face from the image data processed by the image processing unit 118. The face image matching unit 122 compares the area determined by the face image recognition unit 120 that there is a face image with the face image data registered in the database 116. Thereby, it is possible to specify whether or not the person included in the reproduced image is registered in the database 116 and who is registered.

  The additional image generation unit 124 generates additional image data to be superimposed on the reproduced image signal reproduced from the recording medium 112. The display image generation unit 126 combines the additional image data output from the additional image generation unit 124 with the reproduction image data output from the image processing unit 118. The display image generation unit 126 generates a composite image in which the additional image is superimposed on the reproduced image. However, the output image of the display image generation unit 126 may be composed of only a reproduced image or only an additional image in addition to the composite image.

  The display unit 128 displays the image data generated by the display image generation unit 126 as an image so that the user can visually recognize the image data. The display unit 128 is incorporated in the playback device, for example. The display unit 128 includes a display device such as a liquid crystal display (LCD) or an organic EL display and a driving circuit thereof.

  The audio processing unit 130 decodes the compressed audio data read from the recording medium 112 and performs various processes. The human voice recognition unit 132 determines whether or not there is a voice corresponding to a human voice from the reproduced voice signal obtained by the voice processing unit 130. The human voice collation unit 134 compares the portion determined as the human voice by the human voice recognition unit 132 with the voice data registered in the database 116. When the human voice recognition unit 132 determines that the voice is a human voice, the volume detector 136 detects the volume level. The audio output device 138 is a device that acoustically outputs the reproduced audio generated by the audio processing unit 130. The audio output device 138 includes, for example, a speaker incorporated in the playback device 110 and its drive circuit.

  The additional image display determination unit 140 determines the content of the additional image to be generated by the additional image generation unit 124 based on the matching result of the face image matching unit 122 and the matching result of the human voice matching unit 134.

  The external output unit 142 is a means for outputting the image signal generated by the display image generation unit 126 and the reproduced audio signal by the audio processing unit 130 to the outside. For example, the external output drive circuit and the connection terminal or transmission antenna It consists of.

  The camera unit 150 includes a lens and an image sensor, and can be used to capture face image data to be registered in the database 116. That is, the image processing unit 118 performs color balance and gamma correction on the image signal captured by the camera unit 150, adjusts the size and the like, and registers the image signal as face image data in the database 116. The image processing unit 118 also extracts the feature amount of the photographed image from the camera unit 150 and registers the extracted feature amount in the database 116 as face image feature amount data.

  The microphone 152 can be used to collect a human voice that is the basis of voice data in the database 116. The sound processing unit 130 performs necessary processing on the sound signal collected by the microphone 152 and then registers the sound signal in the database 116 as sound data. For this processing, the audio processing unit 130 includes a preamplifier that amplifies the output of the microphone 52 and an A / D converter that digitizes the analog output of the preamplifier. The voice processing unit 130 also extracts the feature amount of the collected sound data and registers it in the database 116 as voice feature amount data.

  The CPU 144 is a central processing unit that not only controls the additional image generation unit 124 and the display image generation unit 126 but also controls the entire playback device 110 so that image processing and audio processing operate synchronously.

  The input device 146 is a device for a user to input an operation mode, an operation condition, and the like to the playback device 110, and includes various switches or buttons, operable elements displayed on a menu screen, and the like.

  The authoring processing unit 160 converts the output signals of the image processing unit 118 and the audio processing unit 130 added with the additional image generated by the additional image generation unit 124 as a sub-picture into data conforming to a prescribed format. The authoring processing unit 160 records the processing result on the recording medium 164 via the recording medium driving device 162. The recording medium 164 is a recording storage destination of data generated by the authoring processing unit 160. The recording medium driving device 162 drives the recording medium 164 by a control method corresponding to the recording medium 164, and reads / writes signals from / to the recording medium 164.

  It is obvious that the recording medium 112 may be the recording destination of the processing result of the authoring processing unit 160. In this case, of course, the recording medium 112 is not a read-only recording medium but a recordable medium. In the embodiment shown in FIG. 9, only the recording medium 112 and the recording medium 164 are illustrated separately so that the storage location of each data can be easily understood.

  FIG. 10 is a flowchart showing the characteristic operation of this embodiment. With reference to FIG. 10, the characteristic operation of the present embodiment will be described.

  First, personal information data of a necessary person is registered in advance in the database 116 (S101). As described above, each person's personal information data includes face image data and face image feature amount data indicating the feature amount, voice data and voice feature amount data indicating the feature amount, and various other attributes. Consists of data. The attribute data includes, for example, name, nickname, sex ", age, date of birth, and portrait.

  Of the data in the database 116, face image data, voice data, and data indicating these feature quantities can be registered in the database 116 using the camera unit 150, the microphone 152, the image processing unit 118, and the audio processing unit 130. A specific method will be described later.

  The audio processing unit 130 decompresses the compressed audio data read from the recording medium 112 and supplies the reproduced audio signal to the human voice recognition unit 132, the audio output device 138, the external output unit 142, and the authoring processing unit 160. The human voice recognition unit 132 extracts and analyzes a human voice from the reproduced voice signal, and determines whether or not a human voice is included (S102). If a human voice is included, the human voice recognition unit 132 supplies the extracted voice data to the human voice verification unit 134.

  The human voice collation unit 134 extracts the feature amount from the human voice data from the human voice recognition unit 132, collates it with the voice feature amount data in the database 116, and selects a person having a feature amount having a high correlation of a certain level or more. Search is performed (S103). When voice feature data having a degree of correlation equal to or greater than a certain value exists, the human voice verification unit 134 notifies the additional image display determination unit 140 of a person having the voice feature data as a candidate for additional image display ( S104).

  On the other hand, the image processing unit 118 decompresses the compressed image data reproduced from the recording medium 112 to generate reproduced image data. The face image recognition unit 120 extracts a human face image from the reproduced image, and supplies the extracted face image data to the face image matching unit 122 (S105). The face image collation unit 122 extracts the feature amount from the face image data from the face image recognition unit 120 and collates it with the face image feature amount data in the database 116 to obtain a face image feature amount having a high correlation of a certain level or more. The person who has it is searched (S106). When face image feature amount data having a degree of correlation equal to or greater than a certain value exists, the face image matching unit 122 sets a person having the face image feature amount data as an additional image display exclusion candidate to the additional image display determination unit 140. Notification is made (S107).

  The additional image display determination unit 140 excludes the exclusion candidate extracted from the reproduction image (exclusion candidate from the face image verification unit 122) from the candidate extracted from the reproduction sound (notification information from the human voice verification unit 134). (S108). Thereby, in the same scene, the person (S107) in the screen can be excluded from the person candidates (S104) who are likely to be speakers. That is, it is possible to specify a person who is not in the shooting screen but has recorded voice. The additional image display determination unit 140 notifies the additional image generation unit 124 of the person specified in this way.

  The additional image generation unit 124 acquires the face image data of the person notified from the additional image display determination unit 140 from the database 116 (S109), and generates an additional image of a predetermined size including the face image (S110). The additional image generation unit 124 supplies the generated additional image to the display image generation unit 126 and the authoring processing unit 160.

  The display image generation unit 126 superimposes the additional image from the additional image generation unit 124 on the reproduced image data from the image processing unit 118 to generate composite image data. The composite image data is applied to the display unit 128 and displayed as shown in FIG. Of course, it may be output from the external output unit 142 to the outside.

  Further, the authoring processing unit 160 multiplexes the reproduction image signal from the image processing unit 118, the reproduction audio signal from the audio processing unit 130, and the additional image from the additional image generation unit 124 to generate one video content. . At this time, a video signal is generated by superimposing the additional image as a pictogram-like sub-picture on the reproduced image signal (S111). The authoring processing unit 160 records the video content thus generated on the recording medium 164 by the recording medium driving device 162 (S112). For example, a DVD in VIDEO-VIDEO format is created. In the case of the DVD-VIDEO format, by turning on the “subtitle” function, information of a person uttering outside the angle of view can be viewed while being overlaid on the main video.

  The database 116 will be described. Similar to the database 16, the database 116 may be stored in the recording medium 112 or built in the playback device 110. When the playback device 110 is network compatible, the database 116 may be prepared on a server connected via the network. When the recording medium 112 is detachable from the playback device 110, a method of directly referring to the database on the recording medium 112 even when using a database recorded on the recording medium 112 by loading it into the storage area of the playback device 110. Either may be sufficient. In the former case, a removable recording medium in which only the database is recorded may be prepared.

  The structure of the database 116 is the same as the structure of the database 16. The personal information data of each person registered in the database 116 necessarily includes face image data for use in face image matching and pictogram display, and voice data for use in voice matching. In addition, data such as name, nickname, gender, age, date of birth, and caricature are stored as additional attribute information. These additional attribute information may be used for pictogram display at the same time as or in place of the face image data.

  As the number of people registered in the database 116 increases, there is a classification of a person who wants to display as a pictograph and a person who does not want to display. In order to correspond to this classification, it is preferable to provide an item for setting whether or not to display a pictograph for each personal information data. Which is set by default when newly registering newly registered personal information data may be determined in accordance with the usage status.

  Similarly to the database 16, the database 116 is preferably versatile so that it can be read and used by another device.

  The pictograph display of the present embodiment will be described. Displaying information about a person who can only hear a voice as a pictograph is useful because it can add visibility. And the effect can be further increased by devising the display method.

  If the user can set the validity / invalidity of the pictograph display function, it is possible to respond to a request to view only the original recorded image. This is effective when the person who is not shown in the image is not cared about, or when a known voice is heard.

  The information displayed as a pictograph is one or more items among the items of the database 116. If an item to be displayed can be selected, it can be linked to information that the user wants to know. For example, if “name” is displayed at the same time as the face image, it becomes easier to identify a person even when it is difficult to determine who the face image alone is. Of course, the same effect can be obtained by displaying only the “name”.

  If the data items to be displayed for all the pictogram display target persons are unified, there is a possibility that the corresponding item is not registered in some persons. In order to cope with such a situation, it is preferable to give priority to each item of the pictograph display data and display the next item when the upper item is not registered. For example, it is assumed that “nickname” is set as the first priority and “name” is set as the second priority. In this case, “nickname” is displayed for a person for whom the item “nickname” is registered, and “name” is displayed for a person for whom the item “nickname” is not registered. The task of setting such priorities becomes more complicated as the number of items of personal information data of each person that can be registered in the database 116 increases. To this end, the user can set up to a finite number of higher ranks, and the playback apparatus 110 may automatically allocate items below that rank. This reduces the burden on the user. Even if the user wants to add any data to the items in the personal information data in the database 116, the face image data, which is a registration-required item, is assigned to any priority order, so that it is displayed on the pictograph. Data is secured.

  It is desirable that the pictograph on the screen does not block as much as possible the portion of high interest in the reproduced image. In general, the degree of attention of an image tends to be higher in the central part than in the peripheral part. Therefore, the display location of the pictograph is basically arranged in the peripheral part of the reproduced image. However, in a scene where the subject is moving, the attention degree of the peripheral portion may increase as the subject moves. At this time, it is not desirable that the place where the degree of attention is high and the place where the pictograph is displayed overlap. If the user can set the display location of the pictograph, such a situation can be avoided.

  It is convenient if the user can also set the size of the pictograph. For example, there are formats that allow you to select the size from a specified level expressed as “Large”, “Medium”, “Small” or the number of dots, or a format that allows you to select a size using a chart with similar expressions. It is done. This is because the number of pixels of the display screen is large, such as when viewing on a relatively small display screen provided in the playback apparatus 10 and when viewing on a relatively large screen such as a television via an external connection function. Useful when different. This is because there is a non-negligible difference in the size of the pictograph required to satisfy the visibility.

  Further, regarding the shape, if options such as a rectangle and a circle are provided, the user's preference can be utilized. However, it is necessary to limit the data that can be displayed according to the set shape. For example, data that can be expressed while satisfying visibility within a small external dimension is limited. The actual restriction may be provided in a manner suitable for each playback device. When such a shape that restricts is set as the shape of the pictograph, the highest priority is displayed among the displayable data candidates. For both the size and the shape, it is preferable that the size and shape of the actual pictograph change simultaneously with the setting change operation.

  If an upper limit is set for the number of pictographs displayed on the screen at the same time, it is possible to prevent the main picture and other pictographs from being obscured by the appearance of a large number of pictographs. If the user can set the upper limit number or the maximum number of pictographs displayed on one screen or for the same scene, an appropriate change can be made according to the size of the television monitor to be reproduced and the visual recognition ability. When the number of displays is limited in this way, what kind of pictograph is given priority for display becomes important. The simplest method is to update the pictographs one after another when the speaker is recognized. This is a so-called “late arrival priority type” focusing on the chronological utterance order. The display target is always from the latest speaker to the speaker at a time point that goes back by the number equal to the display upper limit of the pictograph. The display is not updated as long as the same person continues to speak, but is updated when another person's speech is inserted.

  In addition, a method linked to the utterance volume can be considered. The sound volume of the person is measured using the sound volume detection unit 136, and the display priority order of the pictograph is determined from the result. If the priority increases in the order of the volume, the pictograph of the person who can easily reach the ear is displayed with priority. On the other hand, if the priority increases in order of increasing volume, a pictograph of a person who is more difficult to hear and misses is displayed with priority.

  It is also effective to determine the person who wants to display with priority. This can be realized by having a pictograph display priority setting value in the registration item of the database 116 and the additional image display determination unit 140 performs processing according to this setting value.

  For example, when the video source is a home video, it is possible that the photographer speaks frequently from the shortest distance. At this time, if the priorities are given as described above, the photographer's pictograph is displayed with a very high frequency. Although it is certain that the photographer often does not appear in the video, it is relatively easy to understand who it was. Therefore, by making it possible to input who the photographer is and making the pictograph of that person different from other methods or not displaying it, the complexity of pictograph display can be reduced. .

  Along with improved visibility, an entertaining interface is also possible. In the first method, the size of the pictograph is changed according to the volume of the uttered voice. The volume data detected by the volume detection unit 136 and the external dimensions of the pictograph are correlated. For example, when the magnitude is changed in three stages, two threshold levels are provided in the volume data, and if the threshold level is lower than the lower threshold level, the threshold value is small. If it is above, increase it.

  In the second method, the display position of the pictograph is changed according to the directivity angle of the voice. The sound volume detection unit 136 analyzes the reproduced sound, and examines the direction from which the viewer recognizes the sound with respect to the speaker output. When the reproduced sound is in the two-channel mode, the pictograph is changed in the horizontal direction according to the sound directivity angle and displayed. In the surround mode, the pictograph is arranged with the left and right direction of the sound as left and right and the front and back direction as up and down. For example, a pictograph corresponding to the sound heard from the right front is displayed at the upper right of the reproduced image.

  Since the voicing volume and the directivity angle change constantly, the pictogram display should follow the changes. By updating the pictograph display at an appropriate time interval, it is possible to visually represent the change trend of the voice.

In addition, based on data registered in the database 116, it may be possible to color-code according to gender, age, or the like.
If the display timing is taken into consideration, the display can be made easier to see. For example, let us consider a case where a short utterance “A” uttered with a surprising beat is detected. When a quick reaction is made to such a sound detection result, the pictograph is displayed for an instant. This makes it very difficult to confirm who the person displayed on the pictograph is. Therefore, the authoring processing unit 160 is set so as to start displaying a pictograph corresponding to a certain voice in advance for a certain period of time. Also, at the end of utterance, the display is set to disappear after holding the display for a certain time. That is, the authoring processing unit 160 performs the authoring so that the display of the sub picture starts before the start of the utterance and the display of the sub picture ends after the end of the utterance. By doing this, a time margin for confirming the pictograph before and after the actual utterance time is created, and the visibility for a short utterance is improved. Alternatively, if it is possible to detect the utterance time in advance, the timing adjustment described above may be performed only for utterances less than a predetermined time.

  When sub-pictures are set by combining the various pictograph display methods described above, the following is performed. That is, if authoring processing for generating a plurality of sub-picture channels is possible, a plurality of channels with different combination methods are prepared. For example, it is possible to use a method in which a certain channel is a priority display for a later arrival, another channel is a priority display for a volume, and another channel is color-coded according to a directivity angle and sex. If this is authored in the DVD VIDEO format, various displays can be enjoyed by switching “subtitles”. FIG. 11 shows an example of a pictograph display in which the upper limit number of pictograms displayed at the same time is 3, and the voicing volume and the directional angle of utterance are linked.

  Thus, in this embodiment, it is possible to create an information medium capable of displaying, as a sub-picture, a pictograph indicating information related to a person who can hear only the reproduced voice without appearing in the reproduced image.

  If the reproduced voice signal and the voice data in the database 116 are collated with their respective feature amounts and a correct person cannot be selected, incorrect data is used for the pictograph. When the user notices an error by looking at the displayed pictograph, the present embodiment can correct the content correctly. FIG. 12 shows a flowchart of the correction operation.

  The user looks at the reproduced display image and confirms whether the voice to be heard and the personal information displayed on the pictograph are correct or incorrect (S121). If an error is found and corrected, the correction function is used (S122). While viewing the screen of the display unit 128, the input device 146 such as a switch or a touch panel is used to correct so that correct information is displayed as a histogram (S123 to S131).

  This will be specifically described. When the user selects correction (S122), the correction mode is entered and a screen for selecting a pictograph on the screen is displayed (S123). When selecting a pictograph, a pictograph indicating “no corresponding person” is displayed for a voice that has been excluded from the display target because there is no corresponding person in the database 116 so that it can be selected. It is desirable to do. As a result, it is possible to correct an error in which the human voice matching unit 134 could not identify the reproduced voice signal and the voice data in the database 116.

  The pictograph selected on the pictograph selection screen changes its shape, color, border, etc. and displays some highlights. The user selects a pictograph to be corrected using the input device 146 (S124). Immediately after the selection, a dialog for confirming the intention of correction, “Are you sure to correct?” Is displayed (S125). Subsequently, data to be used after correction is searched and selected (S126).

  At the same time, the target range of the pictograph to be corrected is selected (S127). Providing such a selection step can reduce the possibility of erroneous selection. For example, the first option is limited to “only selected”. The second option is limited to “all those where the same person as the selected pictograph is recognized”. The third option is limited to “one in which the same person as the selected pictograph is recognized, and one having a lower correlation rate of voice collation than the selected scene”. As a method for expressing these options, a method suitable for each reproducing apparatus is selected.

  Whether or not correction can be executed is confirmed (S128). In accordance with the user's decision to execute correction, actual pictograph display data replacement processing is performed (S129). At this time, if the range of simultaneous correction with the target pictograph is reflected in the database 116, the accuracy of human voice verification after the next time is improved.

  When there are other pictographs to be corrected (S130), correction is performed by the same procedure. If there is no other pictograph to be corrected (S130), the correction mode is exited (S131), and a series of correction operations is completed.

  With the above configuration and operation, when an image signal and an audio signal are reproduced, information related to a person who can hear only a voice without being reflected in the image can be visually recognized.

  In this embodiment, the camera unit 150 and the microphone 152 can be used to register face image data and voice data in the database 116. With reference to FIG. 13, a method of registering the captured image data and audio data in the database 116 will be described.

  The user selects a database registration function from the function menu of the playback device 110 (S141), and creates new personal information in the database 116 (S142).

  First, face image data is registered (S143). Specifically, the camera unit 150 is activated and is ready for shooting (S144). A person to be registered is taken as a subject, and an image including the face is photographed (S145). At this time, an image captured by the camera unit 150 is displayed on the display unit 128. If the person who is the subject himself / herself can confirm the positional relationship between the camera unit 150 and his / her face while viewing the display image on the display unit 128, efficient shooting can be performed. When the subject is accurately captured within the shooting angle of view, the switch of the input device 146 is pressed to perform shooting, and the face image of the subject is captured (S146). The image processing unit 118 processes the image data from the camera unit 150 into a format and size suitable for registration in the database 116. At this time, the image processing unit 118 calculates a face image feature amount from the captured face image data (S147), and registers it in the database 116 together with the face image data (S148). In this face image feature amount calculation process, a function for calculating a face image feature amount of a person detected from a reproduced image can be used.

  Next, voice data is registered (S149). The audio processing unit 130 activates a function for processing an audio signal collected by the microphone 152 (S150). When the sound collection is ready and the sound can be collected, the display unit 128 displays a message prompting the user to input a voice. If a phrase suitable for registration is displayed and the registrant speaks it, it is preferable because the registrant does not get lost in the phrase to be spoken. When recording a voice, the switch of the input device 146 is pressed at the start and end of recording, or the recording period is set while the switch is being pressed.

  The voice processing unit 130 temporarily holds the voice from the microphone 152 (S151). Then, this sound is reproduced together with the display to confirm with the user whether registration is allowed (S152). If the user refuses to register, the voice capture is performed again. When the user approves the registration, the voice processing unit 130 processes the captured voice into a format and size suitable for registration in the database 116, calculates voice feature quantities (S153), and obtains voice data and voice feature quantities. Registration in the database 116 (S154). In this voice feature amount calculation process, a function for calculating a voice feature amount from the reproduced voice can be used.

  Subsequently, other items such as a name are input to the database 116 (S155).

  The registration order shown in FIG. 13 is an example. For example, it is obvious that the face image data may be registered after first inputting “name”.

1 is a schematic block diagram of a playback apparatus according to a first embodiment of the present invention. It is explanatory drawing of the imaging | photography condition. It is a schematic diagram which shows the reproduction | regeneration state in a prior art example. It is a schematic diagram which shows the reproduction | regeneration state by a present Example. It is a flowchart explaining the production | generation process of the additional image in a present Example. It is a flowchart explaining the pictograph correction process in a present Example. It is a schematic block diagram of the second embodiment of the present invention. It is a flowchart explaining the personal information registration process to the database in 2nd Example. It is a schematic block diagram of the third embodiment of the present invention. It is a flowchart explaining the production | generation process of the additional image in 3rd Example. It is a figure which shows the example of a pictogram display in 3rd Example. It is a flowchart explaining the pictograph correction process in 3rd Example. It is a flowchart explaining the personal information registration process to the database in 3rd Example.

Explanation of symbols

10, 10a, 110: Playback device 12, 112: Recording medium 14, 114: Recording medium driving device 16, 16a, 116a: Database 18, 118: Image processing unit 20, 120: Face image recognition unit 22, 122: Face image Verification unit 24, 124: Additional image generation unit 26, 126: Display image generation unit 28, 128: Display unit 30, 130: Voice processing unit 32, 132: Human voice recognition unit 34, 134: Human voice verification unit 36, 136 : Volume detection unit 38, 138: Audio output device 40, 140: Additional image display determination unit 42, 142: External output unit 44, 144: CPU
46, 146: input device 50, 150: camera unit 52, 152: microphone 160: authoring processing unit 162: recording medium driving device 164: recording medium

Claims (17)

Image processing means for reproducing image data read from a recording medium and outputting a reproduced image signal;
Audio processing means for reproducing audio data read from the recording medium and outputting a reproduced audio signal;
A database that records voice data and face image data of multiple people,
Human voice recognition means for discriminating and recognizing human voice from the reproduced voice signal;
A voice collation means for collating and identifying a voice recognized by the voice recognition means and voice data registered in the database;
Face image recognition means for discriminating and recognizing a person's face from the reproduced image signal;
Face image collating means for collating and identifying the face of the person recognized by the face image recognizing means and the face image data registered in the database;
An additional image display determining means for determining a person who excludes the person identified by the face image matching means from the persons identified by the voice matching means in the same scene, as an additional image display target;
Additional image generation means for reading information indicating the target person determined by the additional image display determination means from the database and generating an additional image to be combined with the reproduced image signal;
And a display image generating means for synthesizing the additional image with the reproduced image signal.   The playback apparatus according to claim 1, wherein the database includes data supplied to the additional image generation unit separately from the face image data used by the face image matching unit. The database has, for each person, information indicating whether or not the reproduction image signal and the additional image can be combined in the display image generation unit,
3. The playback apparatus according to claim 1, wherein the display image generation unit synthesizes the additional image with the playback image signal when information allowing composition is registered in the database. 4. Furthermore,
A camera unit for taking images,
A microphone that picks up sound,
The playback apparatus according to any one of claims 1 to 3, further comprising means for registering a face image photographed by the camera unit and sound collected by the microphone in the database.   5. The reproducing apparatus according to claim 1, further comprising means for a user to set whether or not the additional image generation means needs to generate the additional image.   Means for setting which information the user uses to generate the additional image when there is a plurality of information that can be used to generate the additional image in the information of each person in the database; The reproducing apparatus according to claim 1, wherein the reproducing apparatus is characterized in that:   The priority order can be set for the plurality of pieces of information when there is a plurality of pieces of information that can be used to generate the additional image in the information of each person in the database. The reproducing apparatus as described.   When there is a plurality of pieces of information that can be used to generate the additional image in the information of each person in the database, the user has means for setting the priority order of all or a part of the plurality of pieces of information. The playback apparatus according to claim 7, characterized in that:   9. The reproduction apparatus according to claim 1, wherein the display image generation unit arranges the additional image around an image indicated by the reproduction image signal.   And authoring means for generating information including an image signal obtained by synthesizing the additional image as a sub-picture with the reproduced image signal and the reproduced audio signal and recording the information on the recording medium or a recording medium different from the recording medium. The playback apparatus according to claim 1, wherein the playback apparatus is provided.   The reproduction apparatus according to claim 10, wherein the number of the additional images generated by the additional image generation unit for the same scene is limited.   12. The playback apparatus according to claim 10, wherein when a plurality of the additional images exist for the same scene, the display priority order of the plurality of additional images is determined in the order of utterance.   12. The playback apparatus according to claim 10 or 11, wherein when a plurality of the additional images are present for the same scene, the display priority order of the plurality of additional images is determined in the order of the sound volume. .   11. The display priority order of the plurality of additional images is determined according to the display priority of each person registered in the database when there are a plurality of the additional images for the same scene. Or the reproducing apparatus of 11. Furthermore, it has means for detecting the direction of the reproduced audio signal,
The playback apparatus according to claim 10, wherein the authoring means links the display position of the sub-picture in the direction.   11. The playback apparatus according to claim 10, wherein the authoring means performs an authoring process so as to start displaying a sub-picture temporally before utterance.   11. The playback apparatus according to claim 10, wherein the authoring means performs an authoring process so as to end the display of the sub-picture with a time delay from the end of the utterance.

Images