JP5349651B2 - Electronic device, face image extraction control method and program - Google Patents

Description

  The present invention relates to an image display control technique for a user interface that enables a user to efficiently select a playback start position of video content data.

  Generally, electronic devices such as a video recorder and a personal computer can record and reproduce various video content data such as television broadcast program data. In this case, a title name is added to each video content data stored in the electronic device, but it is not possible for the user to grasp what the content of each video content data is based on the title name alone. Have difficulty. For this reason, in order to grasp the content of the video content data, it is necessary to reproduce the video content data. However, reproduction of video content data with a long total time length requires a lot of time even when a fast-forward reproduction function or the like is used.

  Patent Document 1 discloses an apparatus having a character list display function. This apparatus has a function of displaying a face image of each character in the video content side by side as a list of characters in the video content.

JP 2001-309269 A

  By the way, when displaying face images side by side, it is preferable that the face images have the same position and size in the image as well as the display size. For this purpose, it is necessary to appropriately determine the cutout range of the face area from the original image.

  However, considering the alignment and position of the face with other face images, that is, the position and size of the face in the image will be constant, determine the cutout range of the face area from the original image Then, for example, when the face area is located at the end of the original image, the cutout range must be determined by protruding the original image. Therefore, a mechanism for appropriately adjusting in such a case is necessary.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an electronic device, a face image extraction control method, and a program that appropriately control the extraction of a person's face image from video content data. And

According to the embodiment, the electronic device is an electronic device for extracting a human face image from video content, and includes a detection unit, an adjustment unit, and an extraction unit. The detection unit determines a cutout range for cutting out the detected face image when the frame of the video content includes a face image. The adjusting means adjusts the first face image included in the cutout range and outputs the second face image when the cutout range extends outside the frame. The extraction means outputs the second face image and time stamp information corresponding to the time point when the first face image appears in the video content. The time stamp information is used to display the second face image corresponding to the time point when the first face image appears in the video content.

The perspective view which shows the example of the external appearance of the electronic device which concerns on one Embodiment of this invention. 2 is an exemplary block diagram showing an example of the system configuration of the electronic apparatus of the embodiment. FIG. Block diagram for explaining a face image list display function of the electronic device of the embodiment 6 is a diagram showing an example of section attribute information (attribute detection result information) used in the electronic apparatus of the embodiment A block diagram showing a functional configuration of a program used in the electronic apparatus of the embodiment 6 is a diagram showing an example of an indexing view screen displayed on the display device by the electronic apparatus of the embodiment. FIG. The figure which shows the structural example of the face thumbnail display area contained in the indexing view screen of FIG. The figure for demonstrating the relationship between the face thumbnail display area and the jade thumbnail display area which are contained in the indexing view screen of FIG. FIG. 6 is a first diagram for explaining cooperative display control between a face thumbnail display area and a jagged thumbnail display area included in the indexing view screen of FIG. FIG. 6 is a second diagram for explaining cooperative display control between a face thumbnail display area and a jagged thumbnail display area included in the indexing view screen of FIG. FIG. 10 is a view showing a list display example of face images adjusted at the time of extraction by the electronic apparatus of the embodiment. 1st figure for demonstrating the 1st example of the extraction principle of the face image performed with the electronic device of the embodiment 2nd figure for demonstrating the 1st example of the extraction principle of the face image performed by the electronic device of the embodiment A flowchart showing an example of a first procedure of face image extraction processing executed by the electronic apparatus of the embodiment 1st figure for demonstrating the 2nd example of the extraction principle of the face image performed with the electronic device of the embodiment 2nd figure for demonstrating the 2nd example of the extraction principle of the face image performed with the electronic device of the embodiment 3rd figure for demonstrating the 2nd example of the extraction principle of the face image performed by the electronic device of the embodiment FIG. 4 is a fourth diagram for explaining a second example of the face image extraction principle executed by the electronic apparatus of the embodiment. A flowchart showing an example of a second procedure of face image extraction processing executed by the electronic apparatus of the embodiment

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, with reference to FIG. 1 and FIG. 2, the structure of the electronic device which concerns on one Embodiment of this invention is demonstrated. The electronic apparatus according to the present embodiment is realized by, for example, a notebook portable personal computer 10 that functions as an information processing apparatus.

  The personal computer 10 can record and reproduce video content data (audio visual content data) such as broadcast program data and video data input from an external device. That is, the personal computer 10 has a television (TV) function for viewing and recording broadcast program data broadcast by a television broadcast signal. This TV function is realized by, for example, a TV application program installed in the personal computer 10 in advance. The TV function also has a function of recording video data input from an external AV device, and a function of reproducing recorded video data and recorded broadcast program data.

  Further, the personal computer 10 includes a face image list display function for displaying a list of face images of persons appearing in video content data such as video data and broadcast program data stored in the personal computer 10, and the video content data. For example, a thumbnail image display function for displaying an image (reduced image) corresponding to each of a plurality of frames extracted at equal time intervals.

  For example, the face image list display function and the thumbnail image display function are implemented as one function in the TV function. The face image list display function is one of video indexing functions for presenting an outline of video content data to the user. This face image list display function can present to the user which person appears in which time zone in the entire video content data. The face image list display function can also display a list of persons by paying attention to a predetermined attribute section included in the video content data.

  FIG. 1 is a perspective view of the computer 10 with the display unit opened. The computer 10 includes a computer main body 11 and a display unit 12. The display unit 12 incorporates a display device including a TFT-LCD (Thin Film Transistor Liquid Crystal Display) 17.

  The display unit 12 is attached to the computer main body 11 so as to be rotatable between an open position where the upper surface of the computer main body 11 is exposed and a closed position covering the upper surface of the computer main body 11. The computer main body 11 has a thin box-shaped housing, and has a keyboard 13 on its upper surface, a power button 14 for powering on / off the computer 10, an input operation panel 15, a touch pad 16, and a speaker. 18A, 18B, etc. are arranged.

  The input operation panel 15 is an input device that inputs an event corresponding to a pressed button, and includes a plurality of buttons for starting a plurality of functions. These button groups also include operation button groups for controlling TV functions (viewing, recording, and reproduction of recorded broadcast program data / video data). In addition, a remote control unit interface unit 20 for executing communication with a remote control unit for remotely controlling the TV function of the computer 10 is provided on the front surface of the computer main body 11. The remote control unit interface unit 20 includes an infrared signal receiving unit and the like.

  On the right side of the computer main body 11, for example, an antenna terminal 19 for TV broadcasting is provided. Further, on the back surface of the computer main body 11, for example, an external display connection terminal corresponding to the HDMI (high-definition multimedia interface) standard is provided. The external display connection terminal is used to output video data (moving image data) included in video content data such as broadcast program data to an external display.

  Next, the system configuration of the computer 10 will be described with reference to FIG.

  As shown in FIG. 2, the computer 10 includes a CPU 101, a north bridge 102, a main memory 103, a south bridge 104, a graphics processing unit (GPU) 105, a video memory (VRAM) 105A, a sound controller 106, a BIOS- ROM 109, LAN controller 110, hard disk drive (HDD) 111, DVD drive 112, video processor 113, memory 113A, card controller 113, wireless LAN controller 114, IEEE 1394 controller 115, embedded controller / keyboard controller IC (EC / KBC) 116 TV tuner 117, EEPROM 118, and the like.

  The CPU 101 is a processor that controls the operation of the computer 10 and executes various application programs such as an operating system (OS) 201 and a TV application program 202 that are loaded from the hard disk drive (HDD) 111 to the main memory 103. . The TV application program 202 is software for executing a TV function. The TV application program 202 includes a live reproduction process for viewing broadcast program data received by the TV tuner 117, a recording process for recording the received broadcast program data in the HDD 111, and broadcast program data / data recorded in the HDD 111. A reproduction process for reproducing video data is executed. The CPU 101 also executes a BIOS (Basic Input Output System) stored in the BIOS-ROM 109. The BIOS is a program for hardware control.

  The north bridge 102 is a bridge device that connects the local bus of the CPU 101 and the south bridge 104. The north bridge 102 also includes a memory controller that controls access to the main memory 103. The north bridge 102 also has a function of executing communication with the GPU 105 via a PCI EXPRESS standard serial bus or the like.

  The GPU 105 is a display controller that controls the LCD 17 used as a display monitor of the computer 10. A display signal generated by the GPU 105 is sent to the LCD 17. The GPU 105 can also send a digital video signal to the external display device 1 via the HDMI control circuit 3 and the HDMI terminal 2.

  The HDMI terminal 2 is the above-described external display connection terminal. The HDMI terminal 2 can send an uncompressed digital video signal and a digital audio signal to the external display device 1 such as a television with a single cable. The HDMI control circuit 3 is an interface for sending a digital video signal to the external display device 1 called an HDMI monitor via the HDMI terminal 2.

  The south bridge 104 controls each device on an LPC (Low Pin Count) bus and each device on a PCI (Peripheral Component Interconnect) bus. The south bridge 104 includes an IDE (Integrated Drive Electronics) controller for controlling the hard disk drive (HDD) 111 and the DVD drive 112. Further, the south bridge 104 has a function of executing communication with the sound controller 106.

  Furthermore, a video processor 113 is connected to the south bridge 104 via a PCI EXPRESS standard serial bus or the like.

  The video processor 113 is a processor that executes various processes related to the above-described video indexing. The video processor 113 functions as an indexing processing unit for executing a video indexing process. That is, in the video indexing process, the video processor 113 extracts a plurality of face images from the moving image data included in the video content data, and time indicating the time when each extracted face image appears in the video content data. Output stamp information, etc. The extraction of the face image is executed by, for example, a face detection process for detecting a face area from each frame of moving image data, a cutout process for cutting out the detected face area from the frame, and the like. The face area can be detected by, for example, analyzing an image feature of each frame and searching for an area having a feature similar to a face image feature sample prepared in advance. The face image feature sample is feature data obtained by statistically processing the face image features of a large number of persons.

  Furthermore, the video processor 113 also executes, for example, a process for detecting a commercial (CM) section included in the video content data and an audio indexing process. Usually, the time length of each CM section is set to one of several predetermined time lengths. In addition, there is a certain period of silence before and after the CM section. Therefore, in the CM section detection process, for example, audio data included in the video content data is analyzed, and a silent section having a certain length of time is detected. Then, a section including partial data that is sandwiched between two detected silent sections and has a time length equal to or greater than a certain value is detected as a CM section.

  In the audio indexing process, the audio data included in the video content data is analyzed to detect a music section in which the music flows and a talk section in which a person is talking. It is processing. In the audio indexing process, for example, the characteristics of the frequency spectrum of the audio data are analyzed, and the music section and the talk section are detected according to the characteristics of the frequency spectrum. Since the characteristics of the frequency spectrum corresponding to the music section and the characteristics of the frequency spectrum corresponding to the talk section are different from each other, the music section and the talk section can be detected by analyzing the characteristics of the frequency spectrum. In addition, a section having a frequency spectrum in which the characteristics of the frequency spectrum corresponding to the music section and the characteristics of the frequency spectrum corresponding to the talk section are mixed can be detected as a superimposed section in which the music section and the talk section overlap. .

  In the talk section detection process, for example, speaker segmentation technology or speaker clustering technology is used, and thereby switching of speakers is also detected. A period during which the same speaker (or the same speaker group) continuously talks is one talk section.

  Furthermore, the audio indexing process detects a cheer level for each partial data (data of a certain length of time) in the video content data, and detects a swell level for each partial data in the video content data. The swell level detection process is executed.

  The cheer level indicates the size of the cheer. Cheers are the sounds of many people. The sound of many human voices has a certain frequency spectrum distribution. In the cheer level detection process, the frequency spectrum of the audio data included in the video content data is analyzed, and the cheer level of each partial data is detected according to the analysis result of the frequency spectrum. The excitement level is a volume level of a section in which a volume level above a certain level is continuously generated for a certain length of time. For example, the volume level of a sound such as a relatively large applause and a loud laughter is the excitement level. In the swell level detection process, the volume distribution of the audio data included in the video content data is analyzed, and the swell level of each partial data is detected according to the analysis result.

  The memory 113A is used as a working memory for the video processor 113. A large amount of calculation is required to execute the indexing process (CM detection process, video indexing process, and audio indexing process). In the present embodiment, a video processor 113 that is a dedicated processor different from the CPU 101 is used as a back-end processor, and indexing processing is executed by the video processor 113. Therefore, the indexing process can be executed without increasing the load on the CPU 101. Since the CM detection process can be executed by analyzing the audio data as described above, the CM detection process will be treated as one process in the audio indexing process below.

  The sound controller 106 is a sound source device, and outputs audio data to be reproduced to the speakers 18A and 18B or the HDMI control circuit 3.

  The wireless LAN controller 114 is a wireless communication device that performs wireless communication of, for example, IEEE 802.11 standard. The IEEE 1394 controller 115 executes communication with an external device via an IEEE 1394 standard serial bus.

  The embedded controller / keyboard controller IC (EC / KBC) 116 is a one-chip microcomputer in which an embedded controller for power management and a keyboard controller for controlling the keyboard (KB) 13 and the touch pad 16 are integrated. . The embedded controller / keyboard controller IC (EC / KBC) 116 has a function of powering on / off the computer 10 in accordance with the operation of the power button 14 by the user. Further, the embedded controller / keyboard controller IC (EC / KBC) 116 has a function of executing communication with the remote control unit interface 20.

  The TV tuner 117 is a receiving device that receives broadcast program data broadcast by a television (TV) broadcast signal, and is connected to the antenna terminal 19. The TV tuner 117 is realized as a digital TV tuner capable of receiving digital broadcast program data such as terrestrial digital TV broadcast. The TV tuner 117 also has a function of capturing video data input from an external device.

  Next, the face image list display function executed by the TV application program 202 will be described with reference to FIG.

  As described above, the indexing process (video indexing process and audio indexing process) for video content data such as broadcast program data is executed by the video processor 113 functioning as an indexing processing unit.

  Under the control of the TV application program 202, the video processor 113 executes an indexing process on video content data such as recorded broadcast program data designated by the user, for example. The video processor 113 can also execute an indexing process on the broadcast program data in parallel with a recording process for storing the broadcast program data received by the TV tuner 117 in the HDD 111.

  In the video indexing process (also referred to as face image indexing process), the video processor 113 analyzes moving image data included in the video content data in units of frames. Then, the video processor 113 extracts a human face image from each of a plurality of frames constituting the moving image data, and outputs time stamp information indicating a time point at which each extracted face image appears in the video content data. . As the time stamp information corresponding to each face image, the elapsed time from the start of the video content data to the appearance of the face image, the frame number from which the face image is extracted, or the like can be used.

  Furthermore, the video processor 113 also outputs the size (resolution) of each extracted face image. The face detection result data (face image, time stamp information TS, and size) output from the video processor 113 is stored as face image indexing information in the database 111A. The database 111A is a storage area for storing indexing data prepared in the HDD 111.

  In this video indexing process, the video processor 113 also executes a thumbnail image acquisition process of the thumbnail image display function in parallel with the face image extraction process of the face image list display function. A thumbnail image is an image (reduced image) corresponding to each of a plurality of frames extracted from video content data at regular time intervals, for example. That is, the video processor 113 sequentially extracts frames, for example, at predetermined equal time intervals from the video content data regardless of whether or not the frame includes a face image, and images (thumbnail images) corresponding to the extracted frames. And time stamp information TS indicating the time when the thumbnail image appears. Thumbnail image acquisition result data (thumbnail and time stamp information TS) output from the video processor 113 is also stored as thumbnail indexing information in the database 111A.

  In the audio indexing process, the video processor 113 analyzes audio data included in the video content and detects a plurality of types of attribute sections (CM section, music section, and talk section) included in the video content data. Then, section attribute information indicating the start time and end time of each detected attribute section is output. This section attribute information is stored in the database 111A as attribute detection result information. Further, in the audio indexing process, the video processor 113 also executes the cheering level detection process and the excitement level detection process described above. The result of the cheer level detection process and the result of the excitement level detection process are also stored in the database 111A as part of the attribute detection result information.

  As shown in FIG. 4, the attribute detection result information (section attribute information) includes, for example, a CM section table, a music section table, a talk section table, and a cheer / climax table.

  The CM section table stores CM section attribute information indicating the start time and end time of the detected CM section. When there are a plurality of CM sections in the sequence from the start position to the end position of the video content data, CM section attribute information corresponding to each of the plurality of CM sections is stored in the CM section table. In the CM section table, start time information and end time information indicating the start time and end time of the CM section are stored for each detected CM section.

  The music section table stores music section attribute information indicating the start time and end time of the detected music section. When there are a plurality of music sections in the sequence from the start position to the end position of the video content data, music section attribute information corresponding to each of the plurality of music sections is stored in the music section table. In the music section table, start time information and end time information indicating the start time and end time of the music section are stored for each detected music section.

  The talk section table stores music section attribute information indicating the start time and end time of the detected talk section. When there are a plurality of talk sections in the sequence from the start position to the end position of the video content data, talk section attribute information corresponding to each of the plurality of talk sections is stored in the talk section table. In the talk section table, start time information and end time information indicating the start time and end time of the talk section are stored for each detected talk section.

  The cheer / climax table stores cheer levels and excitement levels in each partial data (time segments T1, T2, T3,...) Of a certain time length in the video content data.

  As shown in FIG. 3, the TV application program 202 includes a face image list display processing unit 301 for executing a face image list display function. The face image list display processing unit 301 is realized as an indexing viewer program, for example, and uses indexing information (face image indexing information, thumbnail indexing information, section attribute information, and the like) stored in the database 111A. Displays an indexing view screen for an overview of the content data.

  Specifically, the face image list display processing unit 301 reads face image indexing information (face image, time stamp information TS, and size) from the database 111A, and uses the face image indexing information to generate video content data. A list of face images of people appearing is displayed on a two-dimensional display area (hereinafter referred to as a face thumbnail display area) on the indexing view screen. In this case, the face image list display processing unit 301 divides the total time length of the video content data into a plurality of time zones, for example, at regular intervals, and for each time zone, extracts the face image from the extracted face images. A predetermined number of face images appearing on the screen are selected. Then, the face image list display processing unit 301 displays the selected predetermined number of face images side by side for each time period.

  That is, the two-dimensional face thumbnail display area includes a plurality of face image display areas arranged in a matrix including a plurality of rows and a plurality of columns. A plurality of time zones constituting the total time length of the video content data are assigned to each of the plurality of columns. Specifically, for example, in each of the plurality of columns, a plurality of time zones having the same time length obtained by dividing the total time length of the video content data at equal intervals by the number of the plurality of columns. Are assigned to each. Of course, the time zone allocated to each column does not necessarily have the same time length.

  The face image list display processing unit 301, based on the time stamp information TS corresponding to each face image, on the face image display area for the number of rows belonging to each column, faces belonging to the time zone assigned to each column. For example, the images are arranged and displayed in the order of appearance frequency order (face image detection time length order). In this case, for example, face images corresponding to the number of lines are selected in descending order of appearance frequency (appearance frequency) from the face images belonging to the time zone assigned to each column, and the selected face images are arranged in the appearance frequency order. They are arranged side by side from top to bottom. Of course, instead of the appearance frequency order, the face images that appear in the time zone assigned to each column may be displayed in the order of appearance.

  With this face image list display function, it is possible to easily show to the user which person appears in which time zone in the entire video content data. A specific configuration example of the face thumbnail display area will be described later with reference to FIG.

  The face image list display processing unit 301 also executes a thumbnail image display function. The face image list display processing unit 301 reads the thumbnail indexing information (thumbnail and time stamp information TS) from the database 111A, and uses the thumbnail indexing information to convert each thumbnail image below or above the face thumbnail display area. On the thumbnail display area (hereinafter referred to as “jabal thumbnail display area”) arranged on one side, the thumbnail images are displayed in a line in order of appearance time.

  Depending on the video content data, there may be a time period when the face image does not appear. Therefore, by displaying not only the face thumbnail display area but also the loose thumbnail display area on the indexing view screen, the contents of the video content data in that time period can be presented to the user even during the time period when the face image does not appear. Can do.

  Further, the face image list display processing unit 301 reads section attribute information (CM section attribute information, music section attribute information, talk section attribute information) from the database 111A, and ends the video content data from the start position according to the section attribute information. A face thumbnail display area on the indexing view screen includes a section bar including a bar area indicating the position (section from the start time to the end time) of each attribute section (CM section, music section, talk section) in the sequence up to the position. Is displayed on one of the lower side and the upper side. This section bar includes, for example, three sub-section bars, that is, a CM section bar indicating the position of the CM section, a music section bar indicating the position of the music section, and a talk section bar indicating the position of the talk section.

  When a plurality of CM sections exist in the sequence, a plurality of bar areas indicating the positions of the plurality of CM sections are displayed in the CM section bar. Similarly, when there are a plurality of music sections in the sequence, a plurality of bar areas indicating the positions of the plurality of music sections are displayed in the music section bar, and there are a plurality of talk sections in the sequence. A plurality of bar areas indicating positions of the plurality of talk sections are displayed in the talk section bar. In the following, each of a plurality of CM sections distributed in the sequence is referred to as a partial CM section, and each of a plurality of music sections distributed in the sequence is referred to as a partial music section, and is distributed in the sequence. Each of the plurality of talk sections existing in this manner is referred to as a partial talk section.

  By displaying a section bar on the indexing view screen, it is possible to determine in which attribute section (CM section, music section, talk section) each face image in the face thumbnail display area is a face image of a person appearing. It can be presented to the user. Therefore, the user can find the data position where playback should start from the entire video content data in consideration of the characters and the section attributes.

  Further, the face image list display processing unit 301 reads cheer level information and excitement level information from the database 111A, and in accordance with the cheer level information and excitement level information, the cheer level in the sequence from the start position to the end position of the video content data And a graph indicating the change of the swell level, respectively, are displayed on the level display area arranged on the lower side or the upper side of the face thumbnail display area on the indexing view screen.

  By looking at this level display area, the user is presented in which part of the video content data has a large cheering area and in which part of the video content data has a large swell. be able to. The user can find a data position where reproduction should be started from the entire video content data in consideration of the characters, section attributes, cheer level, and excitement level.

  Next, the functional configuration of the TV application program 202 will be described with reference to FIG.

  The TV application program 202 includes a recording processing unit 401, an indexing control unit 402, a reproduction processing unit 403, and the like in addition to the face image list display processing unit 301 described above.

  The recording processing unit 401 executes a recording process of recording broadcast program data received by the TV tuner 117 or video data input from an external device in the HDD 111. Further, the recording processing unit 401 receives broadcast program data designated by recording reservation information (channel number, date and time) preset by the user using the TV tuner 117, and reserves to record the broadcast program data in the HDD 111. Recording processing is also executed.

  The indexing control unit 402 controls the video processor (indexing processing unit) 113 to cause the video processor 113 to execute indexing processing (video indexing processing and audio indexing processing). The user can specify whether or not to perform the indexing process for each broadcast program data to be recorded. For example, for broadcast program data to be recorded for which execution of the indexing process is instructed, the indexing process is automatically started after the broadcast program data is recorded in the HDD 111. The user can also specify video content data to be indexed from video content data already stored in the HDD 111.

  The playback processing unit 403 executes processing for playing back each video content data stored in the HDD 111. In addition, the playback processing unit 403 displays the selected face image when a playback instruction event is input by a user operation while one face image in the face image list of a certain video content data is selected. It has a function of starting playback of video content data from a time point before a predetermined time.

  By selectively specifying a plurality of attribute sections, the user can determine the playback position of the video content data while viewing the face image list focusing on various attributes.

  Next, a specific configuration of the indexing view screen will be described with reference to FIGS.

  FIG. 6 shows an example of an indexing view screen displayed on the LCD 17 by the face image list display processing unit 301. This indexing view screen is a screen obtained by indexing certain video content data (for example, broadcast program data). On the indexing view screen, the above-mentioned face thumbnail display area for displaying a list of face images, the above-mentioned level display area, the above-mentioned section bar, and the above-mentioned for displaying a list of thumbnail images in a loose format. And a thumbnail display area.

  Here, the loose format is a display format in which a selected thumbnail image is displayed in a normal size (full size), and the other thumbnail images are displayed with a reduced size in the horizontal direction. In FIG. 6, the larger the distance from the selected thumbnail image, the smaller the horizontal size.

  In the level display area, a graph indicating a change in the cheer level (a graph indicated by a dark color in FIG. 6) and a graph indicating a change in the cheer level (a graph indicated by a light color in FIG. 6) are displayed. .

  The section bar includes a talk section bar, a music section bar, and a CM section bar. In the CM section bar, a bar area (black belt-like area in FIG. 6) is displayed at the position of each CM section (each partial CM section). In the music section bar, a bar area (a cross-hatched band-shaped area in FIG. 6) is displayed at the position of each music section (each partial music section). In the talk section bar, a bar area (hatched band area in FIG. 6) is displayed at the position of each talk section (each partial talk section). The user can select any one of the talk section bar, the music section bar, and the CM section bar by operating the buttons on the remote control unit and the cursor keys such as up / down / left / right. The user can also select one bar area in the selected section bar by operating the buttons of the remote control unit and the cursor keys such as up / down / left / right.

  FIG. 7 shows a configuration example of the face thumbnail display area.

  The face thumbnail display area includes a plurality of face image display areas arranged in a matrix including a plurality of rows and a plurality of columns. In FIG. 7, the face thumbnail display area is composed of 6 rows × 16 columns. The number of face image display areas included in the face thumbnail display area is 96.

  Each of the columns 1 to 16 includes, for example, a plurality of time zones having the same time length T obtained by dividing the total time length of the video content data at equal intervals by the number of columns (= 16). Assigned to each.

  For example, if the total time length of the video content data is 2 hours, the 2 hours are divided into 16 time zones at equal intervals. In this case, the time length T of each time zone is 7.5 minutes. For example, column 1 is assigned the time zone from the beginning 0:00:00 to 0:07:30, and column 2 is assigned the time zone from 0:07:30 to 0:15:00. Column 3 is assigned a time zone from 0:15:00 to 0:22:30. The time length T of each time zone changes according to the total time length of the video content data.

  Of course, the length of the time zone assigned to each of the plurality of columns is not necessarily the same.

  The face image list display processing unit 301 uses the time stamp information corresponding to each face image extracted by the video processor 113 to display the time allocated to each column on the six face image display areas in each column. For example, the face images belonging to the band are displayed side by side in the above-mentioned frequency order. In this case, the face image list display processing unit 301 selects face images corresponding to the number of lines (six) from among the face images belonging to the time zone assigned to the display processing target column, and corresponds to the selected number of lines. Display face images side by side.

  In this way, the face thumbnail display area uses a time axis with the left end position (1, 1) as a base point and the right end position (6, 16) as the end of video content data.

  The size of the face image displayed in each face image display area of the face thumbnail display area can be selected from “large”, “medium”, and “small”. The number of rows and columns is changed according to the size of the face image selected by the user. The relationship between the size of the face image and the number of rows and columns is as follows.

(1) “Large”; 3 rows x 8 columns (2) “Medium”; 6 rows x 16 columns (3) “Small”: 10 rows x 24 columns “Large” Each face image is displayed with a size of 180 × 180 pixels, for example. In the case of “medium”, each face image is displayed with a size of 90 × 90 pixels, for example. In the case of “small”, each face image is displayed with a size of 60 × 60 pixels, for example. The default face image size is set to “medium”, for example.

  Each face image in the face thumbnail display area is set to one of two states, a “normal” state that is not selected and a “focus” state that is selected. The size of the face image in the “focus” state is set larger than the size in the “standard” state (180 × 180, 90 × 90, or 60 × 60). FIG. 7 shows a case where the face image at coordinates (5, 12) is in the “focus” state.

  The number of thumbnail images displayed in the loose thumbnail display area is set to, for example, 240, 144, 96, or 48 according to the user setting. The default is 240 sheets, for example.

  The thumbnail image is set to one of two states, a “normal” state that is not selected and a “focus” state that is selected. The thumbnail image in the “focus” state is displayed in a larger size than the other thumbnail images.

  Next, the relationship between the face thumbnail display area and the jagged thumbnail display area will be described with reference to FIG.

  A set of face image display areas belonging to the same column, that is, each column in the face thumbnail display area is referred to as a “large section”. Further, the “large section” further divided is referred to as “small section”. The number of small sections included in one large section is given by a quotient obtained by dividing the number of thumbnail images displayed in the jagged thumbnail display area by the number of columns in the face thumbnail display area. For example, if the face thumbnail display area is 6 rows × 16 columns and the number of thumbnail images displayed in the loose thumbnail display area is 240, the number of small sections included in one large section is 15 (= 240 ÷ 16). One large section includes 15 small sections. In other words, 15 thumbnail images belong to the time zone corresponding to one large section.

  In addition, the face image list display processing unit 301 displays a vertically long bar connecting the large section and the thumbnail image. This vertically long bar is used to indicate which small section of the 15 small sections included in the large section corresponds to the small section corresponding to a certain thumbnail image. The vertically long bar is displayed at the position of the small section corresponding to the thumbnail image among the 15 small sections included in the large section. For example, if a certain thumbnail image is the top image of 15 thumbnail images belonging to the time zone corresponding to a large section, that is, the image corresponding to the top small section in the large section, the thumbnail image Are connected to the left end of the large section by a vertically long bar. Also, for example, if a certain thumbnail image is a terminal image of 15 thumbnail images belonging to a time zone corresponding to a certain large section, that is, an image corresponding to a small end section in the large section, The thumbnail image is connected to the right end of the large section by a vertically long bar.

  Further, the face image list display processing unit 301 displays time information indicating the time point when the thumbnail image appears on the indexing view screen based on the time stamp information of the thumbnail image.

  The “change current position” button is an operation button for changing a thumbnail image to be selected. When the user operates the left cursor key or the right cursor key while the “current position change” button is focused, the thumbnail image to be selected moves to the left or right, for example, in units of one small section.

  When a face image on the face thumbnail display area is selected by the user on the indexing view screen including the face thumbnail display area and the bellows thumbnail display area having such a relationship, the face image list display processing unit 301 The thumbnail image display control is performed in the loose thumbnail display area in order to identify the small section including the time indicated by the time stamp information of the face image and display the thumbnail image corresponding to the small section in the normal size.

  That is, as shown in FIG. 9, when a face image (a1) is selected by the user, a thumbnail image (a2) in a time zone in which the person of the face image appears is automatically selected in conjunction with it. Will be displayed.

  On the other hand, when a thumbnail image on the loose thumbnail display area is selected by the user on the indexing view screen, the face image list display processing unit 301 displays the time in the small section corresponding to the thumbnail image as the time stamp information. Are detected, and all the detected face images are highlighted so as to be distinguishable from other face images.

  That is, as shown in FIG. 10, when a thumbnail image (b1) is selected by the user, a face image of a person appearing in a small section corresponding to the thumbnail image is automatically selected in conjunction with it. Is highlighted (b2).

  Therefore, first, the user can confirm the thumbnail image in the time zone (scene) in which the person of the face image appears on the loose thumbnail display area by selecting the face image on the face thumbnail display area. Second, by selecting a thumbnail image on the rose thumbnail display area, the face image of a person appearing in the time zone corresponding to the thumbnail image can be confirmed on the face thumbnail display area.

  After confirming this, if the user indicates the start of playback of the video content data with the face image selected in the face thumbnail display area, a predetermined time before the time indicated by the time stamp information of the face image. The playback of the video content data can be started from the point of time, and if the playback of the video content data is instructed while the thumbnail image on the jagged thumbnail display area is selected, the start position (time point) of the subsection corresponding to the thumbnail image ) Can start reproduction of the video content data.

  In this way, in the computer 10 in which the face thumbnail display area and the jagged thumbnail display area are displayed and controlled in conjunction with each other, an image of a scene in which a desired person appears can be presented according to a simple operation. A face image of a person appearing in a scene can be presented according to a simple operation.

  By the way, each face image displayed in the face thumbnail display area has the same position and size of the face in the image as shown in FIG. That is, the video processor 113 has a function of appropriately determining a cutout range when a face area is cut out from each frame of video content data.

  Here, for example, a case where the face region is located at the end of the frame is assumed. In this case, if the cutout range of the face area is determined so that the face position and size in the image are constant, the cutout range will protrude from the frame. Therefore, in the computer 10, the video processor 113 has an adjustment function when the determined cutout range extends outside the frame. Hereinafter, this point will be described in detail.

  FIG. 11 is a diagram illustrating a first example of the adjustment function provided in the video processor 113 of the computer 10.

  In the figure, the hatched portions indicated by symbol a are portions that have been adjusted by the video processor 113. For example, the face image in the first row and the fifteenth column is a face region located at the upper right end of the frame. It is cut out. The principle of face image extraction by the video processor 113 from the frame in which the face area is located at the end will be described with reference to FIGS.

  FIG. 12 is a diagram illustrating the principle of normal face image extraction. When the video processor 113 detects a face area, the video processor 113 determines a cutout range (broken frame) so that the position and size of the face are constant. Then, the face image is extracted by enlarging / reducing the image within the range to match the display size in the face thumbnail display area. On the other hand, FIG. 13 is a diagram illustrating the principle of face image extraction when the determined cutout range extends beyond the frame. As illustrated in FIG. 13A, the determined cutout range extends beyond the frame. In this case, as shown in FIG. 13B, the video processor 113 masks a defect portion outside the frame with a predetermined color in addition to taking out the image within the frame, thereby reducing the position and size of the face. Realize face image extraction that is constant. As the masking color, a color designated in advance may be used. Alternatively, colors around the face outline may be collected and used.

  Next, the procedure of face image extraction processing executed by the video processor 113 will be described with reference to the flowchart of FIG.

  First, the video processor 113 searches for a face region within the processing target frame (step S101). If a face area is detected (YES in step S102), the video processor 113 checks whether or not the range to be clipped is within the frame (step S103). If it is within the frame (YES in step S103), the video processor 113 executes the segmentation as it is (step S104).

  On the other hand, if not within the frame (NO in step S103), the video processor 113 cuts out an area in the frame and supplements the other area by masking (step S105). Then, the video processor 113 stores the cut face image in the database (step S106).

  After the completion of storing the extracted face image in the database or when no face area is detected (NO in step S102), the video processor 113 checks whether there is a remaining frame (step S107). If it exists (YES in step S107), the process from step S101 is repeated, and if it does not exist (NO in step S107), this process ends.

  Next, the second extraction principle of the face image by the video processor 113 from the frame where the face area is located at the end will be described with reference to FIGS. 15 to 18.

  In the above description, the adjustment method for masking a defect portion generated in order to make the position and size of the face in the image constant has been described. Here, a certain amount of face position deviation and face size variation are described. An adjustment method for allowing the above will be described.

  As in the case shown in FIG. 13A, as shown in FIG. 15, when the determined cutout range protrudes from the frame, the video processor 113 sets the cutout range so that the cutout range fits in the frame. Move. In this case, the face position is slightly shifted to the upper right as compared with other face images, but the face image can be extracted only from the image within the frame.

  On the other hand, FIG. 16 shows a case where such movement cannot be performed. In the case shown in FIG. 16A, even if the cutout region is moved in the upper left direction, the upper left portion protrudes outside the frame at the movement destination. Further, in the case shown in FIG. 16B, since it protrudes in both the upper and lower directions, it cannot be moved in the first place.

  In such a case, the video processor 113 reduces the cutout range so that the cutout range fits within the frame. More specifically, in the case shown in FIG. 16A, the video processor 113, as shown in FIG. 17, fixes the upper left corner of the cutout area and fixes the lower right edge of the cutout area. The part is reduced so as to be directed to the center on the diagonal, and the cutout range is accommodated in the frame. Similarly, in the case shown in FIG. 16B, as shown in FIG. 18, the video processor 113 reduces both the upper left end and the lower right end of the cutout region so as to be directed to the diagonal center, Fit the cutout area in the frame.

  In this case, the size of the face is slightly different (larger) than the other face images, but the face image can be extracted using only the image within the frame.

  FIG. 19 is a flowchart showing a second procedure of face image extraction processing executed by the video processor 113.

  First, the video processor 113 searches for a face region within the processing target frame (step S201). If a face area is detected (YES in step S202), the video processor 113 checks whether or not the range to be clipped is within the frame (step S203).

  If the range to be cut out does not fit in the frame (NO in step S203), the video processor 113 moves or reduces the range to fit in the frame (step S204). The video processor 113 performs face segmentation (step S205) and stores the face image in the database (step S206).

  After completion of storing the extracted face image in the database or when no face area is detected (NO in step S202), the video processor 113 checks whether there is a remaining frame (step S207). If it exists (YES in step S207), the process from step S201 is repeated, and if it does not exist (NO in step S207), this process ends.

  As described above, in the present embodiment, it is possible to appropriately control the extraction of a human face image from video content data.

  Note that a user interface function may be provided for the user to set which of adjustment by masking and adjustment by movement / reduction is applied.

  In this embodiment, the example in which the indexing process including face image extraction is performed by the video processor 113 has been described. However, for example, the TV application program 202 may be provided with a function of executing the indexing process. In this case, the indexing process including face image extraction is executed by the CPU 101 under the control of the TV application program 202.

  Furthermore, the electronic device according to the present embodiment can be realized not only by the computer 10 but also by various consumer electronic devices such as an HDD recorder, a DVD recorder, and a television device. In this case, the function of the TV application program 202 can be realized by hardware such as a DSP or a microcomputer.

  That is, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine a component suitably in different embodiment.

  DESCRIPTION OF SYMBOLS 10 ... Electronic device (computer), 111A ... Database, 113 ... Video processor, 117 ... TV tuner, 301 ... Face image list display process part, 402 ... Indexing control part, 403 ... Reproduction process part.

Claims (17)

An electronic device for extracting a human face image from video content,
Detecting means for determining a cutout range for cutting out the detected face image when a face image is included in the frame of the video content;
An adjustment unit that adjusts the first face image included in the cutout range and outputs a second face image when the cutout range extends outside the frame;
Extraction means for outputting the second face image and time stamp information corresponding to a point in time when the first face image appears in the video content;
I have a,
The time stamp information is an electronic device used for displaying the second face image in correspondence with a point in time when the first face image appears in the video content .   When the cutout range is included in a frame, the extraction unit outputs a first face image included in the cutout range and time stamp information corresponding to a point in time when the first face image appears in the video content. The electronic device according to claim 1. And a display unit configured to display at least one of a plurality of face images output by the extraction unit side by side corresponding to the time point when each face image appears in the video content using the time stamp information. 1. The electronic device according to 1 .   When displaying a plurality of face images output by the extraction means, and when the time point at which each face image appears in the video content is within the first period, the display means The electronic device according to claim 3, wherein at least one of a plurality of face images output by the extraction unit is displayed side by side in accordance with the appearance frequency of a person corresponding to the image.   The electronic device according to claim 1, wherein the adjusting unit masks a defect portion outside the frame with a predetermined color. The electronic device according to claim 5 , wherein the adjustment unit performs masking with a color around a face outline in a frame.   The electronic device according to claim 1, wherein the adjustment unit moves the cutout range so that the cutout range is within a frame.   The electronic device according to claim 1, wherein the adjustment unit reduces the cutout range so that the cutout range is within a frame. A face image extraction control method in an electronic device for extracting a person's face image from video content,
When a face image is included in the frame of the video content, a cutout range for cutting out the detected face image is determined,
When the cutout range extends outside the frame, the first face image included in the cutout range is adjusted and a second face image is output,
Outputting a second face image, and the time stamp information of the first face image corresponds to the time that appears in the video content,
The time stamp information is a face image extraction control method used for displaying the second face image in correspondence with a point in time when the first face image appears in the video content . A computer for extracting human face images from video content,
Detecting means for determining a cutout range for cutting out the detected face image when a face image is included in the frame of the video content;
Adjusting means for adjusting the first face image included in the cutout range and outputting the second face image when the cutout range extends outside the frame;
Extraction means for outputting the second face image and time stamp information corresponding to a point in time when the first face image appears in the video content;
To function as,
The time stamp information is a program used for displaying the second face image in correspondence with a point in time when the first face image appears in the video content . When the cutout range is included in a frame, the extraction unit outputs a first face image included in the cutout range and time stamp information corresponding to a point in time when the first face image appears in the video content. The program according to claim 10 . Display means for displaying at least one of a plurality of face images output by the extraction means in correspondence with a point in time when each face image appears in the video content using the time stamp information. The program according to claim 10 , which is made to function as: When displaying a plurality of face images output by the extraction means, and when the time point at which each face image appears in the video content is within the first period, the display means The program according to claim 12 , wherein at least one of a plurality of face images output by the extraction unit is displayed side by side in accordance with the appearance frequency of a person corresponding to the image. The program according to claim 10 , wherein the adjustment unit masks a missing portion outside the frame with a predetermined color. The program according to claim 14 , wherein the adjusting means masks with a color around a face outline in a frame. The program according to claim 10 , wherein the adjustment unit moves the cutout range so that the cutout range is within a frame. The program according to claim 10 , wherein the adjustment unit reduces the cutout range so that the cutout range is within a frame.

Images