JP2012095334A - Electronic apparatus, and face image display unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an electronic apparatus capable of showing what person appears in what degree of frequency of appearance at what time zone in entire video content data without reproducing the video content data.SOLUTION: A face image list display process part 301 displays a list of face images of entire video content data in a two-dimensional face thumbnail display area in sizes corresponding to a frequency of appearance of a person in each appearance time-zone.

Description

  The present invention relates to an electronic device and a face image display method for displaying a face image appearing in video content data.

  Generally, electronic devices such as a video recorder and a personal computer can record and reproduce various video content data such as television broadcast program data. In this case, a title name is added to each video content data stored in the electronic device, but it is not possible for the user to grasp what the content of each video content data is based on the title name alone. Have difficulty. For this reason, in order to grasp the content of the video content data, it is necessary to reproduce the video content data. However, reproduction of video content data with a long total time length requires a lot of time even when a fast-forward reproduction function or the like is used.

  Patent Document 1 discloses an apparatus having a character list display function. This apparatus has a function of displaying a face image of each character in the video content side by side as a list of characters in the video content.

JP 2001-309269 A

  However, simply displaying a list of characters as in Patent Document 1, the user cannot grasp in which time zone in the video content data the characters appear. For example, the user may desire to selectively reproduce only the time zone in which a person (or group of persons) interested in the user appears. Therefore, it is necessary to realize a new function for presenting which person appears in which time zone in the entire video content data. Furthermore, there is a desire to increase the number of displayed face images as much as possible.

  The present invention has been made in consideration of the above-described circumstances, and which person appears in what time zone and at what frequency in the entire video content data without reproducing the video content data. It is an object of the present invention to provide an electronic device and a face image display method capable of presenting an image.

  An electronic apparatus according to an example of the present invention includes a storage device that stores face image indexing information including a plurality of face images extracted from video content data, and a classification unit that classifies face images included in the face image indexing information. Calculating means for calculating the appearance frequency of each classified face image; and a face image list displaying means for displaying a list of each classified face image, wherein each classified face image is: And a face image list display means that is displayed in a size based on the appearance frequency.

The perspective view which shows the example of the external appearance of the electronic device which concerns on one Embodiment of this invention. 2 is an exemplary block diagram showing an example of the system configuration of the electronic apparatus of the embodiment. FIG. FIG. 3 is an exemplary block diagram for explaining a face image list display function of the electronic apparatus of the embodiment. 2 is an exemplary block diagram illustrating a functional configuration of a program used in the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing an example of an indexing view screen displayed on the display device by the electronic apparatus of the embodiment. FIG. The figure which shows the structural example of the face thumbnail display area contained in the indexing view screen of FIG. FIG. 7 is a diagram for explaining a relationship between a face thumbnail display area and a bellows thumbnail display area included in the indexing view screen of FIG. 6. The figure which shows the example of the arrangement | sequence format of the face image displayed on the large area of a face thumbnail display area. The figure which shows the example of the setting screen used in order for a user to select one arrangement | sequence form from the arrangement | sequence form shown to FIG. 8 (A)-FIG.8 (C). The figure which shows the example of the indexing view screen which displayed the face image by the arrangement | sequence form shown to FIG. 8 (A). The figure which shows the example of the face image enlargedly displayed when a mouse is moved over the face image in the face thumbnail display area. The figure which shows the example of the screen for setting the time until a user displays a face image. The figure which shows the example of the setting screen for displaying the original image from which the face image was cut out. 6 is an exemplary flowchart illustrating an example of a procedure of face image list display processing executed by the electronic apparatus of the embodiment. The figure which shows a mode that each face image was classified according to the category with the electronic device of the embodiment. 6 is an exemplary flowchart illustrating an example of the procedure of display processing executed by the electronic apparatus of the embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First, with reference to FIG. 1 and FIG. 2, the structure of the electronic device which concerns on one Embodiment of this invention is demonstrated. The electronic apparatus according to the present embodiment is realized by, for example, a notebook portable personal computer 10 that functions as an information processing apparatus.

  The personal computer 10 can record and reproduce video content data (audio visual content data) such as broadcast program data and video data input from an external device. That is, the personal computer 10 has a television (TV) function for viewing and recording broadcast program data broadcast by a television broadcast signal. This TV function is realized by, for example, a TV application program installed in the personal computer 10 in advance. The TV function also has a function of recording video data input from an external AV device, and a function of reproducing recorded video data and recorded broadcast program data.

  Furthermore, the personal computer 10 has a face image list display function for displaying a list of face images of persons appearing in video content data such as video data and broadcast program data stored in the personal computer 10. This face image list display function is implemented as one function in the TV function, for example. The face image list display function is one of video indexing functions for presenting an outline of video content data to the user. This face image list display function can present to the user which person appears in which time zone in the entire video content data. The face image list display function can also display a list of persons by paying attention to a predetermined attribute section included in the video content data.

  FIG. 1 is a perspective view of the computer 10 with the display unit opened. The computer 10 includes a computer main body 11 and a display unit 12. The display unit 12 incorporates a display device including a TFT-LCD (Thin Film Transistor Liquid Crystal Display) 17.

  The display unit 12 is attached to the computer main body 11 so as to be rotatable between an open position where the upper surface of the computer main body 11 is exposed and a closed position covering the upper surface of the computer main body 11. The computer main body 11 has a thin box-shaped housing, and has a keyboard 13 on its upper surface, a power button 14 for powering on / off the computer 10, an input operation panel 15, a touch pad 16, and a speaker. 18A, 18B, etc. are arranged.

  The input operation panel 15 is an input device that inputs an event corresponding to a pressed button, and includes a plurality of buttons for starting a plurality of functions. These button groups also include operation button groups for controlling TV functions (viewing, recording, and reproduction of recorded broadcast program data / video data). In addition, a remote control unit interface unit 20 for executing communication with a remote control unit for remotely controlling the TV function of the computer 10 is provided on the front surface of the computer main body 11. The remote control unit interface unit 20 includes an infrared signal receiving unit and the like.

  On the right side of the computer main body 11, for example, an antenna terminal 19 for TV broadcasting is provided. Further, on the back surface of the computer main body 11, for example, an external display connection terminal corresponding to the HDMI (high-definition multimedia interface) standard is provided. The external display connection terminal is used to output video data (moving image data) included in video content data such as broadcast program data to an external display.

  Next, the system configuration of the computer 10 will be described with reference to FIG.

  As shown in FIG. 2, the computer 10 includes a CPU 101, a north bridge 102, a main memory 103, a south bridge 104, a graphics processing unit (GPU) 105, a video memory (VRAM) 105A, a sound controller 106, a BIOS- ROM 109, LAN controller 110, hard disk drive (HDD) 111, DVD drive 112, video processor 113, memory 113A, wireless LAN controller 114, IEEE 1394 controller 115, embedded controller / keyboard controller IC (EC / KBC) 116, TV tuner 117 , And EEPROM 118 and the like.

  The CPU 101 is a processor that controls the operation of the computer 10 and executes various application programs such as an operating system (OS) 201 and a TV application program 202 that are loaded from the hard disk drive (HDD) 111 to the main memory 103. . The TV application program 202 is software for executing a TV function. The TV application program 202 includes a live reproduction process for viewing broadcast program data received by the TV tuner 117, a recording process for recording the received broadcast program data in the HDD 111, and broadcast program data / data recorded in the HDD 111. A reproduction process for reproducing video data is executed. The CPU 101 also executes a BIOS (Basic Input Output System) stored in the BIOS-ROM 109. The BIOS is a program for hardware control.

  The north bridge 102 is a bridge device that connects the local bus of the CPU 101 and the south bridge 104. The north bridge 102 also includes a memory controller that controls access to the main memory 103. The north bridge 102 also has a function of executing communication with the GPU 105 via a PCI EXPRESS standard serial bus or the like.

  The GPU 105 is a display controller that controls the LCD 17 used as a display monitor of the computer 10. A display signal generated by the GPU 105 is sent to the LCD 17. The GPU 105 can also send a digital video signal to the external display device 1 via the HDMI control circuit 3 and the HDMI terminal 2.

  The HDMI terminal 2 is the above-described external display connection terminal. The HDMI terminal 2 can send an uncompressed digital video signal and a digital audio signal to the external display device 1 such as a television with a single cable. The HDMI control circuit 3 is an interface for sending a digital video signal to the external display device 1 called an HDMI monitor via the HDMI terminal 2.

  The south bridge 104 controls each device on an LPC (Low Pin Count) bus and each device on a PCI (Peripheral Component Interconnect) bus. The south bridge 104 includes an IDE (Integrated Drive Electronics) controller for controlling the hard disk drive (HDD) 111 and the DVD drive 112. Further, the south bridge 104 has a function of executing communication with the sound controller 106.

  Furthermore, a video processor 113 is connected to the south bridge 104 via a PCI EXPRESS standard serial bus or the like.

  The video processor 113 is a processor that executes various processes related to the above-described video indexing. The video processor 113 functions as an indexing processing unit for executing a video indexing process. That is, in the video indexing process, the video processor 113 extracts a plurality of face images from the moving image data included in the video content data, and time indicating the time when each extracted face image appears in the video content data. Output stamp information, etc. The extraction of the face image is executed by, for example, a face detection process for detecting a face area from each frame of moving image data, a cutout process for cutting out the detected face area from the frame, and the like. The face area can be detected by, for example, analyzing an image feature of each frame and searching for an area having a feature similar to a face image feature sample prepared in advance. The face image feature sample is feature data obtained by statistically processing the face image features of a large number of persons.

  The memory 113A is used as a working memory for the video processor 113. In order to execute the indexing process (CM detection process, video indexing process), a large amount of calculation is required. In the present embodiment, a video processor 113 that is a dedicated processor different from the CPU 101 is used as a back-end processor, and indexing processing is executed by the video processor 113. Therefore, the indexing process can be executed without increasing the load on the CPU 101.

  The sound controller 106 is a sound source device, and outputs audio data to be reproduced to the speakers 18A and 18B or the HDMI control circuit 3.

  The wireless LAN controller 114 is a wireless communication device that performs wireless communication of, for example, IEEE 802.11 standard. The IEEE 1394 controller 115 executes communication with an external device via an IEEE 1394 standard serial bus.

  The embedded controller / keyboard controller IC (EC / KBC) 116 is a one-chip microcomputer in which an embedded controller for power management and a keyboard controller for controlling the keyboard (KB) 13 and the touch pad 16 are integrated. . The embedded controller / keyboard controller IC (EC / KBC) 116 has a function of powering on / off the computer 10 in accordance with the operation of the power button 14 by the user. Further, the embedded controller / keyboard controller IC (EC / KBC) 116 has a function of executing communication with the remote control unit interface 20.

  The TV tuner 117 is a receiving device that receives broadcast program data broadcast by a television (TV) broadcast signal, and is connected to the antenna terminal 19. The TV tuner 117 is realized as a digital TV tuner capable of receiving digital broadcast program data such as terrestrial digital TV broadcast. The TV tuner 117 also has a function of capturing video data input from an external device.

  Next, the face image list display function executed by the TV application program 202 will be described with reference to FIG.

  As described above, the indexing process (video indexing process) for video content data such as broadcast program data is executed by the video processor 113 functioning as an indexing processing unit.

  Under the control of the TV application program 202, the video processor 113 executes an indexing process on video content data such as recorded broadcast program data designated by the user, for example. The video processor 113 can also execute an indexing process on the broadcast program data in parallel with a recording process for storing the broadcast program data received by the TV tuner 117 in the HDD 111.

  In the video indexing process (also referred to as face image indexing process), the video processor 113 analyzes moving image data included in the video content data in units of frames. Then, the video processor 113 extracts a human face image from each of a plurality of frames constituting the moving image data, and outputs time stamp information indicating a time point at which each extracted face image appears in the video content data. . As the time stamp information corresponding to each face image, the elapsed time from the start of the video content data to the appearance of the face image, the frame number from which the face image is extracted, or the like can be used.

  Furthermore, the video processor 113 also outputs the size (resolution) of each extracted face image. The face detection result data (face image, time stamp information TS, and size) output from the video processor 113 is stored as face image indexing information in the database 111A. The database 111A is a storage area for storing indexing data prepared in the HDD 111.

  Further, in the video indexing process, the video processor 113 also executes a thumbnail image acquisition process in parallel with the process of extracting the face image. A thumbnail image is an image (reduced image) corresponding to each of a plurality of frames extracted from video content data at regular time intervals, for example. That is, the video processor 113 sequentially extracts frames, for example, at predetermined equal time intervals from the video content data regardless of whether or not the frame includes a face image, and images (thumbnail images) corresponding to the extracted frames. And time stamp information TS indicating the time when the thumbnail image appears. Thumbnail image acquisition result data (thumbnail and time stamp information TS) output from the video processor 113 is also stored as thumbnail indexing information in the database 111A.

  As shown in FIG. 3, the TV application program 202 includes a face image list display processing unit 301 for executing a face image list display function. The face image list display processing unit 301 is realized as an indexing viewer program, for example, and uses video index data (face image indexing information, thumbnail indexing information, etc.) stored in the database 111A to outline video content data. Display the indexing view screen for a bird's-eye view.

  Specifically, the face image list display processing unit 301 reads face image indexing information (face image, time stamp information TS, and size) from the database 111A, and uses the face image indexing information to generate video content data. A list of face images of people appearing is displayed on a two-dimensional display area (hereinafter referred to as a face thumbnail display area) on the indexing view screen. In this case, the face image list display processing unit 301 divides the total time length of the video content data into a plurality of time zones, for example, at regular intervals, and for each time zone, extracts the face image from the extracted face images. A predetermined number of face images appearing on the screen are selected. Then, the face image list display processing unit 301 displays the selected predetermined number of face images side by side for each time period.

  That is, the two-dimensional face thumbnail display area includes a plurality of face image display areas arranged in a matrix including a plurality of rows and a plurality of columns. A plurality of time zones constituting the total time length of the video content data are assigned to each of the plurality of columns. Specifically, for example, in each of the plurality of columns, a plurality of time zones having the same time length obtained by dividing the total time length of the video content data at equal intervals by the number of the plurality of columns. Are assigned to each. Of course, the time zone allocated to each column does not necessarily have the same time length.

  The face image list display processing unit 301, based on the time stamp information TS corresponding to each face image, on the face image display area for the number of rows belonging to each column, faces belonging to the time zone assigned to each column. For example, the images are arranged and displayed in the order of appearance frequency order (face image detection time length order). In this case, for example, face images corresponding to the number of lines are selected in descending order of appearance frequency (appearance frequency) from the face images belonging to the time zone assigned to each column, and the selected face images are arranged in the appearance frequency order. They are arranged side by side from top to bottom. Of course, instead of the appearance frequency order, the face images that appear in the time zone assigned to each column may be displayed in the order of appearance.

  With this face image list display function, it is possible to easily show to the user which person appears in which time zone in the entire video content data. A specific configuration example of the face thumbnail display area will be described later with reference to FIG.

  In addition, the face image list display processing unit 301 reads thumbnail indexing information (thumbnail and time stamp information TS) from the database 111A, and uses the thumbnail indexing information to display each thumbnail image below or above the face thumbnail display area. On the thumbnail display area arranged on one side (hereinafter referred to as the “jabal thumbnail display area”), the thumbnail images are displayed in a line in order of appearance time.

  Depending on the video content data, there may be a time period when the face image does not appear. Therefore, by displaying not only the face thumbnail display area but also the loose thumbnail display area on the indexing view screen, the contents of the video content data in that time period can be presented to the user even during the time period when the face image does not appear. Can do.

  Next, the functional configuration of the TV application program 202 will be described with reference to FIG.

  The TV application program 202 includes a recording processing unit 401, an indexing control unit 402, a reproduction processing unit 403, and the like in addition to the face image list display processing unit 301 described above.

  The recording processing unit 401 executes a recording process of recording broadcast program data received by the TV tuner 117 or video data input from an external device in the HDD 111. Further, the recording processing unit 401 receives broadcast program data designated by recording reservation information (channel number, date and time) preset by the user using the TV tuner 117, and reserves to record the broadcast program data in the HDD 111. Recording processing is also executed.

  The indexing control unit 402 controls the video processor (indexing processing unit) 113 to cause the video processor 113 to execute video indexing processing. The user can specify whether or not to perform the indexing process for each broadcast program data to be recorded. For example, for broadcast program data to be recorded for which execution of the indexing process is instructed, the indexing process is automatically started after the broadcast program data is recorded in the HDD 111. The user can also specify video content data to be indexed from video content data already stored in the HDD 111.

  The playback processing unit 403 executes processing for playing back each video content data stored in the HDD 111. In addition, the playback processing unit 403 displays the selected face image when a playback instruction event is input by a user operation while one face image in the face image list of a certain video content data is selected. It has a function of starting playback of video content data from a time point before a predetermined time.

  Note that the indexing process is not necessarily executed by the video processor 113. For example, the TV application program 202 may be provided with a function for executing the indexing process. In this case, the indexing process is executed by the CPU 101 under the control of the TV application program 202.

  Next, a specific configuration of the indexing view screen will be described with reference to FIGS.

  FIG. 5 shows an example of an indexing view screen displayed on the LCD 17 by the face image list display processing unit 301. This indexing view screen is a screen obtained by indexing certain video content data (for example, broadcast program data). The indexing view screen includes the above-described face thumbnail display area for displaying a list of face images, and the above-described jagged thumbnail display area for displaying a list of thumbnail images in a loose format.

  Here, the loose format is a display format in which a selected thumbnail image is displayed in a normal size (full size), and the other thumbnail images are displayed with a reduced size in the horizontal direction. In FIG. 5, the larger the distance from the selected thumbnail image, the smaller the horizontal size.

  FIG. 6 shows a configuration example of the face thumbnail display area.

  The face thumbnail display area includes a plurality of face image display areas arranged in a matrix including a plurality of rows and a plurality of columns. In FIG. 6, the face thumbnail display area is composed of 6 rows × 16 columns. The number of face image display areas included in the face thumbnail display area is 96.

  Each of the columns 1 to 16 includes, for example, a plurality of time zones having the same time length T obtained by dividing the total time length of the video content data at equal intervals by the number of columns (= 16). Assigned to each.

  For example, if the total time length of the video content data is 2 hours, the 2 hours are divided into 16 time zones at equal intervals. In this case, the time length T of each time zone is 7.5 minutes. For example, column 1 is assigned the time zone from the beginning 0:00:00 to 0:07:30, and column 2 is assigned the time zone from 0:07:30 to 0:15:00. Column 3 is assigned a time zone from 0:15:00 to 0:22:30. The time length T of each time zone changes according to the total time length of the video content data.

  Of course, the length of the time zone assigned to each of the plurality of columns is not necessarily the same.

  The face image list display processing unit 301 uses the time stamp information corresponding to each face image extracted by the video processor 113 to display the time allocated to each column on the six face image display areas in each column. For example, the face images belonging to the band are displayed side by side in the above-mentioned frequency order. In this case, the face image list display processing unit 301 selects face images corresponding to the number of lines (six) from among the face images belonging to the time zone assigned to the display processing target column, and corresponds to the selected number of lines. Display face images side by side.

  In this way, the face thumbnail display area uses a time axis with the left end position (1, 1) as a base point and the right end position (6, 16) as the end of video content data.

  The size of the face image displayed in each face image display area of the face thumbnail display area can be selected from “large”, “medium”, and “small”. The number of rows and columns is changed according to the size of the face image selected by the user. The relationship between the size of the face image and the number of rows and columns is as follows.

(1) “Large”; 3 rows x 8 columns (2) “Medium”; 6 rows x 16 columns (3) “Small”: 10 rows x 24 columns “Large” Each face image is displayed with a size of 180 × 180 pixels, for example. In the case of “medium”, each face image is displayed with a size of 90 × 90 pixels, for example. In the case of “small”, each face image is displayed with a size of 60 × 60 pixels, for example. The default face image size is set to “medium”, for example.

  Each face image in the face thumbnail display area is set to one of two states, a “normal” state that is not selected and a “focus” state that is selected. The size of the face image in the “focus” state is set larger than the size in the “standard” state (180 × 180, 90 × 90, or 60 × 60). FIG. 6 shows a case where the face image at the coordinates (1, 12) is in the “focus” state.

  The number of thumbnail images displayed in the loose thumbnail display area is set to, for example, 240, 144, 96, or 48 according to the user setting. The default is 240 sheets, for example.

  The thumbnail image is set to one of two states, a “normal” state that is not selected and a “focus” state that is selected. The thumbnail image in the “focus” state is displayed in a larger size than the other thumbnail images.

  Next, the relationship between the face thumbnail display area and the jagged thumbnail display area will be described with reference to FIG.

  A set of face image display areas belonging to the same column, that is, each column in the face thumbnail display area is referred to as a “large section”. Further, the “large section” further divided is referred to as “small section”. The number of small sections included in one large section is given by a quotient obtained by dividing the number of thumbnail images displayed in the jagged thumbnail display area by the number of columns in the face thumbnail display area. For example, if the face thumbnail display area is 6 rows × 16 columns and the number of thumbnail images displayed in the loose thumbnail display area is 240, the number of small sections included in one large section is 15 (= 240 ÷ 16). One large section includes 15 small sections. In other words, 15 thumbnail images belong to the time zone corresponding to one large section.

  When a certain thumbnail image on the jagged thumbnail display area is selected, the face image list display processing unit 301 selects a plurality of columns (a plurality of large sections in the face thumbnail display area based on the time stamp information of the selected thumbnail image). The column (large section) to which the time zone to which the selected thumbnail image belongs is assigned. The selected large section is a large section that includes the selected thumbnail image as a small section. Then, the face image list display processing unit 301 highlights the selected large section.

  As described above, when a thumbnail image on the jagged thumbnail display area is selected, a column (large section) to which the time zone to which the selected thumbnail image belongs is assigned from among a plurality of columns in the face thumbnail display area. Is automatically selected. Thereby, the user can identify which column (large section) in the face thumbnail display area corresponds to the selected thumbnail image. Furthermore, the vertically long bar allows the user to identify which thumbnail image the selected thumbnail image corresponds to in which column (large section).

  By the way, the size of the face image displayed in each face image display area of the face thumbnail display area can be set to change the size of the face image according to the appearance frequency.

  This setting can be further selected from a plurality of arrangement formats prepared in advance. For example, as shown in FIGS. 8A to 8C, the user can select one arrangement format from three arrangement formats. The array format shown in FIG. 8 shows one large section, and this array format is displayed in 8 columns. In FIGS. 8A to 8C, a rectangular area in each area is an area where a face image is displayed. The numbers that can be displayed in the buy rectangular area represent the order of appearance frequency.

  The arrangement format shown in FIG. 8A has a large face image area, a medium face image area, and a small face image area. In the large image area, one face image having the highest appearance frequency is displayed. In the middle face image area, four face images having the second to fifth appearance frequencies are displayed. In the small face image area, sixteen face images having the appearance frequency from the sixth to the 21st are displayed.

  The arrangement format shown in FIG. 8B has a large face image area and a medium face image area. In the large image area, two face images of the face having the highest appearance frequency and the second face are displayed. In the middle face image area, four face images having the third to sixth appearance frequencies are displayed.

  The arrangement format shown in FIG. 8C has a large face image area and a medium face image area. In the large image area, one face image having the highest appearance frequency is displayed. In the middle face image area, eight face images having the second to eighth appearance frequencies are displayed.

  FIG. 9 shows an example of a window for the user to select one array format from the three types of array formats shown in FIGS. 8 (A) to 8 (C). As shown in FIG. 9, three examples of the arrangement format of the large section are displayed. Radio buttons are provided above the respective array formats. As for the radio button, after selecting only one of the three buttons, the selected arrangement format is displayed by operating the OK button.

  FIG. 10 shows a display example of the array format shown in FIG. As shown in FIG. 10, face images are displayed in an array format similar to that shown in FIG. 8A in large sections of eight columns on the loose thumbnail display area. In each large section, face images are displayed in order from the face with the highest appearance frequency.

  Note that the images in the middle face image area and the small face image area are small and difficult to distinguish. However, in this apparatus, when a predetermined time elapses in a state where the pointer 501 is placed on the middle face image area and the small face image area and is still (so-called mouse over), a large face image, for example, a large face image is obtained. A face image 502 having the same size as the area image may be displayed in a pop-up beside the pointer 501 (FIG. 11).

  The user can set the time from when the mouse is over until the face image pops up. The setting window shown in FIG. 12 can be set by selecting the check box 511 and changing the number of seconds in the rectangular area 512.

  When the set time and the mouse over time elapse, an event is input to the face image list display processing unit 301, and the face image list display processing unit 301 performs enlarged display of the face image.

  Further, depending on the setting, it is possible to display an original image in which a face image is cut out after the mouse is over. This is effective in all the large image area, medium face image area, and small face image area. FIG. 13 shows an example of a setting window for displaying the original image from which the face image is cut out. As shown in FIG. 13, when the user selects one of the two radio buttons 521 and 522 that can be exclusively selected, the face image enlarged display and the cut-out original image display in the middle face image area and the small face image area are displayed. One is set.

  When the set time and the mouse over time elapse, an event is input to the face image list display processing unit 301, and the face image list display processing unit 301 displays the enlarged face image or the cut-out original image according to the setting. Do.

  Next, the procedure of the display process executed by the face image list display processing unit 301 will be described with reference to the flowchart of FIG.

  First, the face image list display processing unit 301 sorts the face images stored in the database 111A in the order of appearance times of the face images based on the time stamp information corresponding to the face images (step S101). Next, the face image list display processing unit 301 assigns to each column (large section) by referring to the designated by the user and equally dividing the total time length of the video content data to be indexed by the number of columns, for example. Calculate the time zone. The face image list display processing unit 301 then displays the number of face images displayed in a large section corresponding to the selected display format from among the face images belonging to the time zone assigned to the display processing target column. Is selected (step S102).

  The face image list display processing unit 301 preferentially selects a face image that frequently appears in the time zone from the face images belonging to the time zone assigned to the display processing target column. For example, the video processor 113 can execute a clustering process for classifying each face image of the same person into the same category based on the correlation of the feature data of each extracted face image. In this case, as shown in FIG. 15, each face image is classified for each person (that is, for each category) and stored in the database 111A. In FIG. 15, category 1 represents a set of face images of a certain person A, category 2 represents a set of face images of a certain person B, and category 3 represents a face image of a certain person B. A set of The face image list display processing unit 301 preferentially selects face images of people who appear frequently. Note that in a time zone in which only a face image of the same person continues to appear for a relatively long time, a plurality of different face images of the same person may be displayed on a column corresponding to the time zone.

  Next, the face image list display processing unit 301 displays the selected face images, for example, in order of appearance frequency in a plurality of face image display areas in the display processing target column (step S103). A face image having a higher appearance frequency is displayed in the upper face image display area.

  The processes in steps S102 and S103 are repeatedly executed while updating the numbers of the display process target columns until the processes for all the columns are completed (steps S104 and S105).

  When the processing for all the columns is completed (NO in step S104), the face image list display processing unit 301 converts each thumbnail image stored in the database 11A to a thumbnail thumbnail based on the time stamp information of each thumbnail image. The thumbnail images are displayed in a line in the order in which they appear on the display area (step S106).

  Next, the procedure of the reproduction process executed by the reproduction processing unit 403 will be described with reference to the flowchart in FIG.

  When the mouse is moved over a certain face image in response to a user operation and an event is input to the face image list display processing unit 301 (YES in step S301), the face image list display processing unit 301 It is determined whether or not the setting for performing enlarged display is performed (step S302). If the setting is to enlarge the display (YES in step S302), the face image list display processing unit 301 enlarges and displays the selected face image (step S303). If there is no setting for enlargement display (NO in step S302), that is, if the setting is to display a cut-out original image of the face image, the face image list display processing unit 301 displays the cut-out original image of the selected face image (step). S303).

  As described above, in the present embodiment, a list of face images is displayed for the entire video content data in a size corresponding to the appearance frequency for each appearance time zone. Therefore, it is possible to present to the user which person appears in what time zone in the entire video content data and how often without reproducing the video content data.

  In the present embodiment, the face image indexing information is generated by the video processor 113 functioning as an indexing processing unit. For example, when the broadcast program data includes indexing information corresponding to the broadcast program data in advance. Does not need to be indexed. Therefore, the face image list display function of the present embodiment can be realized only by the database 111A and the face image list display processing unit 301.

  Further, since all the face image list display processing procedures of the present embodiment can be realized by software, the same effects as those of the present embodiment can be obtained by introducing this software into a normal computer through a computer-readable storage medium. Can be easily realized.

  In addition, the electronic device of the present embodiment can be realized not only by the computer 10 but also by various consumer electronic devices such as an HDD recorder, a DVD recorder, and a television device. In this case, the function of the TV application program 202 can be realized by hardware such as a DSP or a microcomputer.

Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine suitably the component covering different embodiment.
The invention described in the scope of the claims at the time of the patent assessment of the original application of the present application will be added below. [1] A storage device that stores a plurality of face images extracted from video content data, time stamp information indicating when each face image appears in the video content data, and face image indexing information including the time image information.
Means for assigning a plurality of time zones to the video content data and extracting face images belonging to each time zone based on the time stamp information;
Means for classifying face images of the same person from the extracted face images;
Means for calculating the appearance frequency of each classified face image;
Face image list display means for displaying a list of a plurality of face images included in the face image indexing information in a face image display area within a two-dimensional display area, wherein the face image display area includes the plurality of time zones. The face image list display means includes a plurality of columns of time zone display areas corresponding to each time zone, and each face image displayed in each time zone display area is displayed in a size based on the appearance frequency. An electronic device comprising:
[2] The image processing apparatus further includes an indexing processing unit that extracts a plurality of face images from the video content data and outputs the time stamp information indicating a time point at which each extracted face image appears in the video content data. [1] The electronic device according to [1].
[3] The apparatus further comprises means for a user to specify a display format displayed in the face image display area from a plurality of display formats displayed in a size based on the appearance frequency. Or the electronic device of any one of [2].
[4] When the face image list display means receives an event for one face image in response to a user operation, the face image list display means enlarges the one face image. The electronic device according to any one of [1], [2], and [3], wherein the electronic device is displayed or an image from which the face image is extracted is displayed.
[5] The electronic device according to [4], further comprising setting means for setting a user operation to which the event is input.
[6] A pointer whose display position changes according to the user's operation is displayed.
The electronic device according to [5], wherein the setting unit sets a time during which the pointer is stationary on the face image.
[7] generating face image indexing information including a plurality of face images extracted from video content data and time stamp information indicating a time point at which each face image appears in the video content data;
A plurality of time zones are allocated to the video content data, and face images belonging to the time zones are respectively extracted based on the time stamp information.
Classify face images of the same person from the extracted face images belonging to the same time zone,
Calculate the appearance frequency of each classified face image,
A face image display method for displaying a list of a plurality of face images included in the face image indexing information in a face image display area in a two-dimensional display area,
The face image display area has a plurality of columns of time zone display areas corresponding to the plurality of time zones, and each face image displayed in each time zone display area has a size based on the appearance frequency. A face image display method, characterized in that the face image is displayed.
[8] A plurality of display formats to be displayed in a size based on the appearance frequency are prepared, and a face image is displayed in the display area for each time period according to the display format specified by the user. 7] The face image display method according to [7].
[9] When an event occurs for one face image in response to a user operation, the one face image is enlarged or displayed, or an image from which the face image is extracted is displayed. [7] The face image display method according to [7].
[10] A pointer whose display position changes according to the user's operation is displayed.
[9] The face image display method according to [9], wherein the event occurs when a time during which the pointer is stationary on the face image has passed a set time.

  DESCRIPTION OF SYMBOLS 10 ... Electronic device (computer), 111A ... Database, 113 ... Video processor, 117 ... TV tuner, 301 ... Face image list display process part, 402 ... Indexing control part, 403 ... Reproduction process part.

Claims (13)

A storage device for storing face image indexing information including a plurality of face images extracted from video content data;
Classification means for classifying the face image included in the face image indexing information;
Calculating means for calculating the appearance frequency of each classified face image;
Face image list display means for displaying a list of each classified face image, wherein each classified face image is displayed in a size based on the appearance frequency. An electronic device comprising: The face image indexing information includes time stamp information indicating when each face image appears in the video content data,
A plurality of time zones assigned to the video content data, and further comprising extraction means for extracting face images belonging to each time zone based on the time stamp information,
The calculating means calculates the appearance frequency of each classified face image for each of the plurality of time zones,
The face image list display means displays a list of the classified face images in the face image display area in the two-dimensional display area,
The face image display area includes a plurality of time zone display areas corresponding to the plurality of time zones, and the plurality of time zone display areas corresponding to a first time zone of the plurality of time zones. Each face image displayed in the first time zone display area is displayed in a size based on the appearance frequency in the first time zone,
The electronic device according to claim 1.   The electronic apparatus according to claim 1, further comprising an indexing processing unit that extracts a plurality of face images from the video content data.   The electronic apparatus according to claim 3, wherein the indexing processing unit outputs the time stamp information indicating a point in time when each extracted face image appears in the video content data.   5. The device according to claim 1, further comprising means for a user to specify a display format displayed in the face image display area from a plurality of display formats displayed in a size based on the appearance frequency. The electronic device of Claim 1.   The face image list display means enlarges and displays the one face image when an event for one face image is input in response to a user operation. The electronic device according to claim 1, wherein an image from which the face image is extracted is displayed.   The electronic apparatus according to claim 6, further comprising a setting unit configured to set a user operation for inputting the event. A pointer whose display position changes according to the user's operation is displayed,
The electronic device according to claim 7, wherein a time during which the pointer is stationary on the face image is set by the setting unit. Classifying face images included in face image indexing information including a plurality of face images extracted from video content data;
Calculate the appearance frequency of each classified face image,
A method of displaying a list of each classified face image, wherein each classified face image is displayed in a size based on the appearance frequency. The face image indexing information includes time stamp information indicating when each face image appears in the video content data,
A plurality of time zones are assigned to the video content data, and face images belonging to each time zone are extracted based on the time stamp information,
Calculating the appearance frequency of each classified face image for each of the plurality of time zones;
A list of each classified face image is displayed in a face image display area in a two-dimensional display area,
The face image display area has a plurality of columns of time zone display areas corresponding to the plurality of time zones,
Each face image displayed in the first time zone display area of the plurality of time zone display areas corresponding to the first time zone of the plurality of time zones is in the first time zone. Displayed in a size based on frequency of appearance,
The face image display method according to claim 9.   The face image display method according to claim 9, wherein a plurality of display formats that are displayed in a size based on the appearance frequency are prepared, and a face image is displayed according to a display format specified by a user.   10. When an event that occurs for one face image occurs in response to a user operation, the one face image is enlarged or displayed, or an image from which the face image is extracted is displayed. The face image display method described. A pointer whose display position changes according to the user's operation is displayed,
The face image display method according to claim 12, wherein the event occurs when a time during which the pointer is stationary on the face image has passed a set time.

Images