JP5317710B2 - Image processing apparatus, control method therefor, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To highly reliably record a position at which a face newly appears with a small processing load. <P>SOLUTION: An image processing apparatus includes a means for detecting a face from a moving images, a tracking means for tracking the face of a main object by tracking conditions including other than the position and size thereof, a holding means (S416) for holding the information of the position and size of the detected face, a first determination means (S411) for determining whether the face matches with the information of the position or the size held in the holding means when any face other than that of the main object is newly detected, and a recording control means (S407, S419) for executing control so as to record a face index specifying a frame included in a period during which it is determined that the new face is detected. The recording control means does not record the face index (S418) for a period of detecting the face for which the face index is already recorded in the other period of face detection among the faces determined as matching by the first determination means. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus that detects a face of a person from a moving image and records information about a frame in which the face is detected, a control method thereof, a program, and a recording medium.

  Conventionally, various GUIs (graphic user interfaces) have been proposed in order to improve operability related to reproduction of moving images.

  According to Patent Literature 1, a moving image playback device is disclosed in which a person appearing in moving image content and its appearance position can be presented to a user appropriately and easily. According to this reproducing apparatus, a list of face images indicating positions where a human face is newly detected in a moving image is displayed, and a moving image in a section corresponding to the face image selected by the user from the images is reproduced. be able to.

  FIG. 8 shows an example of display (hereinafter referred to as face index display) displaying a list of face images detected in a moving image during reproduction.

  A display screen 801 is a display screen on which face index display is performed, and displays a representative image, supplementary information, a list of detected face images, and the like for one moving image selected for performing face index display.

  A representative image 802 is a representative image of the entire currently selected moving image. For example, the first frame (or first I picture) of the moving image or a frame including the face detected first.

  The scene information 803 is information attached to the moving image, and displays the recording date and time and recording time of the moving image.

  Images 804 to 808 are images obtained by reducing the frames including the newly detected face in the moving image, and are displayed on the timeline 810 in time series. When the number of newly detected faces in the moving image is less than a predetermined number (here, 5), it is blank. When the number of faces newly detected in the moving image exceeds a predetermined number, it is possible to move to the next page by a user operation.

  A cursor 809 is a cursor for selecting a desired image from images displayed on the timeline 810, and is moved by a user operation. Then, the playback of the moving image can be started from the position of the image selected by the cursor. For example, in FIG. 8, the cursor 809 is placed on the image 806, and when a playback instruction is received from the user in this state, playback of a moving image can be started from the frame position of the image 806. Such face index display is performed based on face index information generated at the time of shooting a moving image. The face index information is a file recorded as header information of a moving image file or a separate file from the moving image file. In the face index information, information for specifying a frame of an image to be displayed on the timeline 810 (hereinafter referred to as a face index) is recorded. The face index is recorded by performing a face index recording request when a face detection process is performed for each frame by the imaging device at the time of shooting and it is determined that a new face has appeared.

  FIG. 9 shows the recording request timing of each face index included in the face index information used for conventional face index display.

  Periods 901, 903, 911 to 915, and 921 to 923 indicate periods during which the face detection circuit included in the imaging apparatus has detected a face. It is assumed that there are four seconds or more between the period 912 and the period 913, between the period 913 and the period 914, between the period 922 and the period 923, and all other periods are less than 4 seconds. The period 913 is a period of less than 1 second. This example is an example in which a maximum of three people are detected at the same time, and are assumed to be a main subject, face ID1, and face ID2 for the following explanation. It is assumed that the main subject is recognized as the main subject by a predetermined algorithm. The face to which the main subject is assigned is subjected to tracking processing by matching by luminance in the period 902, and the face detected in the periods 901 and 903 is detected in the period 903 although the face detection is once interrupted. Can be regarded as the face of the same person. However, matching by luminance is performed only on the main subject because of a heavy processing load. Therefore, tracking processing is not performed on faces other than the main subject, and faces assigned with face ID 1 and face ID 2 are the same face if the face detection period is interrupted even if the same ID is assigned. Is not limited. That is, the faces detected in the periods 911 to 915 to which the face ID 1 is assigned may be faces of different persons or the faces of the same person. Further, the faces detected in the periods 921 to 923 to which the face ID 2 is assigned may be faces of different persons, or may be faces of the same person. However, there are periods in which faces with different face IDs are detected at the same time. If it is detected at the same time, a plurality of IDs are not assigned to one person, so the faces of the persons are different for each face ID. That is, the face ID assigned to other than the main subject is merely an identifier for identifying a plurality of faces detected at the same time. Once face detection is interrupted, the assigned face ID expires, and when a new face is detected, the ID with the smallest number among the invalid face IDs is assigned. Such face ID management is performed using a face management table prepared in a temporary storage medium during shooting.

The face index number 950 indicates the timing of a face index recording request that is performed when a face is detected as shown in FIG. 9, and the face index number of each face index. A face index recording request is made each time a face to which face IDs 1 and 2 that are faces other than the main subject are assigned is newly detected. In this way, in the example of FIG. 9, eight face indexes of face index numbers 0 to 7 are recorded in the face index information. In the face index display at the time of reproduction, eight images corresponding to the face index numbers 0 to 7 are displayed on the timeline.
JP 2008-017041 A

  In the example of FIG. 9, even if the detection period is interrupted, the face assigned with the same ID may be the face of the same person, but each time it is newly detected even if it is the face of the same person The face index is recorded. For example, even if the face of the same person A is detected, the face detection is interrupted for a moment because the person A turns downward for a moment, and the face detection is performed again because the person A faces the camera again. The face index is recorded. For example, the detection start timing in the period 912 (timing for requesting the recording of the face index with the face index number 2) may be such a case because the time when the detection is interrupted from the period 911 is short. However, in such a case, it cannot be said that it is the timing when the person A newly appears from the viewpoint of the viewer, and it is difficult to think of the viewer's desire to start playback from this timing. In other words, at the timing of the conventional face index recording request, there is a possibility that the face of the same person with many face movements will appear in the face index display described above rather than a newly appearing person. In some cases, the search performance in the index display was impaired.

  Further, in the example of FIG. 9, the face index is recorded even at the timing of the period 913 in which the face can be detected only for a moment such as less than 1 second. For example, it is assumed that the face detected in the period 913 is the face of the person B. According to the face index in this period 913 (face index number is 3), an image showing the face of person B is displayed in the face index display. Therefore, viewing this image, the viewer selects the image with the face index number 3 in the hope of viewing the movie related to the person B, and starts reproduction from that position. However, the video in which the person B is actually shown is a short time of less than 1 second, and there is a high possibility that the viewer's desire to view the video related to the person B cannot be sufficiently satisfied. Such face index images also contribute to the loss of searchability in face index display.

  Further, since the face of the main subject is subjected to the tracking process, it is rarely determined that the face has appeared newly. Compared to this, it is more likely that a face other than the main subject is newly appeared by the amount not subjected to the tracking process. For this reason, although it is considered that the user is paying attention to the main subject, the face index display has many images representing scenes where faces other than the main subject appear.

  In view of the above problems, an object of the present invention is to provide an imaging apparatus that records a position where a face newly appears with a small processing load with higher reliability.

In order to achieve the above object, an image processing apparatus according to claim 1 of the present invention provides:
Face detection means for detecting a face from each frame of the video;
Of the faces detected by the face detection means, for the face determined as the main subject, a first period in which the main subject is detected by the face detection means and the main subject are detected in the video. A period after the first period, a second period that satisfies a first condition that includes conditions other than the position, size, and time of the face relating to the main object; and the main object is detected again by the face detection means. Information for specifying a frame for the first period is recorded as face index information for a period that is continuous with the third period, and information for specifying a frame for the third period is the face index information. As well as controlling not to record,
Among the faces detected by the face detection means, for a specific face other than the main subject, a first period in which the specific face is detected by the face detection means in the video, and the specific face A face that satisfies a second condition relating to at least one of the second period within a predetermined time after the detection of the detection and the position and size of the specific face detected in the first period is the face detection For a period continuous with the third period detected by the means, information specifying a frame for the first period is recorded as the face index information, and information for specifying a frame for the third period is as the face index information and said Rukoto which have a control means for controlling so as not to record.

  According to the present invention, it is possible to more reliably record a position where a new face appears with a small processing load.

  Preferred embodiments of the present invention will be described below with reference to the drawings.

<Configuration of digital video camera>
FIG. 1 is a block diagram showing a configuration of a digital video camera as an example of an imaging apparatus of the present invention.

  Reference numeral 101 denotes a lens unit, which is a correction unit that has a function of correcting an imaging position moved by movement of a fixed lens group for focusing, a variable power lens group, a diaphragm, and a variable power lens group, and a function of performing focus adjustment. Consists of a lens group. An image sensor 102 converts an optical image into an electrical signal. A camera signal processing circuit 103 performs image processing such as predetermined color conversion processing and reduction processing on the electrical signal generated by the image sensor 102 and outputs image data suitable for the output destination.

  Reference numeral 104 denotes a face detection circuit. The camera signal processing circuit 103 transmits image data suitable for face detection to the face detection circuit 104. The face detection circuit 104 performs predetermined processing on the input image data, and then extracts a candidate group of eyes, nose, mouth, and ears by pattern matching. Further, from the extracted eye candidate group, those satisfying preset conditions (for example, distance between two eyes, inclination, etc.) are determined as eye pairs, and only eye candidates having eye pairs are determined. Narrow down as a group. Then, the face candidate is detected by associating the narrowed-down eye candidate group with other parts (nose, mouth, and ears) that form a face corresponding thereto, and through a preset non-face condition filter. The face detection circuit 104 outputs the face information according to the face detection result, and ends the process. At this time, face detection information, which is a feature quantity such as the number, position, and size of faces, is stored in the memory 109 described later.

  As described above, it is possible to detect the subject information by extracting the feature amount of the image data by analyzing the image.

  Reference numeral 108 denotes a flash ROM, 109 denotes a memory, 110 denotes a camera control microcomputer, and 120 denotes a bus. Each block including the camera signal processing circuit 103 and the face detection circuit 104 is connected via a bus 120. The flash ROM 108 is a rewritable nonvolatile memory in which programs executed by the camera control microcomputer 110 and various parameters are stored in advance. A memory 109 is a volatile memory, and each block connected via the bus 120 such as a camera control microcomputer 110 is used as a work area.

  The camera control microcomputer 110 controls the camera signal processing circuit 103, the face detection circuit 104, etc., and performs various controls on the camera side.

  Camera functions for the face, such as face AF that adjusts the focus on the face, face AE that optimizes exposure according to the brightness of the face, and WB that optimizes white balance according to the face color Can be actuated to be optimal. These are performed under the control of the camera control microcomputer 110 based on the face detection information stored in the memory 109.

  The camera control microcomputer 110 determines the main subject from the plurality of face information. This determination is performed by a predetermined algorithm based on the position of the face, its size, etc. from the detected plurality of face information. Alternatively, the user may have a function of selecting an arbitrary face, and this face information may be used as the main subject.

  The camera control microcomputer 110 performs a tracking process on the main subject.

  The camera signal processing circuit 103 extracts a luminance component from the photographed image and stores the further reduced image data in the memory 109. The camera control microcomputer 110 cuts out the area determined as the main subject and the surrounding image data, and compares and tracks between the frames. This is hereinafter referred to as luminance matching. By this processing, even if the face detection circuit 104 cannot detect it, it is interpolated.

  For example, the main subject can be recognized even when the main subject faces down and the facial features cannot be detected, and the face AF or the like can be continuously operated.

  Reference numeral 105 denotes a video signal processing circuit which receives image data from the camera signal processing circuit 103 and the compression / decompression circuit 106 described later, performs image processing suitable for each block, and outputs it. In addition, the video signal processing circuit 105 can superimpose a bitmap. As a result, the device status, warning, various setting screens, and the like are displayed. In addition, the reproduction image is reduced and held in a memory, or the image is output as a display image. By reducing and displaying a plurality of frames, it is possible to realize a GUI such as an index screen in which representative images of each scene are arranged or a face index display.

  Reference numeral 106 denotes a compression / decompression circuit, which compresses image data by the MPEG2 (Moving Picture Experts Group phase 2) method, and writes compressed video data to a memory 114 (to be described later) via a bus 121 (to be described later). On the other hand, the compressed video data stored in the memory 114 is read, reproduced as image data, and output to the video signal processing circuit 105.

  In the MPEG2 video compression technique (moving picture compression method), there are an intra-frame coded picture (I picture), an inter-frame coded picture P (forward prediction) picture, and B (bidirectional prediction) picture. Of these, the I picture can be reproduced by itself.

  In addition, in the MPEG2 system, there is PTS (Presentation Time Stamp: Reproduction Time Information) indicating a reproduction time. For example, by adding this to each frame, the reproduction time of each frame can be detected during reproduction.

  The compression / decompression circuit 106 has a function that allows a recorder control microcomputer 111 (to be described later) to read at least a PTS of an I picture and information for specifying a position in the compressed video data during a compression operation. By accumulating and recording this information, the position of an arbitrary I picture can be searched, so that an arbitrary I picture can be reproduced.

  Reference numeral 111 denotes a recorder control microcomputer which controls recording / reproduction and display of moving images and management information. Reference numeral 113 denotes a flash ROM, which is a rewritable nonvolatile memory in which programs executed by the recorder control microcomputer 111, various parameters, and the like are stored. Reference numeral 114 denotes a memory, which is a volatile memory used as a work area by the recorder control microcomputer 111, the compression / decompression circuit 106, and the like. 121 is a bus. Reference numeral 115 denotes a memory card interface (card I / F), and reference numeral 116 denotes a memory card. The card I / F 115 stores the memory card 116 and transmits / receives commands and data. The memory card 116 is a card-type memory card such as an SD memory card, and a recording medium for recording the compressed video data generated by the compression / decompression circuit 106 as a moving image file according to a predetermined format compatible with a computer such as a FAT file system. It is. Note that any recording medium capable of recording compressed video data may be used, and not only a memory card but also a DVD or a hard disk.

  Reference numeral 107 denotes a liquid crystal panel for receiving and displaying image data from the video signal processing circuit 105.

  Although not particularly shown, a microphone unit, a speaker, and an external output path are also provided for audio, and are compressed and multiplexed together with the image by the compression / decompression circuit 106.

  Reference numeral 112 denotes a group of operation switches for the user to input operations. The operation switch group 112 is for selecting a camera mode (including a still image shooting mode and a moving image shooting mode) mainly for camera shooting, a playback mode for mainly playback, and a power-off mode for turning off the power. The switch is also provided. The camera control microcomputer 110 and the recorder control microcomputer 111 accept user operations on the operation switch group 112 and perform various operations in accordance with the accepted operations (accepting means).

  Note that communication is performed between the camera control microcomputer 110 and the recorder control microcomputer 111, and necessary information is transmitted and received.

  Information requesting recording of a face index, which will be described later, is transmitted from the camera control microcomputer 110 to the recorder control microcomputer 111. The face index is information for searching for a place (frame) where a new face appears.

  The recorder control microcomputer 111 records face index information in management information described later.

  The digital video camera of the present invention can switch at least the moving image shooting mode and the playback mode by operating the operation switch group 112. In the moving image shooting mode, a moving image can be shot and a moving image file can be recorded on the memory card 116. In the reproduction mode, a moving image file recorded on the memory card can be reproduced, and a face index can be displayed.

<Movie shooting mode processing>
FIG. 2 is a flowchart for explaining the moving image shooting mode processing of the present invention. When the operation switch group 112 is operated to switch to the moving image shooting mode, the moving image shooting mode process is started. The moving image shooting mode processing is realized by the camera control microcomputer 110 and the recorder control microcomputer 111 cooperating to develop and execute a program recorded in the memory 109 or the flash ROM 113 in the memory 109 or the memory 114. To do.

  In step S201, first, so-called through display is performed in which an image captured by the image sensor 102 is displayed as a moving image on the liquid crystal panel 107 in almost real time.

  In step S <b> 202, it is determined whether or not a moving image shooting start instruction has been given by the shooting button included in the operation switch group 112. If it is determined that the moving image shooting start instruction is not issued, the process proceeds to step S210. If it is determined that the moving image shooting start instruction is issued, the process proceeds to step S203.

  In step S203, moving image shooting / recording processing is performed. In moving image shooting / recording processing, for example, when the frame rate is set to 30 fps, image data (moving image frames) obtained at 1/30 second intervals obtained by the image sensor 102 is temporarily stored in the buffer memory. Then, the image data (moving image frame) temporarily stored in the buffer memory is compressed by the compression / decompression circuit 106, and the processed image data is stored in the write output buffer secured in the memory 114. When the image data is stored in the write output buffer for a predetermined number of seconds, the image data is sequentially written from the write output buffer to the memory card 116. At the same time, information serving as header information of the moving image data is recorded in the memory 114.

  In step S204, face index recording request processing is performed. This is a process for requesting to record a face index for a frame in which a new face appears in a moving image being shot in order to record face index information used for face index display during reproduction. Details will be described later with reference to FIG.

  In step S205, face index information generation processing is performed. This is a process of generating face index information to be actually recorded based on the face index recording request in step S204. Here, the face index information is held in the memory 114 and updated sequentially until the current moving image shooting is completed. Details of this processing will be described later with reference to FIG.

  Note that the processes in steps S203 to S205 are performed substantially in parallel.

  In subsequent step S206, it is determined whether or not the operation mode has been switched from the moving image shooting mode to another mode by operating the operation switch group 112 by the user. If it is determined that the moving image shooting mode has been switched to another mode, the process proceeds to step S207.

  In step S207, moving image shooting end processing is started. In the end processing, the image data (moving image frame) temporarily stored in the buffer memory at this time is compressed by the compression / decompression circuit 106, and the processed image data is stored in the write output buffer secured in the memory 114. Store. Then, the compressed image data stored in the write output buffer is written to the memory card 116. When the image data stored in the buffer memory has been written to the memory card 116, the information recorded in the memory 114 in step S203 is written (recorded) to the memory card 116 as header information. Then, the moving image file on the memory card 116 is closed to generate a moving image file. Further, a management information file related to the generated moving image file is recorded. The management information file will be described later with reference to FIG. In the management information file, the face index information generated in step S205 is recorded. The management information file is recorded so as to have a correspondence relationship with the captured moving image file. For example, a correspondence relationship can be established by setting the portion before the file name extension of the moving image file and the portion before the file name extension of the management information file to the same name. When the end process of step S207 is performed, the moving image shooting mode process ends.

  On the other hand, if it is determined in step S206 that the operation mode has not been switched from the moving image shooting mode to another mode, the process proceeds to step S208. In step S208, it is determined whether or not a moving image shooting end instruction has been issued by operating a shooting button included in the operation switch group 112. If it is determined that the moving image shooting end instruction has not been given, the process returns to step S203 to continue shooting the moving image. If it is determined that the moving image shooting end instruction has been issued, the process proceeds to step S209. In step S209, a moving image shooting end process is performed. Since this end process is the same as the above-described step S207, its description is omitted. When the termination process is performed in step S209, the process proceeds to step S210.

  In step S210, it is determined whether the operation mode is switched (changed) from the moving image shooting mode to another mode by operating the operation switch group 112 by the user. If it is determined that the mode has not been switched to another mode, the process returns to step S202 to continue the moving image shooting mode process. If it is determined that the moving image shooting mode has been switched to another mode, the moving image shooting mode process is terminated.

<Face index recording request processing>
Details of the face index recording request processing in step S204 of FIG. 2 will be described. The face index recording request process is performed using a face management table prepared in the memory 109.

  FIG. 3 shows an example of a face management table used in the face index recording request process. The face management table is information held in the memory 109 in order to perform face index request processing during moving image shooting / recording processing. In the face management table, a face ID is assigned to the currently detected face or a face that has just disappeared after being detected, and the appearance time, latest appearance time, disappearance time, position / size are assigned to each face ID. Record the requested flag. The appearance time is time information for identifying the frame at which the frame in which the face with the face ID is detected for the first time is the frame after the start of photographing. The latest appearance time is time information for specifying the frame at which the first frame in the period in which the face with the face ID is most recently detected is the frame after the start of imaging. The disappearance time is time information for specifying the point in time when the first frame in the period in which the face ID is not detected is detected when the face with the face ID is not currently detected. The position / size is area information indicating at what position and in what size the face with the face ID is detected in the last detected frame. The coordinate information in the frame and the area where the face is detected. The vertical size x horizontal size is recorded. The requested flag is a flag indicating whether or not a face index recording request has already been made for the face having the face ID.

  Hereinafter, the effective face ID is assumed to be a face ID in which information of appearance time, latest appearance time, position / size is stored in the face management table, that is, a face ID currently in use. In addition, the invalid face ID is a face ID in which all of the appearance time, latest appearance time, disappearance time, position / size information, and requested flag are cleared, that is, a face ID that is not used. To do.

  FIG. 4 shows a flowchart of the face index recording request process in step S204 of FIG. This face index recording request processing is realized by the camera control microcomputer 110 reading out a program recorded in the flash ROM 108, developing it in the memory 109 as a work area, and executing it.

  When the face index recording request process is started, first, in step S401, the face management table is initialized. All face IDs are invalid (no face is detected), and the face management table is stored in the memory 109 with the appearance time, latest appearance time, disappearance time, position / size, and requested flag all cleared. prepare.

  In step S402, the internal variable flag prepared in the memory 109 is initialized and set to zero.

  In step S <b> 403, face information, which is a face detection result by the face detection circuit 104, is acquired for an image of one frame imaged by the image sensor 102. If this is the first frame after the start of the face index recording request process, the main subject is determined from the acquired face information, and information necessary for the main subject tracking process is recorded (held) in the memory 109. Then, a face ID is newly issued for a face detected other than the main subject, and information about each face is recorded in the face management table.

  In step S404, it is determined whether or not the main subject has disappeared. Although the main subject is tracked according to the tracking condition based on the luminance matching, it is determined that the main subject has disappeared if the tracking condition is not satisfied as a result of analyzing the image captured by the image sensor 102. If it is determined that the main subject has disappeared, the process proceeds to step S405. If the main subject has not disappeared, the process proceeds to step S406.

  In step S405, since the main subject has disappeared, a new main subject is determined from the other faces currently detected. Of the valid face IDs managed in the face management table, one face as a main subject is determined by a predetermined algorithm, and information relating to this face is necessary for tracking the main subject recorded in the memory 109. Update information. From the face management table, all information related to the face determined as the main subject is cleared, and the face ID is invalidated.

  In step S406, it is determined whether or not the face information acquired in step S403 matches the main subject requirement condition. This determination is a determination as to whether or not a new face of the main subject for which a face index recording request is to be made has appeared. Since the tracking process is performed on the main subject as described above, the determination as to whether or not to make a face index recording request because of the appearance of the main subject depends on conditions different from the determination for a face other than the main subject described later. Done. If it is determined that the requirement for the main subject is met, the process proceeds to step S407, the internal variable flag is set to 1, and the process proceeds to step S408. By setting the internal variable flag to 1 in this way, a face index recording request is transmitted in step S421 described later, and the face index is recorded by the recorder control microcomputer 111 (recording control). If it is determined in step S406 that the required condition for the main subject is not met, the internal variable flag remains 0 and the process proceeds to step S408.

  In step S408, from the face information acquired in step S403, the face for which no corresponding face has been detected from the current frame among the face IDs for which the disappearance time is not recorded among the valid face IDs in the face management table. It is determined whether or not there is an ID. Of the valid face IDs in the face management table, the face ID with no disappearance time recorded is the face ID of the face that has been detected up to the previous frame. If it is determined that the corresponding face has been detected, that is, if it is determined that no face has disappeared, the process proceeds to step S410, and if it is determined that there is a face ID for which no corresponding face has been detected, that is, it disappears from the current frame. If it is determined that there is a face that has been moved, the process proceeds to step S409.

  In step S409, the current frame time is recorded as the disappearance time of the face ID of the lost face in the face management table. Even if the face that could be detected in the previous frame could not be detected for a while, the face was actually captured even though the person of that face was actually imaged due to a slight turn away. It may not be detected. For this reason, even when a face disappears without being detected, the disappearance time is recorded and face information is retained while the face ID remains valid for a short time after the disappearance. When the disappearance time is recorded, the process proceeds to step S410.

  In step S410, it is determined from the face information acquired in step S403 whether or not a face corresponding to the face ID in which the disappearance time is recorded is present in the current frame among the valid face IDs in the face management table ( First determination means). In this determination, the face included in the face information acquired in S403 has a predetermined threshold value in one or both of the positions and sizes recorded for the effective face ID in which the disappearance time is recorded. Judgment is made based on whether or not they match. If it is determined that there is a face corresponding to the face ID in which the disappearance time is recorded, the process proceeds to step S411. If it is determined that there is no corresponding face, the process proceeds to step S412.

  In step S411, the disappearance time of the corresponding face ID is deleted (cleared), and the current frame time is recorded as the latest appearance time. As a result, the face that has disappeared for a while but the same face as previously detected has been restored, and face index recording requests are repeatedly made for the face of the same person with many face movements. The problem etc. are suppressed.

  In step S412, it is determined whether there is a face ID for which 4 seconds have elapsed from the disappearance time among the valid face IDs in the face management table. If it is determined that there is a face ID for which 4 seconds have elapsed from the disappearance time, the process proceeds to step S413, and if it is determined that there is no face ID for which 4 seconds have elapsed from the disappearance time, the process proceeds to step S414.

  In step S413, all the information recorded for the face ID for which 4 seconds have elapsed from the disappearance time is erased and set as an invalid face ID. The face ID is invalidated 4 seconds after the disappearance, when 4 seconds have elapsed from the disappearance time, it is not disappeared by a small movement such as turning away the face. This is because it is considered that they are no longer in the range. Note that the threshold value of 4 seconds may be set as appropriate depending on how long the face person has disappeared from the range in which the previously detected face is not actually captured. Since it is a matter, it may not be 4 seconds. When the process of step S413 is performed, the process proceeds to step S414.

  In step S414, it is determined from the face information acquired in step S403 whether there is a face that does not correspond to any of the valid face IDs in the face management table. The case where there is a face that does not correspond to any of the valid face IDs in the face management table is a case where a new face that is not currently managed in the face management table is detected. In this case, the process proceeds to step S415. On the other hand, if it is determined to be false in step S414, a new face has not been detected, and the process proceeds to step S417.

  In step S415, it is determined whether or not the detected new face is the main subject by a predetermined algorithm (third determining means). If it is determined to be the main subject, it is not managed in the face management table, so information necessary for tracking processing of the main subject is recorded (held) in the memory 109, and the process proceeds to step S417. On the other hand, if it is determined that the detected new face is not the main subject, the process proceeds to step S416, where a new face ID is issued for the detected new face, and the appearance time and latest information for the new face are displayed in the face management table. Record appearance time (equal to appearance time), position and size. The new face ID is assigned the ID having the smallest number among the invalid face IDs (invalid face IDs) and becomes a valid face ID. Then, it progresses to step S417.

  In step S417, it is determined whether or not there is a face ID for which 1 second has elapsed from the latest appearance time among the valid face IDs in the face management table and for which no disappearance time is recorded (second determination). means). This is a determination as to whether or not there is a face that has been detected continuously for one second from the latest appearance time. If it is determined that there is no face ID for which one second has elapsed from the latest appearance time, the process proceeds to step S421, and if it is determined that there is a face ID for which one second has elapsed from the latest appearance time, the process proceeds to step S418. Here, it is determined whether or not the detection is continued for 1 second in order to prevent a face index recording request for a face that has been detected only for a moment, and for the I picture used for face index display during playback. This is to ensure that the face is always shown in the image. By requesting face index recording on the condition that the face has been detected for more than 1 second, it is possible to prevent face indexes from being recorded for faces with extremely short detection times. Suppressing the increase. Whether or not it is 1 second is a design matter, and is not limited to 1 second, and it may be determined whether or not it has been continuously detected for a predetermined time in a time longer than the appearance interval of the I picture.

  In step S418, it is determined by referring to the face management table whether or not a requested flag is set for the face ID of the face that has been continuously detected for one second from the latest appearance time. If it is, the face index recording request has already been made at the time of appearance before the latest appearance time, so that the face index recording request is not made this time, and the process proceeds to step S421. Note that when the requested flag is attached, the face ID has an appearance before the latest appearance time, and thus the face ID is determined to be true in step S410. On the other hand, if it is determined that the requested flag is not attached, the process proceeds to step S419.

  In step S419, since there is a completely new face that has been detected continuously for more than 1 second from the appearance time, the internal variable flag is set to 1. By setting the internal variable flag to 1 in this way, a face index recording request is transmitted in step S421 described later, and the face index is recorded by the recorder control microcomputer 111 (recording control). In step S420, the requested flag in the face management table is set to 1. Then, it progresses to step S421.

  In step S421, information for specifying the current frame and information on the internal variable flag in the memory 109 are transmitted to the recorder control microcomputer 111. At this time, if the internal variable flag is 1, it is a face index recording request. When the process of step S421 is completed, the process returns to step S402 and the process for the next frame is performed.

<Face index information generation processing>
FIG. 5 shows a flowchart of the face index information generation process in step S205 of FIG. The face index information generation process is realized by the recorder control microcomputer 111 reading out the program recorded in the flash ROM 113, developing it in the memory 114 as a work area, and executing it.

  In step S501, first, a timer is started.

  In step S502, the information transmitted in step S421 of FIG. 4 is received and temporarily stored in the memory 114. In step S503, it is determined whether 0.5 second has elapsed since the timer was started in step S501. If it is determined that 0.5 seconds have not elapsed, the process returns to step S502 to further receive and temporarily store information. That is, the information transmitted in step S421 is accumulated in the memory 114 for a period of 0.5 seconds. If it is determined in step S503 that 0.5 second has elapsed, the process proceeds to step S504.

  In step S504, it is determined whether or not there is a face index recording request in the information for 0.5 second period accumulated in the memory 114. If there is a face index recording request, the process proceeds to step S505. If not, the process returns to step S501, and the timer is reset and restarted to measure the next 0.5 second period, and the process is repeated.

  In step S505, a frame for identifying a frame for which a face index recording request is made is referred to from the information for 0.5 second period accumulated in the memory 114. Then, the PTS indicating the position of the I picture immediately before the frame is recorded in the face index information prepared in the memory 114. At this time, in order to identify each face index, face index numbers are assigned in order of recording and managed in time series. When the process of step S505 is completed, the process returns to step S501, and the timer is reset and restarted to measure the next 0.5 second period, and the process is repeated.

  In this way, face index information is generated and updated in the memory 114 during shooting, and the face index information at the end of shooting is stored in the management information file of the memory card 116 as described above in steps S207 and S209 in FIG. Record.

<Management information file>
FIG. 6 shows a configuration example of the management information file recorded on the memory card 116 in steps S207 and S209 of FIG. The management information file 601 includes basic information 602, a search table 603, a model information table 604, and face index information 605. In the basic information 602, basic information such as a compression method of compressed video data, a frame rate, and the number of pixels is recorded. The search table 603 is a search table required for special playback such as fast-forwarding or when displaying a frame for a specified time. Information indicating the ID of all I pictures included in the moving image file, the PTS of each I picture, and the number of bytes of data position from the beginning of the moving image file is recorded. If the moving image file is divided into packets, there may be information on how many packets each I picture is included in. With this search table 603, the position of the I picture in the moving image file can be specified as long as the PTS of the I picture is known.

  The model information table 604 is an area in which a manufacturer ID and model ID are recorded, and a unique ID is assigned to each manufacturer and product.

  In the face index information 605, a face index information ID 606, a face index number 607, and each face index 608 are recorded.

  In the face index information ID 606, an identifier indicating that this area is a portion where face index information is recorded is recorded. If the playback apparatus can recognize this identifier, the information in this area can be used as face index information, and face index display can be performed. In a playback apparatus that cannot identify the face index information ID (it does not know that this ID indicates face index information), the face index information 605 is not used.

  The face index number 607 indicates the number of face indexes recorded in each face index 608, and is recorded for a maximum of N. N is a predetermined maximum value. This is because an increase in face information is disadvantageous in terms of working memory capacity and search speed, so that an upper limit is set.

  Each face index 608 is information for specifying an image to be displayed in the face index display. For each face index, the face index number and the PTS temporarily stored in the memory 114 in step S605 in the face index information generation process of FIG. 5 described above are recorded. Using this PTS, the position in the moving image file of the I picture of each image to be displayed in the face index display can be searched from the search table 603 and specified. Thereby, it is possible to realize a reproduction function such as reproduction from the position of each image (image specified by the face index) designated in the face index display.

  The part before the file name extension of the management information file is the same as the part before the file name extension of the corresponding video file so that the correspondence can be taken. Another file. Note that this management information file may be added as header information in the moving image file instead of being separated from the moving image file.

  The digital video camera of the present embodiment can perform the above-described face index display in the playback mode using this management information file (display control). When face index display in the playback mode is instructed by operating the operation switch group 112, the digital video camera uses the management information in which the file name before the extension is the same as the video file that is the target of face index display. Search for a file. Then, based on the face index information 605 included in the searched management information file, control is performed so that the face index display as shown in FIG. 8 is performed. In the images 804 to 808 in FIG. 8, the images of the frames indicated by the PTSs with face index numbers 0 to 4 recorded in the image management file are displayed. When any one of these images is selected by the user's operation and a reproduction instruction from the user is accepted, the reproduction of the moving image is started from the frame indicated by the PTS of the face index number corresponding to the selected image (reproduction control). ).

<Timing chart of face index recording request when this application is applied>
FIG. 7 shows the recording request timing of each face index included in the face index information according to the present invention when the same moving image as that described above with reference to FIG. 9 is taken.

  The periods 901 to 903, 911 to 915, and 921 to 923 have the same meaning and the same timing as those described above with reference to FIG.

  The face index number 950 indicates the timing of the face index recording request performed by the face index recording request process described with reference to FIG. 4 and the face index number of each face index. According to the face index recording request process of the present application, as shown in the example of FIG. 7, four face index recording requests are made, and four face indexes with face index numbers 0 to 3 are recorded in the face index information. . In the face index display during reproduction, four images corresponding to the face index numbers 0 to 3 are displayed on the timeline.

  The timing of each face index recording request will be described in detail.

  In the face detection in the period 901 and the period 911, the face with the face index number 0 is recorded because the main subject and the face with face ID 1 are newly detected and continue to be detected for 1 second or longer.

  In the face detection in the period 912, the face is detected within 4 seconds after the detection in the period 911 is interrupted, and the same ID is assigned. In other words, the face is determined to be a face corresponding to the face ID having the disappearance time in step S410 in FIG. 4 (the face considered to be the same person as face ID1 that has disappeared after being detected in period 911). . Therefore, face ID1 has already requested a face index recording in period 911, and since it is determined that the requested flag is attached in step S418 in FIG. 4, no new face index recording request is made. . As a result, the face index recording request is omitted when the face of the same person is not detected for a short period of time, and the number of face indexes is prevented from increasing more than necessary.

  In the face detection in the period 913, the face is detected 4 seconds after the detection in the period 912 is interrupted. Therefore, the face ID 1 that is the same ID is assigned, but the face ID 1 has expired once, and it is not determined that the face detected in the period 913 and the face detected in the period 912 are the same face. Absent. However, since the period 913 is less than 1 second, there is no determination of Yes in step S417 in FIG. 4, and no new face index recording request is made. Thus, the face index recording request is omitted during the face detection period having an extremely short detection time, and the number of face indexes is prevented from increasing more than necessary.

  In the detection by the period 914, the face index is recorded with the face index number 2 since the face is continuously detected for 1 second or more after 4 seconds have passed since the detection of the period 913 is interrupted.

  In the face detection in the period 915, a face index recording request is omitted for the same reason as the face detection in the period 912.

  In the period 921, since a face to which the face ID 2 is assigned is newly detected and continuously detected for 1 second or longer, the face index is recorded with the face index number 1. In the face detection in the period 922, a face index recording request is omitted for the same reason as the face detection in the period 912. In the detection by the period 923, the face index is recorded with the face index number 3 because the face is continuously detected for 1 second or more after 4 seconds have passed since the detection of the period 922 was interrupted.

  As described above, in the example of FIG. 7 to which the present invention is applied, four face index numbers of face index numbers 0 to 3 are recorded. The moving images to be recorded are the same as those described above with reference to FIG. 9, but the number of face indexes is smaller than the conventional number described above with reference to FIG. As described above, according to the present invention, it is possible to suppress an unnecessary increase in the number of face indexes without performing tracking processing with a high processing load on faces other than the main subject.

  In the present embodiment, the example in which the present invention is applied during the moving image shooting mode process that is performed when an image capturing apparatus capable of capturing a moving image is described. However, the present invention is not limited to image capturing but can be applied during reproduction. Management information as shown in FIG. 6 is performed by playing back a video that has already been captured and recorded, performing face detection processing from each frame, and performing processing corresponding to the recording request processing in FIG. 4 and the face index information generation processing in FIG. You may create a file.

  As described above, according to the present embodiment, it is possible to prevent an increase in the face index information due to the absence of the tracking process in the face information other than the main subject by adding a simple process. That is, the position where a new face appears can be recorded with higher reliability at the time of shooting so that the number of faces other than the main subject does not increase more than necessary when the face index is displayed during playback.

  Therefore, it is possible to appropriately suppress the number of thumbnails other than the main subject in a GUI such as a timeline and realize a function that is favorable for the user.

  Note that the processing of the above-described embodiment may provide a system or apparatus with a storage medium that records software program codes that embody each function. The functions of the above-described embodiments can be realized by the computer (or CPU or MPU) of the system or apparatus reading out and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying such a program code, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, or the like can be used. Alternatively, a CD-ROM, CD-R, magnetic tape, nonvolatile memory card, ROM, or the like can be used.

  The functions of the above-described embodiments are not only realized by executing the program code read by the computer. In some cases, an OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. include.

  Further, the program code read from the storage medium may be written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Is also included.

It is a block diagram which shows the structure of the digital video camera as one Embodiment of this invention. It is a flowchart explaining the moving image shooting mode processing in the present embodiment. It is an example of the face management table used by the face index recording request process in this embodiment. It is a flowchart explaining the face index recording request | requirement process in this embodiment. It is a flowchart explaining the face index information generation process in this embodiment. It is an example of the management information file produced | generated by the moving image shooting mode process in this embodiment. It is a timing chart which shows the timing of the face index recording request | requirement performed by the face index recording request | requirement process in this embodiment. It is the example of a display which displayed the face image detected in the moving image as a list. It is a timing chart which shows the timing of the conventional face index recording request | requirement.

DESCRIPTION OF SYMBOLS 101 Lens unit 102 Image pick-up element 103 Camera signal processing 104 Face detection circuit 105 Video signal processing circuit 106 Compression / decompression circuit 107 Liquid crystal panel 108 Flash ROM
109 Memory 110 Microcomputer for camera control 111 Microcomputer for recorder control 112 Operation switch group 113 Flash ROM
114 Memory 115 Card I / F
116 memory card 120 bus 121 bus

Claims (15)

Face detection means for detecting a face from each frame of the video;
Of the faces detected by the face detection means, for the face determined as the main subject, a first period in which the main subject is detected by the face detection means and the main subject are detected in the video. A period after the first period, a second period satisfying a first condition including matching conditions other than the face position and size and time relating to the main object, and the main object is detected by the face detection means. Information for specifying a frame for the first period is recorded as face index information for a period continuous with the third period detected again, and information for specifying a frame for the third period is the face The index information is controlled not to be recorded, and among the faces detected by the face detection means, the face detection of the moving image is detected for a specific face other than the main subject. A first period in which the specific face is detected in a stage, a second period within a predetermined time after the specific face is no longer detected, and the specific face that has been detected in the first period For a period in which a face that satisfies the second condition relating to at least one of the position and the size of the face is detected by the face detection means, the information specifying the frame for the first period An image processing apparatus comprising: control means for recording information as face index information and controlling not to record information specifying a frame for the third period as the face index information.   The image processing apparatus according to claim 1, wherein the first condition is a condition including matching by luminance related to the main subject.   The image processing apparatus according to claim 2, wherein the control unit does not perform matching by luminance in the second period for the specific face other than the main subject.   The control means records the face index information when the face detection means continuously detects the specific face other than the main subject for a specific time or longer, and the specific face continues for the specific time or longer. 4. The image processing apparatus according to claim 1, wherein the face index information is not recorded when the face index information is not detected. A holding means for holding area information of at least one of the position and size of the specific face detected immediately before the specific face is no longer detected;
2. The second condition is that a face that matches the area information held in the holding unit is detected within a predetermined time after the specific face is no longer detected. 5. The image processing apparatus according to any one of items 4 to 4.   The said holding | maintenance means erase | eliminates the information currently hold | maintained regarding this face when the said predetermined time passes without the face corresponding to the said area | region information being hold | maintained being detected. Image processing device. It further has an imaging means capable of shooting a video,
The image processing apparatus according to claim 1, wherein the face detecting unit detects a face from each frame of the moving image being shot while the moving image is being shot by the imaging unit. . The moving image is a moving image compressed by a moving image compression method including an intra-frame encoded picture and an inter-frame encoded picture,
The imaging apparatus according to claim 1, wherein the control unit performs control so as to record information specifying a position of an intra-frame encoded picture as the face index information. The moving image is a moving image compressed by a moving image compression method including an intra-frame encoded picture and an inter-frame encoded picture,
The control means controls to record information specifying a position of an intra-frame coded picture as the face index information;
The image processing apparatus according to claim 4, wherein the specific time is longer than an appearance interval of the intra-frame coded picture in the moving image. Display control means for controlling to display an image of a frame specified by the face index information recorded by the control means;
Accepting means for accepting an operation for selecting one image from images displayed by the display control means from a user;
The image processing apparatus according to claim 1, further comprising reproduction control means for controlling the reproduction of the moving image from the position of the frame of the selected image. Face detection means for detecting a face from each frame of the video;
Of the faces detected by the face detection means, for the face determined as the main subject, a first period in which the main subject is detected by the face detection means and the main subject are detected in the video. A period after the first period, a second period satisfying a first condition including matching conditions other than the face position and size and time relating to the main object, and the main object is detected by the face detection means. Control is performed so that the third period is treated as a series of periods in which the main subject exists without treating the third period as a period in which the main subject has newly appeared for a period in which the third period detected again is continuous. With
Among the faces detected by the face detection means, for a specific face other than the main subject, a first period in which the specific face is detected by the face detection means in the video, and the specific face A face that satisfies a second condition relating to at least one of the second period within a predetermined time after the detection of the detection and the position and size of the specific face detected in the first period is the face detection Control that the third period detected by the means is continued as a series of periods related to the specific subject without treating the third period as a period in which the specific subject has newly appeared. And an image processing apparatus. A face detection step for detecting a face from each frame of the video;
Of the faces detected in the face detection step, for the face determined as the main subject, in the moving image, the first period in which the main subject is detected in the face detection step and the main subject are detected. A period after the first period, a second period satisfying a first condition including a matching condition other than a face position, size, and time related to the main object, and the main object is detected in the face detection step. Information for specifying a frame for the first period is recorded as face index information for a period continuous with the third period detected again, and information for specifying a frame for the third period is the face In addition to controlling not to record as index information,
Among the faces detected in the face detection step, for a specific face other than the main subject, a first period in which the specific face is detected in the face detection step in the video, and the specific face A face that satisfies a second condition relating to at least one of the second period within a predetermined time after the detection of the detection and the position and size of the specific face detected in the first period is the face detection For a period continuous with the third period detected in the step, information for specifying a frame for the first period is recorded as the face index information, and information for specifying a frame for the third period is And a control step of controlling not to record the face index information. A face detection step for detecting a face from each frame of the video;
Of the faces detected in the face detection step, for the face determined as the main subject, in the moving image, the first period in which the main subject is detected in the face detection step and the main subject are detected. A period after the first period, a second period satisfying a first condition including a matching condition other than a face position, size, and time related to the main object, and the main object is detected in the face detection step. Control is performed so that the third period is treated as a series of periods in which the main subject exists without treating the third period as a period in which the main subject has newly appeared for a period in which the third period detected again is continuous. With
Among the faces detected in the face detection step, for a specific face other than the main subject, a first period in which the specific face is detected in the face detection step in the video, and the specific face A face that satisfies a second condition relating to at least one of the second period within a predetermined time after the detection of the detection and the position and size of the specific face detected in the first period is the face detection Control to treat the third period as a series of periods related to the specific subject without treating the third period as a period in which the specific subject has newly appeared for a period in which the third period detected in the step is continuous. And a control step for controlling the image processing apparatus.   The program for functioning a computer as each means of the image processing apparatus as described in any one of Claims 1 thru | or 11.   A computer-readable recording medium storing a program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 11.

Images