JP2010140425A - Image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To shorten processing time required for registering an image by reducing arithmetic processing required for registering, for example, a face image, or the like in an image processing system for registering information on a predetermined image part of a plurality of frame images. <P>SOLUTION: When processing a plurality of frame images in time series, an initial searching means searches an image part to be searched in a predetermined local area set in a corresponding frame of a predetermined number of initial frame images, a follow-up searching means sets a local area in the frame by using information based on the search result obtained in a frame image preceding to the initial frame in time series in a frame image subsequent to the initial frame in time series, and searches an image part to be searched in the local area, and an image part information registering means registers information about the image part of a part or the whole of the searched image part or the whole searched image part. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to, for example, registration of a face image in a similar face image search system, and more particularly to a technique for reducing arithmetic processing required for registration of a face image.

With the increase in crime occurrence today, it is required to store surveillance images from a plurality of imaging devices over a long period of time. Until now, it was common to store only the latest surveillance video with a single imaging device in small stores, etc., but with the increase in capacity of magnetic recording devices and the widespread use of imaging devices, it has been easy to Monitoring by is possible. As a result, the burden on the video supervisor is increasing, and for the purpose of reducing the burden on the supervisor, searching for a specific person or a specific scene from the recorded surveillance video can be performed using object detection or image detection such as face detection. This is made possible by a similar image retrieval technique using the composition, color distribution, and the like.
However, the calculation processing for analyzing the object detection such as face detection from the monitoring video by the imaging device and the composition of the image is heavy on the computer, and it is difficult to perform image search in real time. For this reason, research has been conducted on reducing the amount of computation for object detection processing such as face detection and shortening the processing time.

FIG. 7 shows an example of the configuration of a similar face image search system for monitoring video according to the prior art.
The networks 104 and 105 are communication networks such as a dedicated line, an intranet, the Internet, and a wireless LAN (Local Area Network) that perform data communication by connecting devices to each other.
The imaging device 101 is a device such as a network camera that performs digital conversion processing on the captured video and outputs the converted video data to the recording device 102 via the network 104.
The recording device 102 includes a microcomputer and the like, and is a device such as a network digital recorder that records image data input from the imaging device 101 via the network 104 on a recording medium such as an HDD (hard disk device).

  The monitoring terminal 103 is a device such as a personal computer (PC) that displays image data acquired from the recording device 102 via the network 105 on a monitor such as a liquid crystal display or a CRT. The monitoring terminal 103 includes a monitor that displays a recorded image, a search result image, a similar image search menu, a user input unit such as a keyboard and a mouse, and the like. Provide a user interface.

The recording apparatus 102 includes functional units such as a network transmission / reception unit 111, a video recording unit 112, a video distribution unit 113, an image feature amount recording unit 114, an image feature amount extraction unit 115, an image similarity determination unit 116, and a face search unit 117. I have.
The network transmission / reception unit 111 is a processing unit that transmits / receives data to / from devices connected via the networks 104 and 105. Here, the data to be transmitted / received include, for example, a captured image transmitted from the imaging device 101, a video reproduction request or search request or image registration (processing) request from the monitoring terminal 103, a distribution image to the monitoring terminal 103, or a search result. is there.

  A captured image received by the network transmission / reception unit 111 from the imaging apparatus 101 via the network 104 is output from the network transmission / reception unit 111 to the video recording unit 112. As for signals received from the monitoring terminal 103 via the network 105, the video reproduction request signal is sent to the video distribution unit 113, the search request signal is sent to the image similarity determination unit 116, and the image registration request signal is sent to the image feature amount recording unit. 114 respectively. The distribution image is input from the video distribution unit 113 and the search result is input from the image similarity determination unit 116 to the network transmission / reception unit 111 and transmitted to the monitoring terminal 103.

  The video recording unit 112 is a processing unit that records image data input from the imaging apparatus 101 via the network 104 onto a recording medium such as an HDD (not shown). When recording image data, information for later retrieving image data, for example, frame numbers assigned in order from the start of recording so as to be unique within the apparatus, are simultaneously recorded. In addition, information for identifying when the image was taken, for example, the image time is also recorded. Here, as the image time, for example, the device time output from the clock function unit built in the recording apparatus 102 or the device time output from the clock function unit built in the imaging device 101 is used. .

  The video distribution unit 113 is a processing unit that distributes image data recorded in a recording medium such as the video recording unit 112 and the image feature amount recording unit 114. The video distribution unit 113 determines a distribution image according to a video reproduction request signal input from the monitoring terminal 103 via the network 105, reads image data from the recording medium using the frame number and the image time, and Input to the transceiver 111.

The image feature amount recording unit 114 is a processing unit that records an image feature amount on a recording medium. The image feature amount is acquired by outputting the face image data acquired by outputting the image data recorded by the video recording unit 112 to the face search unit 117 to the image feature amount extraction unit 115. When recording the image feature amount, the frame number corresponding to the image data input to the face search unit 117 is also recorded.
Hereinafter, the list data including the frame number and the image feature amount generated by this recording will be referred to as image feature amount list data.
The start of processing is determined by the registration request unit 125.

  The image feature amount extraction unit 115 is a processing unit that calculates the feature amount of the face image data input from the face search unit 117 using an image recognition technique. Here, as the image feature amount, for example, an image color distribution, an edge pattern composition distribution, a combination thereof, or the like is used.

The image similarity determination unit 116 is a processing unit that performs an image search and outputs a search result. The image similarity determination unit 116 outputs an image feature amount of a face image acquired by outputting a search image designated as a template of an image to be searched for determining the similarity to the face search unit 117, and a video recording The similarity is calculated from the image feature amount of the face image of the image data recorded on the recording medium of the unit 112, and a search result is generated from the calculated magnitude of the similarity. Specifically, for example, a search result having a similarity higher than a predetermined value is used. Here, the search image is included in the search request signal input from the monitoring terminal 103 via the network 105. The image feature amount of the face image data recorded on the recording medium is acquired from the image feature amount list data recorded in the image feature amount recording unit 114 using the frame number.
Various methods for calculating the similarity may be used. For example, a paper in Non-Patent Document 1 can be referred to.

The monitoring terminal 103 includes functional units such as a network transmission / reception unit 121, a search request unit 122, a search result display unit 123, a video reproduction unit 124, and a registration request unit 125.
The network transmission / reception unit 121 is a processing unit that transmits / receives data to / from the recording device 102 connected via the network 105. Here, the data to be transmitted / received are, for example, a video reproduction request signal to the recording device 102, a search request signal, a distribution image from the recording device 102, and a search result.

  As a signal received from the recording device 102, the distribution image is output to the video reproduction unit 124, and the search result is output to the search result display unit 123. The video playback request signal is input from the video playback unit 124, the search request signal is input from the search request unit 122, the registration request is input from the registration request unit 125, and the network transmission / reception unit 121. And transmitted to the recording device 102.

The search request unit 122 is a processing unit that converts a search operation performed by the user from a screen displayed on the monitor of the monitoring terminal 103 into a search request signal and outputs the search request signal.
The search result display unit 123 is a processing unit that draws the input search result from the recording device 102 on a screen displayed on the monitor of the monitoring terminal 103.
The video playback unit 124 is a processing unit that converts a playback operation by the user given from the screen displayed on the monitor of the monitoring terminal 103 into a video playback request signal and outputs the video playback request signal. In addition, the video playback unit 124 also performs processing for drawing the input distribution image from the recording device 102 on a screen displayed on the monitor of the monitoring terminal 103.
The registration request unit 125 is a processing unit that converts an image registration operation that is a user's registration start instruction given from a screen displayed on the monitor of the monitoring terminal 103 into a registration request signal and outputs the registration request signal.

JP 2008-109336 A JP 2007-042072 A "Representation model for large-scale image collection", Satoshi Tsunoike et al., Journal of the Japan Photography Society, Vol. 66, No. 1, 2003, P93-P101

  However, the similar face image search system according to the related art has a large amount of face detection processing, which hinders similar face search from a large amount of accumulated video or similar face search in real time. For this reason, in the similar face image search system according to the related art, the calculation processing amount of the process of extracting the face area from the image captured by the imaging device 101 is reduced, and the processing time required for registering the feature amount of the face image is reduced. There was a strong demand for this.

  For example, the feature of surveillance video has not been used to reduce the amount of computation required for face search according to the prior art. In many cases, the surveillance video is captured by a surveillance camera having a fixed shooting direction and shooting range. Also, in the surveillance video by surveillance cameras installed in narrow corridors in buildings and surveillance cameras installed in the escalator lift position, the moving direction of the person being imaged is constant, and the size of the subject in a series of scenes The feature is that the change is almost constant. That is, there is a feature that there is a local region in which a person starts to appear intensively or ends (disappears) intensively.

  The present invention has been made in view of such conventional circumstances. For example, an image processing system capable of reducing the processing time required for registration of an image by reducing the arithmetic processing required for registration of a face image or the like. The purpose is to provide.

In order to achieve the above object, in the present invention, an image processing system for registering information on a predetermined image portion for a plurality of frames of images has the following configuration.
That is, in the present image processing system, the images of the plurality of frames are processed in time series. The initial search means searches for an image portion to be searched in a predetermined local area set in the frame in images of a predetermined number of initial frames. The follow-up search means uses information based on the search result obtained in the image of the previous frame in the time series order within the frame in the image of the frame after the initial frame in the time series order. Then, a local area is set, and an image portion to be searched is searched for in the local area. The image part information registration unit registers information related to the image part for a part or all of the image part searched by the initial search unit and the follow-up search unit.

  Accordingly, a plurality of frames of images are processed in time series order, and an image portion to be searched for is searched for images of subsequent frames in time series order using information obtained from images of previous frames in time series order. Thus, it is possible to reduce the burden required for the search processing of the image portion to be searched, for example, to reduce the calculation processing required for registering a face image or the like, and to shorten the processing time required for image registration. it can.

Here, various image processing systems may be used. For example, an image processing system having an imaging device, a recording device, and a monitoring terminal (device) can be used.
Moreover, as an image, the image (video) imaged by the imaging device can be used, for example.
Further, as the plurality of frames, for example, temporally continuous frames may be used, or frames skipped in time after being thinned out may be used.
Various numbers may be used as the number of frames.

Various predetermined image portions may be used. For example, an image portion such as a human face or a predetermined object can be used.
Various pieces of information relating to the predetermined image portion may be used. For example, information such as a feature amount representing the feature of the image portion can be used.
In addition, as a mode for registering information, for example, a mode in which information is recorded (stored) in a memory can be used.

Various numbers may be used as the initial number of frames (predetermined number), for example, 2.
Various predetermined local regions set in the initial frame may be used, for example, a preset region is used. As a specific example, if the image portion to be searched becomes larger as time passes, a region with a small area is set as the initial local region, and the image portion to be searched becomes smaller as time passes. In such a case, a large area can be set as the initial local region.

Also, as a method of setting a local region using information based on a search result obtained from an image of a previous frame in time series order in an image of a frame after time series order from the initial frame, Various things may be used, for example, the motion vector of the image part used as search object, the size (for example, area) of the image part used as search object, or the size (for example, the image part used as search object) , Area), etc., using one or more pieces of information such as the degree of change, a method for setting a region calculated by a predetermined arithmetic expression can be used.
In addition, as a target for registering information on the searched image part, for example, all of the searched image parts (for example, all frames) may be registered, or a part of the searched image part (for example, for example) , Some frames).

In the present invention, an image processing system for registering information on a predetermined image portion for a plurality of frames of images has the following configuration.
That is, in the present image processing system, the images of the plurality of frames are processed in the reverse time series order (the order opposite to the time series order). The initial search means searches for an image portion to be searched in a predetermined local area set in the frame in images of a predetermined number of initial frames. Information based on the search result obtained by the follow-up search means in the image of the frame subsequent to the initial frame in the reverse time-series order and the image of the previous frame in the reverse time-series order within the frame Is used to set a local region, and an image portion to be searched is searched for in the local region. The image part information registration unit registers information related to the image part for a part or all of the image part searched by the initial search unit and the follow-up search unit.

Therefore, an image part to be processed with respect to an image of a subsequent frame in the reverse time series order using information obtained from the image of the previous frame in the reverse time series order by processing the images of the plurality of frames. Can reduce the burden required for the search processing of the image portion to be searched, for example, reduce the calculation processing required for registration of face images and the like, and shorten the processing time required for image registration. Can be planned.
Here, in the case of processing in the reverse time series order, the temporal order of the frames to be processed is reverse compared to the case of processing in the above time series order. I can say that.

  As one configuration example, in the above-described image processing system that processes in time series order or reverse time series order, (or the initial search means and the follow-up search means) based on information about the image portion searched by the follow-up search means And a search omitting unit that determines a frame that omits the search for the image portion to be searched) and that omits the search in the image of the determined frame.

Therefore, by omitting the search for the image portion to be searched for a predetermined frame, it is possible to further reduce the burden required for the search processing for the image portion to be searched. The required arithmetic processing can be reduced, and the processing time required for image registration can be shortened.
Here, various methods may be used as a method for determining a frame in which the search for the image portion to be searched is omitted. For example, the image portion to be searched has a predetermined size (for example, area). A method may be used that determines a frame that is expected to be less than (or less than).

  As described above, according to the image processing system of the present invention, it is possible to reduce the load required for the search processing of the image portion to be searched, for example, reduce the calculation processing required for registration of face images and the like, The processing time required for image registration can be shortened.

Embodiments according to the present invention will be described with reference to the drawings.
First, the outline of this proposal is shown.
In this example, the feature of the monitoring video that there is a fixed local region where people start to appear intensively is used. By searching for a face in the above-described local region, a scene through which a person passes can be detected.
Hereinafter, the above-described local area that is an area for detecting the first frame of a series of scenes is referred to as an initial search area.

  In this example, a face search is performed in the initial search area and a person is allowed to pass. In the initial search area, the size of the person to be imaged is substantially constant. For example, a small face may be searched in the initial search area. In the frames after the frame in which the face is detected in the initial search area, the motion vector calculated from the barycentric coordinates of the face area already detected in the series of scenes and the face size already detected in the series of scenes are used in the next frame. The center of gravity coordinates and the face size of the face area are estimated, and the face search area and the face search minimum and maximum sizes are changed.

  In this example, the frames may be registered in reverse chronological order. When registering monitoring videos in chronological order, the amount of calculation can be reduced by waiting in a small initial search area and limiting to small-sized faces. On the other hand, when registering in reverse chronological order, a local area where people pass intensively is set as an initial search area, and a face is searched for a large face in the initial search area, thereby detecting a small face of a distant person. The amount of calculation can be reduced. In addition, when registering in reverse chronological order, since a large face image is obtained first in a series of scenes, the face search frame is selected by estimating the face size detected in the frames after the scene. can do.

FIG. 1 shows an example of the configuration of a similar face image search system for monitoring video according to an embodiment of the present invention.
The similar face image search system of this example includes an imaging device 1, a recording device 2, a monitoring terminal 3, a network 4 that connects the imaging device 1 and the recording device 2, and a network that connects the recording device 2 and the monitoring terminal 3. 5 is provided.
Here, as the imaging device 1 and the networks 4 and 5, for example, the same devices as those shown in FIG. 7 (same names) can be used.
Note that the network 4 and the network 5 are not necessarily separate, and the same network may be used.

The recording device 2 includes a network transmission / reception unit 11, a video recording unit 12, a video distribution unit 13, an image feature amount recording unit 14, an image feature amount extraction unit 15, an image similarity determination unit 16, a face search unit 17, and a scene face search unit. 18 is provided.
The recording device 2 includes a microcomputer and the like, and is a device such as a network digital recorder that records image data input from the imaging device 1 via the network 4 on a recording medium such as an HDD (hard disk device).

Here, as the network transmission / reception unit 11, the video recording unit 12, the video distribution unit 13, the image feature quantity extraction unit 15, the image similarity determination unit 16, and the face search unit 17, for example, those shown in FIG. The thing similar to a thing of a name can be used.
In the recording device 2 of this example, for example, compared with the recording device 102 shown in FIG. 7, for example, in addition to the processing units shown in FIG. 7, a scene face search unit 18 is provided. The function of the image feature amount recording unit 14 is different.

The monitoring terminal 3 includes a network transmission / reception unit 21, a search request unit 22, a search result display unit 23, a video reproduction unit 24, and a registration request unit 25.
Here, as the network transmission / reception unit 21, the search request unit 22, the search result display unit 23, and the video reproduction unit 24, for example, those similar to those shown in FIG. 7 (same names) can be used. .
In the monitoring terminal 3 of this example, for example, the function of the registration request unit 25 is roughly different from that of the monitoring terminal 103 shown in FIG.

  The registration request unit 25 of this example has the same function as the registration request unit 125 shown in FIG. 7, for example, and in addition to that, the start of the selection and input of the registration method by the user in the monitoring terminal 3 The registration request signal is created from the value of the end frame number, and the image feature amount recording unit 14 of the recording device 2 has a function of instructing the start of registration (image registration request function).

  The scene face search unit 18 of the recording device 2 is a processing unit that detects a face area from an image recorded in the recording device 2. Here, the face search unit 17 (and the face search unit 117 shown in FIG. 7) has only a function of searching for a predetermined minimum or maximum face from the image. On the other hand, the scene face searching unit 18 records and refers to the internal memory of the barycentric coordinates, size (size), and face detection number of the face area that has been detected so far as a series of scenes. Has a function for calculating a face and calculating a face minimum and maximum size to search for a face.

  The image feature amount recording unit 14 of the recording device 2 uses the scene face search unit 18 to register the feature amount of the face image from the image stored in the HDD or the like of the recording device 2. The registration request unit 25 of the monitoring terminal 3 has a function of changing the registration method in accordance with the registration method designated by the user.

FIG. 2 schematically shows an example of an image of an imaging device 1 (for example, a monitoring camera) installed to monitor a person passing through a corridor in a building from a distance to the front. This is one of the monitoring videos used in the registration process of the similar face image search system for monitoring videos.
The face search process performed by the scene face search unit 18 for each of the following face image registration methods (registration method 1) to (registration method 3) will be described.
(Registration method 1) When registering in chronological order (Registration method 2) When registering in reverse chronological order (Registration method 3) When registering in reverse chronological order and omitting face search for small faces in a series of scenes

(Registration method 1) The case of registering in chronological order will be described.
The frame used for registration is from the frame specified by the user or the first frame of the video stored in the recording device 2 to the frame specified by the user or the final frame of the video stored in the recording device 2. Registration is performed based on each frame number in order from the first frame. Here, a method of registering in the order of frame numbers will be described. For example, when the image difference between frames is small in a scene where a person passes, the frames to be registered may be skipped at a constant value.
Hereinafter, skipping a frame to be registered with a constant value is referred to as frame skipping.

The scene face searching unit 18 does not search for a face in the entire region 31 of the image captured by the imaging device 1 but locally detects a face in a local region 32 where the person starts to appear in the imaging range. A face search is performed in region 32. In this example, the local area 32 is set as the initial search area.
Here, the initial search area is the local area 32, but the user can specify an appropriate area for each video to be used. Also, sample video of surveillance video is acquired before system operation, face search is performed in advance in the entire area of the imaging range of the surveillance camera, human trajectory information is collected, and faces below a predetermined size are collected. The detected area or the like may be automatically set as the initial search area.

  If the face can be detected in the local region 32, a series of scenes through which a person passes is started. From the next frame in the time series order, the barycentric coordinates of the face region detected so far are used. Estimate the face area in the next frame. Here, the coordinates are defined, for example, with the upper left of the image obtained by the imaging apparatus 1 as the origin, the horizontal direction as the X axis, and the vertical direction as the Y axis.

FIG. 3 shows an example in the case of calculating a motion vector for estimating the face area in the next frame from the face detected so far in a series of scenes. This is an example in which a motion vector is obtained in the registration process of the similar face image search system for a monitoring video.
Here, the local region 32 shown in FIG. 3 represents the local region 32 which is the initial search region shown in FIG. In order to calculate the motion vector in a series of scenes, the center of gravity coordinates of the face area in at least two frames is necessary. Therefore, the face search area estimation based on the motion vector is performed after detecting the face in the local area 32. It is after the third frame.

The barycentric coordinates 41 and 42 represent barycentric coordinates of the face area detected in the local area 32. FIG. 3 shows a method for calculating the motion vector of the next frame, assuming that the face of the same person is continuously detected for each frame in a series of scenes and four barycentric coordinates of the face area are obtained. Yes.
In addition to the center-of-gravity coordinates 41 and 42, the center-of-gravity coordinates 43 and 44 are sequentially set as the center-of-gravity coordinates of the already obtained face area.

  Based on the barycentric coordinates 41 to 44, the motion vectors 51, 52, and 53 can be calculated from the movement amounts of the barycentric x-coordinate and y-coordinate between the frames. Specifically, the motion vector 51 calculated from the centroids 41 and 42 of the centroid coordinates of the face area, the motion vector 52 calculated from the centroids 42 and 43 of the centroid coordinates of the face area, and the centroid of the centroid coordinates of the face area There is a motion vector 53 calculated from 43 and 44. A motion vector 54 is calculated from the motion vectors 51, 52 and 53.

The motion vector 54 can be obtained by the following (Equation 1).
The following (Equation 1) uses the motion vector Z _i (i = 1, 2, 3,..., N) obtained from the already detected face coordinates, and the center of gravity of the face area from the current frame to the next frame. This is an equation for calculating a motion vector Z indicating the moving direction of.
Z _i and Z represent vectors, and n is a value of 2 or more.

Here, a weight W _i (> = 1) (i = 1, 2, 3,..., N) is defined for each motion vector Z _i . The motion vectors Z _i between the frames are the barycentric coordinates (X _i , Y _i ) of the face area detected in the frame corresponding to the time i and the barycentric coordinates of the face area detected in the frame corresponding to the time (i−1). Using (X _i−1 , Y _i−1 ), Z _i = (X _i −X _i−1 , Y _i −Y _i−1 ).

Regard the weight W _i of each motion vector Z _i, for example, if W i _{= 1,} the motion vector Z is a case where the equal weight of each motion vector Z _i. If there is little change in the amount of movement of the face area between frames and the actual amount of movement in the next frame is likely to be the same as before, a highly accurate motion vector Z can be obtained. On the other hand, if the moving amount is greatly different between the far side and the near side in the surveillance video, W _i = i is set, and the motion vector Z _i1 obtained far and the motion vector Z _i2 (i2) obtained near > 1) The weight of i1) is differentiated and the most recent motion vector Z _i2 is emphasized, so that the person who is walking suddenly stops or the movement direction is changed. Even when the change is severe, the motion vector Z can follow the movement of the person.
The weight used for calculating the motion vector Z is not limited to the above.

  If the motion vector Z is obtained, the estimated center-of-gravity coordinate of the face coordinate in the next frame can be obtained from the motion vector Z using the center-of-gravity coordinate of the face area in the latest frame as the origin. In the example of FIG. 3, the estimated centroid coordinates 45 of the face area in the next frame estimated from the motion vector 54 are the estimated centroid coordinates 45. If a face search is performed in a small area with the estimated center-of-gravity coordinates 45 as the center (or may not be the center), the amount of calculation required for the face search can be reduced. The size of the small area for performing the face search can be determined by estimating the face size in the next frame, for example. As a specific example, the face size in the next frame can be estimated from the face size history detected in a series of scenes, and this estimated face size is multiplied by a predetermined magnification (for example, a preset value). A face search is performed in this area.

Hereinafter, the region 55 determined from the region size and the motion vector Z is referred to as a tracking face search region.
The amount of calculation can be reduced by performing a face search using the following face search area. Further, for example, in order to further reduce the calculation amount, it is necessary to appropriately set the minimum face size and the maximum face size for performing face search. In this example, the minimum and maximum size of the face search in the next frame is determined based on the estimated face size described above. For example, assuming that the face size of the face detected in the most recent frame is the estimated face size, the minimum face search size is 0.9 × F and the maximum size is 1. based on the number of pixels F next to the estimated face size. Determined as 1 × F.

(Registration Method 2) A case where registration is performed in reverse time series will be described.
In the above (Registration Method 1), the method for searching for a face in time-sequential frames has been described. Here, a method of searching for faces in reverse time series will be described.
The frames used for registration are from the frame specified by the user or the last frame of the video stored in the recording device 2 to the frame specified by the user or the first frame of the video stored in the recording device 2. Similar to (Registration Method 1) described above, registration by skipping frames may be performed.

  When registering in reverse chronological order, a face search is performed in the local region 33 (see FIG. 2) in a frame in which a person cannot be seen from the monitoring video in a series of scenes. The local area 33 is an area in which a person cannot be seen from the monitoring video, and has the largest face size in a series of scenes. For this reason, the face search minimum size when searching for a face in the local region 33 can be made larger than when searching for a face from the entire region. It is self-evident that there is a negative correlation between the minimum face size detectable in face search and the amount of calculation. That is, the amount of calculation can be reduced as the face search minimum size is increased.

When face search is performed in reverse time series order, the motion vectors between the frames are only reversed in direction compared to the case of time series order, and the motion vector calculation method has been described above (registration method 1). It is the same as the case of.
In the case of registration in reverse chronological order, the face size that can be detected in a series of scenes gradually decreases. Therefore, in the calculation method of the minimum and maximum sizes of the face search, for example, the face size in the latest frame that has already been detected in the series of scenes is reduced by a predetermined size or the face that has already been detected in the series of scenes. The face size estimated from the size reduction rate is used.

(Registration method 3) A case will be described in which registration is performed in reverse time-series order and a small-size face search is omitted in a series of scenes.
In the above (registration method 2), the method of registering in reverse time-series order has been described. When registered in reverse chronological order, for example, in a video that a person passes from a distance as shown in FIG. 2, it is possible to detect in order from the face with the largest size for each scene that the person passes.
In order to perform a similar face image search, it is desirable to obtain a large face in a series of scenes. In the case of registration in time series order, it is necessary to make the face search area follow from the initial search window (initial search area) and detect all the faces from the small face to the large face in order. However, in the reverse time series order, a large face can be detected first, so in a series of scenes, it is estimated that a small face has been captured, that is, a face in the front frame in time series order. Search processing can be omitted.

Here, a method for selecting a frame for face search in a series of scenes will be described.
When a face has already been detected in several frames in a series of scenes, it is necessary to determine not to search for a face in subsequent frames and to determine which frames are omitted thereafter.

As a criterion for determining not to search for a face in subsequent frames, for example, whether a predetermined number or more of faces have been detected so far in a series of scenes, or whether a face image of a predetermined size or more has been obtained, etc. There is. Specifically, for example, when a predetermined number or more of faces have been detected in a series of scenes so far, it is determined not to search for a face in subsequent frames, or a face image having a predetermined size or more is selected. If it is not obtained, it can be determined that the face is not searched in the subsequent frames.
Further, in determining which of the subsequent frames can be omitted, for example, by using the estimated size s calculated by the face size reduction ratio α and the omitted number of frames β detected so far, a time series is used. It is possible to estimate from which frame a series of scenes have been started in order, and this estimated frame can be omitted.

FIG. 4 shows an example of a frame in which face search can be omitted when face images are registered in reverse chronological order. This indicates frame control in which the search for a small face is omitted when registration is performed in reverse chronological order in the registration process of the similar face image search system for the monitoring video.
Specifically, registration is performed in reverse time-series order, and faces can be detected in the initial search area in frames 69 and 68, and face search is performed in the follow-up face search area in frames 67, 66, and 65. This is a case where a face can be detected in each frame and five face images are obtained.

Here, the number of horizontal pixels of the face image is used as the face image size. The number of horizontal pixels of the face image obtained in the frame 65 is s65, the number of horizontal pixels of the face image obtained in the frame 66 is s66, the number of horizontal pixels of the face image obtained in the frame 67 is s67, and the frame 68 is obtained. The number of pixels next to the face is s68.
In this example, the number of horizontal pixels of the face image is used as the face image size, but the number of vertical pixels or the area can also be used. If the criterion for determining to omit the face search is that four or more face images are obtained in a series of scenes, the face search after the frame 64 can be omitted.

In the case of the example of FIG. 4, the reduction rate α of the face image can be obtained by the following reduction rate examples (reduction rate example 1) to (reduction rate example 4).
(Example of reduction rate 1)
α = (s68−s67) / s68
(Example 2 of reduction rate)
α = (s67−s66) / s67
(Example 3 of reduction ratio)
α = (s66−s65) / s66
(Example 4 of reduction rate)
α = {(s68−s67) / s68 + (s67−s66) / s67 + (s66−s65) / s66} / 3

The definition of the reduction rate α is the change rate of the face size between the frames, and the above is an example. Further, the number of reductions by the reduction rate α is β. That is, when a reference s65, estimated face size of the frame a series of scene is started, can be obtained by (α β × ^s65). Here, by determining in advance the face size of the frame where the scene starts in time series order as Sfirst, the face search can be omitted up to a frame satisfying Sfirst> (α ^β × s65).
For example, if the number of frames that can be omitted is calculated as 2, the face search for frames 64 and 63 is omitted, and the face search in the initial search area is performed after frame 62 (frames 62 and 61 in the example of FIG. 4). To resume.

  Also, in most scenes where people pass, for example, when the surveillance video is installed at the lift position of the escalator, if the moving speed of the person is constant and the frame for detecting the face is almost the same, The number of frames to be omitted may be fixed.

Next, referring to FIG. 5 corresponding to the above (registration method 1) and (registration method 2) and FIG. 6 corresponding to the above (registration method 3), the similar face targeted for the monitoring video in this example The example of the flow of a registration process of an image search system is shown.
In this example, the process of the similar face image search system is roughly divided into four processes: a registration process, a recording process, a video reproduction process, and a similar image search process. The recording process, the video reproduction process, and the similar image search process are as follows. For example, a process similar to the recording process, the video reproduction process, and the similar image search process according to the related art such as the one shown in FIG. 7 can be used, and detailed description thereof is omitted here.

FIG. 5 shows an example of the flow of registration processing in the (registration method 1) and (registration method 2) in the similar face image search system for monitoring video. Steps S <b> 1 to S <b> 27 are processing performed by the recording device 2, and timing S <b> 31 relates to processing performed by the monitoring terminal 3.
In step S <b> 1, the recording apparatus 2 stands by until a registration request signal created by the registration request unit 25 is received by a registration execution operation (including determination of a registration method) of the monitoring terminal 3 by the user. When reception of the registration request signal is completed, the process proceeds to step S2.
Timing S31 represents communication between the monitoring terminal 3 and the recording device 2.

In step S2, the user's operation content is verified. The verification contents are the type of face image registration method, registration start frame number, and registration end frame number. If the registration request by the user is insufficient, the process proceeds to step S1. If the user registration request is correct, the process proceeds to step S3.
In step S3, the type of the face image registration method received in step S1 is stored in the internal memory, and the process proceeds to step S4.
In step S4, the registration start frame number and registration end frame number received in step S1 are stored in the internal memory. The registration start frame number is set as the current frame number, and the process proceeds to step S5.

In step S5, an initial search area to be used is selected, and the process proceeds to step S6. In this example, when the type of registration method of the face image is (registration method 1) of the above-mentioned image, the position and size of the local area 32 are stored in the internal memory as the position and size of the initial search area. In the case of (registration method 2) of the above-described image, the position and size of the local area 33 are stored in the internal memory as the position and size of the initial search area.
In step S6, the position and size of the face search area are calculated, and the process proceeds to step S7. The position and size of the face search area are obtained by reading out the position and size of the initial search area selected in step S5 from the internal memory.

In step S7, the minimum and maximum sizes of the face search are calculated, and the process proceeds to step S8. The face search minimum and maximum sizes are obtained by reading out the face search minimum size and maximum size for the initial search area selected in step S5 from the internal memory. In this example, these values are set in advance in the system and stored in the internal memory.
In step S8, an image is read based on the current frame number, and the process proceeds to step S9.
In step S9, the current frame number is updated, and the process proceeds to step S10. When the type of registration method of the face image is (registration method 1) of the above-mentioned image, for example, a value obtained by adding 1 to the current frame number is newly set as the current frame number, and ( In the case of the registration method 2), for example, a value obtained by subtracting 1 from the current frame number is newly set as the current frame number.

In step S10, a face search is performed with the position and size of the face search area calculated in step S6 and the face search minimum and maximum size calculated in step S7. If a face (face image portion) is detected, it is determined that a series of scenes has started, and the process proceeds to step S11. If no face is detected, the process proceeds to step S24.
In step S24, registration end determination is performed. This determination is based on the comparison between the current frame number and the end frame number. Specifically, for example, when the current frame number becomes the end frame number, the registration is ended. If it is determined that the registration has been completed, the process proceeds to step S27. Otherwise, the process proceeds to step S8.
In step S11, the number of face detections in the series of scenes stored in the internal memory, the barycentric coordinates of the face area, and the face size detection history are all initialized.
In step S12, an image is read based on the current frame number, and the process proceeds to step S13.

In step S13, if the number of face detections in the series of scenes is 2 or more, the process proceeds to step S14, and if it is less than 2, the process proceeds to step S25.
In step S14, the position of the follow-up face search area is calculated, and the process proceeds to step S15. As described with reference to FIG. 3, the position of the follow-up face search area is calculated from a motion vector obtained from the history of the barycentric coordinates of the face area in a series of scenes.
In step S15, the size of the tracking face search area is calculated, and the process proceeds to step S16. The size of the follow-up face search area is calculated from the estimated face size obtained from the face size history in a series of scenes, as described with reference to FIG.
In step S16, the minimum and maximum size of the face search for the follow-up face search area is calculated, and the process proceeds to step S17. As described above, this size is calculated from the estimated face size obtained from the face size history in a series of scenes.

In step S25, the position and size of the initial search area are calculated, and the process proceeds to step S26. This calculation is performed by the same method as the process of step S6, for example.
In step S26, the minimum and maximum size of the face search for the initial search area is calculated, and the process proceeds to step S17. This calculation is performed by the same method as in step S7, for example.

In step S17, a face search is performed using the position and size of the face search area calculated in steps S14, S15, and S16, or steps S25 and S26, and the minimum and maximum size of the face search. If a face is detected, the process proceeds to step S18. If no face is detected, the process proceeds to step S21.
In step S18, the image feature amount is extracted from the face image detected in step S17, and the process proceeds to step S19.
In step S19, the image feature amount extracted in step S18 is recorded on the recording medium, and the process proceeds to step S20.
In step S20, the barycentric coordinates and size of the face area detected in step S17 are added to the detection history on the internal memory, 1 is added to the face detection count, and the process proceeds to step S21.

In step S21, the current frame number is updated, and the process proceeds to step S22. When the type of registration method of the face image is (registration method 1) of the above-mentioned image, for example, a value obtained by adding 1 to the current frame number is newly set as the current frame number, and ( In the case of the registration method 2), for example, a value obtained by subtracting 1 from the current frame number is newly set as the current frame number.
In step S22, registration end determination is performed. This determination is based on the comparison between the current frame number and the end frame number. Specifically, for example, when the current frame number becomes the end frame number, the registration is ended. If it is determined that the registration has been completed, the process proceeds to step S27. Otherwise, the process proceeds to step S23.

In step S23, it is determined whether or not a series of scenes has ended. For this determination, for example, the continuity of frames in which the face could not be detected or the magnitude relationship with a predetermined value of the size of the face that could be detected in the most recent frame is used as a reference. End when the continuity of frames that could not detect a face exceeds a predetermined value, or end when the size of the face that can be detected in the most recent frame exceeds a predetermined value (or less than a predetermined value) And If it is determined that the series of scenes are still continuing, the process proceeds to step S12. If it is determined that the series of scenes is completed, the process proceeds to step S6.
In step S27, the registration process is terminated.

FIG. 6 shows an example of the flow of registration processing in the above (registration method 3) in the similar face image search system for monitoring video.
Here, the flowchart shown in FIG. 6 is obtained by adding steps S41, S42, and S43 to the flowchart shown in FIG. 5, and steps S1 to S27 similar to those shown in FIG. About S31, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted. Steps S41 to S43 are processes performed by the recording device 2.

In step S41, processing is performed after step S11, and processing in steps S42 and S43 is performed based on the condition determination in step S41. If it is determined in step S23 that the scene has not ended, the process proceeds to step S41.
In step S41, it is determined whether or not to end face search in a frame in a series of scenes early (scene early end determination). This early scene end determination is based on, for example, the size relationship between the number of face detections in a series of scenes and a predetermined value, the size relationship between the face size detected in the previous frame and a predetermined size, or the like. Specifically, for example, when the number of detected faces in a series of scenes exceeds a predetermined value, the process ends early, or the face size detected in the previous frame is less than a predetermined size. Can be finished early. If it is determined by the early scene end determination that face search in the scene is to be ended in the current frame, the process proceeds to step S42, and if it is determined that face search in the scene is to be continued, the process proceeds to step S12.

In step S42, the aforementioned reduction ratio α and omitted frame number β are calculated from the face size history detected in the series of scenes, the number of omissible frames is determined, and the process proceeds to step S43.
In step S43, the current frame number is updated, and the process proceeds to step S24. Here, in order to perform a face search for the next scene, a value obtained by subtracting the omissible frame number β obtained in step S42 from the current frame number is newly set as the current frame number.

As mentioned above, although this proposal was demonstrated by specific embodiment, this proposal is not limited to said embodiment, For example, although the structure which mounts the registration function in the video recording apparatus 2 was demonstrated above, for example. The configuration may be such that it is mounted on a device different from the recording device.
In this example, a configuration has been shown in which a similar face image registration processing time is shortened by shortening a face image detection processing time. However, the present invention can be applied even when a detection target is not a face, and various objects can be applied. By using the detection technique, it can be similarly applied to a similar image search system for a specific object.
Furthermore, in this example, the case where the search image is selected from images captured by the imaging device 1 and recorded in the recording device 2 has been described. For example, an image captured by a device different from the imaging device 1 is used. It is also possible.
As mentioned above, although this proposal was concretely demonstrated by said embodiment, it cannot be overemphasized that said embodiment is an illustration of this proposal and this proposal is not limited to said embodiment.

A specific example of the effect obtained by this example will be described with reference to the monitoring video shown in FIG.
FIG. 2 shows an example of an image of a surveillance camera installed to monitor a person passing through a corridor in a building from a distance to the front. The entire area 31 of the video imaged by the imaging device 1 is shown.
In this example, the search area is reduced by searching for a face in the local area 32 that is an initial search area instead of a face search in the entire area 31 captured by the imaging apparatus 1 by using the feature of the monitoring video. Thus, the amount of calculation can be reduced. Further, after the frame in which the face is detected in the initial search area, the face area in the next frame can be estimated by using the face area and the face size detected in the series of scenes, and the face is searched in the estimated area. Thus, the amount of calculation can be reduced.

  As another example, when a face search is performed in the initial search region 33 in the reverse order of the time series, it is not necessary to search for a face having a small size, thereby reducing the amount of calculation. Furthermore, in the case of registering in reverse chronological order, since a large face can be acquired first in a series of scenes, it is not necessary to search for a face in a frame where a small face exists in subsequent frames. The amount of calculation reduction by reducing the search area or limiting the minimum size and maximum size of the search varies depending on the angle of view, the size of the subject, etc., but is qualitatively obvious. Further, in a series of scenes, a frame that has acquired a large face can be selected as a best shot frame that is most easily identified.

As a specific example, the scene face search unit 18 waits for a person to pass in the local region by using the feature of the monitoring video and paying attention to the local region 32 or the local region 33 of the image obtained by the imaging device 1. . In frames after the initial search area, face search is performed in the follow-up face search area.
When registration is performed in reverse time series order, the face size in a series of scenes is estimated from the face sizes detected so far. By this estimation, it is possible to omit a face search for a frame that is expected to have a small face.

As described above, the similar face image search system of this example includes the following configuration examples (configuration example 1) to (configuration example 5).
(Configuration Example 1) This is a basic configuration example of the initial window.
An imaging device 1, a recording device 2 that records an image captured by the imaging device 1, and a monitoring terminal 3, and a function of retrieving a similar face image for an arbitrary face image from images recorded in the recording device 2; In the similar face image search system having a function of displaying the searched image on the monitoring terminal 3, an initial search function using a fixed local area as a face search area is provided.

(Configuration Example 2) A configuration in which the initial window waits in time series.
An imaging device 1, a recording device 2 that records an image captured by the imaging device 1, and a monitoring terminal 3, and a function of retrieving a similar face image for an arbitrary face image from images recorded in the recording device 2; In the similar face image search system having the function of displaying the searched image on the monitoring terminal 3 and the function according to (Configuration Example 1) described above, a fixed local area having a small area is set as the face search area, An initial search function for searching for a small-sized face and a function for registering frames in chronological order are provided.

(Configuration example 3) Configuration for waiting and following in time series in the initial window The imaging device 1, the recording device 2 for recording an image captured by the imaging device 1, and the monitoring terminal 3, and similar to an arbitrary face image In a similar face image retrieval system having a function of retrieving a face image from an image recorded in the recording device 2, a function of displaying the retrieved image on the monitoring terminal 3, and a function according to the above (Configuration Example 1) A face search in a frame after the face detection frame is performed based on the minimum and maximum size of the face search area and face search calculated from the motion vector and face size of the face area already detected in a series of scenes.

(Configuration example 4) A configuration in which the initial window waits in reverse time-series order.
An imaging device 1, a recording device 2 that records an image captured by the imaging device 1, and a monitoring terminal 3, and a function of retrieving a similar face image for an arbitrary face image from images recorded in the recording device 2; In the similar face image search system having the function of displaying the searched image on the monitoring terminal 3 and the function according to (Configuration Example 1), a fixed local area having a large area is set as a face search area, and An initial search function for searching for a size face and a function for registering frames in reverse time series are provided.

(Structure Example 5) This structure waits in reverse time-series order at the initial window and ends face search early in the scene.
An imaging device 1, a recording device 2 that records an image captured by the imaging device 1, and a monitoring terminal 3, and a function of retrieving a similar face image for an arbitrary face image from images recorded in the recording device 2; In the similar face image search system having a function of displaying the searched image on the monitoring terminal 3 and a function related to (Configuration Example 4), a function of selecting a face search frame is provided.

  In the similar face image search system of this example (an example of an image processing system), in the recording device 2, the image feature amount recording unit 14 is initially searched in the time series order or the reverse time series order by the scene face search unit 18. The initial search means is configured by the function of performing the search, the follow-up search means is configured by the function of performing the subsequent search thereafter, and the search omitting means is configured by the function of omitting the search in a frame that satisfies the predetermined condition. In addition, the image feature information recording unit 14 includes an image partial information registration unit having a function of recording image feature value list data (correspondence between frame numbers and image feature values) in a memory.

Here, the configuration of the system and apparatus according to the present invention is not necessarily limited to the configuration described above, and various configurations may be used. The present invention can also be provided as, for example, a method or method for executing the processing according to the present invention, a program for realizing such a method or method, or a recording medium for recording the program. It is also possible to provide various systems and devices.
The application field of the present invention is not necessarily limited to the above-described fields, and the present invention can be applied to various fields.
In addition, as various processes performed in the system and apparatus according to the present invention, for example, the processor executes a control program stored in a ROM (Read Only Memory) in hardware resources including a processor and a memory. A controlled configuration may be used, and for example, each functional unit for executing the processing may be configured as an independent hardware circuit.
The present invention can also be understood as a computer-readable recording medium such as a floppy (registered trademark) disk or a CD (Compact Disc) -ROM storing the control program, and the program (itself). The processing according to the present invention can be performed by inputting the program from the recording medium to the computer and causing the processor to execute the program.

It is a figure which shows the structural example of the similar face image search system which concerns on one Example of this invention. It is a figure which shows an example of the image | video of a surveillance camera. It is a figure which shows an example in the case of calculating a motion vector. It is a figure which shows an example of the flame | frame which can abbreviate | omit a face search when a face image is registered in reverse time series order. It is a figure which shows an example of the flow of the registration process in (registration method 1) and (registration method 2). It is a figure which shows an example of the flow of the registration process in (registration method 3). It is a figure which shows the structural example of a similar face image search system.

Explanation of symbols

  1 .... Imaging device 2 .... Recording device 3 .... Monitoring terminal 4, 5 .... Network, 11 .... Network transmission / reception unit 12 .... Video recording unit 13 .... Video distribution unit 14 .... Image Feature amount recording unit, 15 .... Image feature amount extraction unit, 16 .... Image similarity determination unit, 17 .... Face search unit, 18 .... Scene face search unit, 21 .... Network transmission / reception unit, 22 .... Search request , 23 .. Search result display section, 24 .. Video playback section, 25 .. Registration request section, 31 .. All areas, 32, 33 .. Local area, 41 to 44 .. Center of gravity coordinates of face area, 45 .. Estimated center-of-gravity coordinates of face area, 51 to 54 .. Motion vector, 55 ..Following face search area, 61, 62, 68, 69 ..Face search frame in initial search area, 63, 64. Search frame, 65-67 Face search frame in the initial search area or follow-up face search area,

Claims (2)

In an image processing system for registering information about a predetermined image portion for images of a plurality of frames,
Processing the images of the plurality of frames in chronological order;
An initial search means for searching an image portion to be searched in a predetermined local region set in the frame in a predetermined number of initial frame images;
In the image of the frame after chronological order from the initial frame, a local region is set using information based on the search result obtained in the image of the previous frame in chronological order within the frame. And follow-up search means for searching for an image portion to be searched in the local region,
Image part information registration means for registering information about the image part for part or all of the image part searched by the initial search means and the follow-up search means;
An image processing system comprising: In an image processing system for registering information about a predetermined image portion for images of a plurality of frames,
Processing the images of the plurality of frames in reverse chronological order;
An initial search means for searching an image portion to be searched in a predetermined local region set in the frame in a predetermined number of initial frame images;
In an image of a frame subsequent to the initial frame in reverse time-series order, a local region using information based on the search result obtained in the image of the previous frame in reverse-time-series order within the frame is used. Tracking search means for searching for an image portion to be searched in the local region,
Image part information registration means for registering information about the image part for part or all of the image part searched by the initial search means and the follow-up search means;
An image processing system comprising:

Images