JP2007134913A - Method and device for selecting image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method capable of selecting an image which is easy for a user to recognize movements and shapes of the arms and legs of a person out of a plurality of camera images. <P>SOLUTION: The method of selecting at least one image out of the plurality of images picked up by a plurality of cameras includes a step (S301) of acquiring a plurality of images picked up by the plurality of cameras and including the person, steps (S302 to S305) of extracting a feature quantity found from a plurality of representative points including in the plurality of images and provided on an outline of a subject silhouette, an unevenness degree decision step (S306) of deciding degrees of unevenness indicating how much the shape of the subject silhouette is uneven from feature values as to the plurality of images, and a step (S307) of selecting at least one image for recognizing shapes of the arms or legs of the person from the plurality of images based upon the degrees of unevenness of the shape of the subject silhouette of the plurality of images. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image selection method for selecting at least one image from images simultaneously captured by a plurality of cameras, and in particular, from the images obtained by simultaneously capturing a scene including a person using a plurality of cameras, The present invention relates to a video selection method for selecting a video whose shape is easy for a user to recognize.

  Conventionally, when a store or a street is imaged and monitored with cameras installed at a plurality of locations, images obtained from each camera are displayed on a monitor, and the images are monitored by an observer. Also, in sports broadcasting such as soccer where it is necessary to capture a wide range using a plurality of cameras, switching of the video used for broadcasting is performed according to instructions from the director. In such a case, it is necessary for the observer or director to watch a plurality of images at the same time, which is a heavy burden. Also, as the number of cameras increases, it becomes difficult to pay attention to multiple images simultaneously.

  On the other hand, a method has been proposed in which a camera image is automatically selected from images simultaneously captured by a plurality of cameras using a selector (see, for example, Patent Document 1 and Patent Document 2).

  In Patent Document 1, a motion vector is detected for each video from videos taken by a plurality of cameras, and when the magnitude of the motion vector is a predetermined value or more, a corresponding camera video is selected by a selector. A method is disclosed. Similarly, a background image is prepared in advance, the presence / absence of an intruding object is determined based on the size of the difference between the background image and the current video, and if there is an intruding object, that is handled There is also a method of selecting a camera image to be performed. Similarly, there is a method of selecting a camera image in which a person is detected by using a person detection method together.

In Patent Document 2, a surveillance person marks a specific person in a plurality of surveillance camera images, and a face image of the marked specific person and a face image of a person captured by a plurality of surveillance cameras are recorded. A method of selectively displaying an image of a specific person by collating is disclosed.
Japanese Patent Laid-Open No. 1-171097 JP 2004-128615 A

  However, even if a camera image selection method typified by the above prior art is used, it is not possible to select and display an image in which the user can easily recognize the movement and shape of the limbs of a person.

  This is because the camera image selection method represented by Patent Document 1 uses a motion vector to select a camera image based on whether the magnitude of the motion vector exceeds a predetermined threshold. In this case, regardless of whether or not the moving object present in the video is a person, the camera video is selected when a temporal change occurs in the video. For this reason, the image is not necessarily easy to recognize the movement and shape of the person's limbs.

  In addition, when a camera image in which a person is present in the image is selected using the person detection method, and a camera image in which a person's face is present in the image is selected using a face detection technique represented by Patent Document 2. In some cases, even if it is possible to select a camera image in which a person is a subject, it is not always possible to select an image in which the movement and shape of the person's limbs can be easily recognized.

  In addition, since a person is an articulated object, a change in posture becomes a problem unlike an object such as a car or a train that does not change its shape. That is, the ease of recognizing the movement and shape of the limbs varies depending on the posture. For this reason, it is difficult to uniquely determine the relative positional relationship between the camera and the subject in advance. Therefore, in order to obtain an image in which the user can easily recognize the movement and shape of the person's limbs, it is necessary to determine the relative positional relationship between the camera and the subject for each posture and movement.

  In order to obtain an image that can easily recognize the movement and shape of a person's limbs, it is desirable that the person's limbs be an image that is not hidden by body parts such as the trunk. Therefore, it is particularly necessary to evaluate the appearance of a person's limbs.

  The present invention has been made to solve the above-described problem, and provides a video selection method capable of selecting a video from which a user can easily recognize the movement and shape of a person's limb from a plurality of camera videos. The purpose is to provide.

  In order to achieve the above object, a video selection method according to the present invention is a video selection method for selecting at least one video from a plurality of videos taken by a plurality of cameras, which is picked up by a plurality of cameras. A video acquisition step for acquiring a plurality of videos including a person and a feature extraction for extracting feature quantities obtained from a plurality of representative points provided on the contour lines of the subject silhouette of the person included in each of the videos An unevenness degree determining step for determining an unevenness degree indicating the degree of unevenness of the shape of the subject silhouette from the feature amount for each of the plurality of images, and the plurality of images based on the unevenness degree of the plurality of images. A video selection step of selecting at least one video for recognizing the shape of the hand or foot of the person from the video.

  The present invention can be realized not only as a video selection method including such characteristic steps, but also as a video selection device using the characteristic steps included in the video selection method as a means. The characteristic steps included in the selection method can be realized as a program for causing a computer to execute the steps. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a communication network such as the Internet.

  ADVANTAGE OF THE INVENTION According to this invention, the image | video selection method which can select the image | video which a user can recognize easily the motion and shape of a person's limb from several camera images can be provided.

  A video selection method according to the present invention is a video selection method for selecting at least one video among a plurality of videos captured by a plurality of cameras, and a plurality of videos including a person captured by a plurality of cameras. A video acquisition step of acquiring, a feature amount extraction step of extracting a feature amount obtained from a plurality of representative points provided on a contour line of a subject silhouette of a person included in each of the plurality of videos; An unevenness degree determining step for determining an unevenness degree indicating the unevenness degree of the shape of the subject silhouette from the feature amount for each, and the human hand or foot from the plurality of images based on the unevenness degree of the plurality of images. A video selection step of selecting at least one video for recognizing the shape of the video. More specifically, in the image selection step, an image having the maximum unevenness is selected.

  According to this configuration, the degree of unevenness of the shape of the subject silhouette in an image including a person is determined. For this reason, it is possible to select an image in which the user can easily recognize the shape of the limb of the person.

  Preferably, in the feature amount extraction step, a distance between a predetermined reference point and the plurality of representative points is extracted as a feature amount.

  By using the representative points on the contour line, the shape of the subject silhouette can be expressed more efficiently than when all the pixel values of the subject silhouette image are used.

  More preferably, the unevenness degree determination step includes a curve generation step of generating a curve by connecting a plurality of feature quantities at the plurality of representative points, a maximum value detection step of detecting a maximum value in the curve, And an unevenness degree calculating step for calculating the unevenness degree based on the local maximum value.

  Thus, by using the number of local maximum values in the curve, it is possible to express the shape of the subject silhouette that strongly reflects the posture of the person's limbs.

  More preferably, the unevenness degree determination step includes a curve generation step of generating a curve by connecting a plurality of feature quantities at the plurality of representative points, a maximum value detection step of detecting a maximum value in the curve, An associating step for associating the local maximum value with a person's hand or foot based on the local maximum value and the number of local maximum values, and an unevenness degree for calculating the unevenness level based on the result of the association and the local maximum value Calculation step. More specifically, in the unevenness degree calculating step, based on the result of the association, one of a local maximum value associated with a hand and a local maximum value associated with a foot is weighted, and the unevenness Calculate the degree.

  By associating the limbs and weighting the maximum value, it becomes possible to select an image while paying attention to the hand or foot. For example, if video selection focusing on the hand is performed, it can be used for shoplifting spot monitoring.

  More preferably, in the unevenness degree determining step, for each of the plurality of videos, a time average of unevenness degrees of a plurality of temporally continuous images is calculated, and the calculation result is calculated as the unevenness degree of the subject silhouette shape. To do.

  In this way, by setting the time average of the unevenness as a new unevenness, it is possible to select an image in which the user can easily recognize the shape and movement of the currently operating limb.

  More preferably, in the video selection step, video images of the subject taken from the opposite direction to the video images are simultaneously selected on the basis of the selected video images. Further, in the video selection step, the video that is capturing the subject may be simultaneously selected from the direction orthogonal to the optical axis direction of the camera that is capturing the video with reference to the selected video.

  Selecting an image captured from another direction makes it easier for the user to recognize the shape and movement of the limb than selecting an image captured from one direction. Further, by selecting a video imaged from another direction, a video image showing a human face can be selected.

  More preferably, the image selection method further includes a face image detection step of detecting an image including a face image of a person from the plurality of images captured by the plurality of cameras. In the image selection step, Further, at least one video for recognizing the person's face is selected from the plurality of videos based on the detection result of the video including the face image.

  According to this method, it is possible to select an image in which the user can easily recognize the face image and the shape of the limbs of the person at the same time.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
First, the video display apparatus according to Embodiment 1 of the present invention will be described. The video display apparatus according to Embodiment 1 selects and displays a video from which a user can easily recognize the shape of a person's limb from videos taken by a plurality of cameras.

  FIG. 1 is a diagram showing a configuration of a video display apparatus according to Embodiment 1 of the present invention. The video display apparatus of FIG. 1 is an apparatus that selects and displays an image in which a user can easily recognize the shape of a person's limb by determining the degree of unevenness indicating the degree of unevenness of the shape of a subject silhouette in an image including a person. And an imaging unit 101, an image holding unit 102, a feature extraction unit 103, a shape determination unit 104, and a camera selection unit 105.

  The imaging unit 101 includes a plurality of cameras 101a to 101h, is a processing unit that captures a plurality of videos, and is an example of a video acquisition unit.

  The image holding unit 102 is a storage device that holds a plurality of images captured by the imaging unit 101. Here, the images held by the image holding unit 102 may be images arranged in time series.

  The feature extraction unit 103 is a processing unit that extracts a contour feature amount representing the contour shape of the subject from the image held in the image holding unit 102, and is an example of a feature amount extraction unit.

FIG. 2 is a diagram illustrating a detailed configuration of the feature extraction unit 103.
As shown in FIG. 2, the feature extraction unit 103 includes a person region extraction unit 1031, a silhouette extraction unit 1032, and a contour feature extraction unit 1033.

  The person area extraction unit 1031 is a processing unit that extracts a person area from the image held in the image holding unit 102. The silhouette extraction unit 1032 is a processing unit that extracts a subject silhouette by performing binarization processing so that the pixel value of the extracted person region is “1” and the others are “0”. The contour feature extraction unit 1033 is a processing unit that extracts a feature amount related to the contour of the subject silhouette with respect to the subject silhouette.

  The shape determination unit 104 is a processing unit that determines the unevenness degree of the shape of the subject silhouette obtained by the silhouette extraction unit 1032 from the contour feature amount extracted by the contour feature extraction unit 1033, and is an example of an unevenness degree determination unit. This unevenness degree is a value that largely reflects the appearance of a person's limbs.

  The camera selection unit 105 selects the shape of a person's limb from a plurality of camera images based on the degree of unevenness of the shape of the subject silhouette determined by the shape determination unit 104 from the plurality of cameras 101a to 101h. Is a processing unit that selects a camera video that is easy to recognize, and is an example of video selection means.

  The display unit 106 is a processing unit that displays the video selected by the camera selection unit 105 on a monitor or the like.

  Thereby, it is possible to select and display an image in which the user can easily recognize the shape of the limb of the person.

  Hereinafter, a video display method by the video display apparatus according to the present embodiment will be described in detail with reference to FIGS.

  FIG. 3 is a flowchart of processing executed by the video display apparatus. FIG. 4 is a diagram for explaining a contour feature amount extraction method.

  First, the person area extraction unit 1031 receives an input image 401 as shown in FIG. 4A from the image holding unit 102 (S301).

  Next, the person area extraction unit 1031 performs background difference processing on the input image 401 using a background image 402 as shown in FIG. 4B to extract a person area (S302). Here, instead of the background difference process, the inter-frame difference process may be performed to extract the person area, or the person area may be extracted using the area division method. Furthermore, when using human detection processing together, M.Oren, C.Papageorgiou, P.Sinha, E.Osuna and T.Poggio, "Pedestrian Detection using wavelet templates", Proc.of CVPR97, pp.193-199 , 1997 may be used to perform person detection processing, and the result may be used to cut out a person region. Furthermore, edge detection processing may be used in combination. In addition, when performing the background difference process, it is necessary to prepare an image as a background without a person in advance and hold it in the image holding unit 102. Furthermore, a background sprite image can be generated from a moving image and used as a background image for background difference processing.

  Next, the silhouette extraction unit 1032 performs a binarization process on the image including the person area extracted in S302 so that the extracted pixel of the person area is an on-pixel and the other is an off-pixel (S303). ). As a result, a subject silhouette 403 of a person image as shown in FIG. 4C is obtained. Here, the subject silhouette 403 is an example in which the hatched portion is an on-pixel portion and the white portion is an off-pixel portion.

  Note that it is also effective to smooth the subject silhouette in order to reduce unnecessary irregularities in the contour caused by the influence of noise.

  Next, the contour feature extraction unit 1033 extracts the contour of the subject silhouette (S304). Contour extraction is possible by extracting boundary points between on-pixels and off-pixels. If the binarization process is not performed in S303, the contour can be extracted by applying a spatial differential filter to the subject silhouette. Further, when thinning the extracted contour, it can be performed using the thinning algorithm of Hilditch. Details of the spatial differential filter and the Hilitch thinning algorithm are described in C. J. Hilditch, “Linear Skeletones From Square Cupboards”, Machine Intelligence 4, Eginburgh Univ. Press, p. 403, 1969.

  Next, the contour feature extraction unit 1033 extracts a contour feature amount expressing the contour shape of the subject silhouette from the extracted contour of the subject silhouette (S305). An example of the contour feature amount extraction method will be described below with reference to FIGS. The contour feature amount is not limited to the following example as long as it is a feature amount expressing the contour shape of the subject silhouette.

  First, the contour feature extraction unit 1033 determines a reference point for the subject silhouette 403 extracted in S304. As a desirable example, as shown in FIG. 4D, the gravity center position g of the subject silhouette is obtained as a reference point. Here, the reference point may be the center of gravity of the contour pixel position of the subject silhouette without using all the pixels of the subject silhouette.

  Next, the contour feature extraction unit 1033 calculates the main axis 405 of the subject silhouette. The main axis 405 of the subject silhouette can be obtained by setting the axis that minimizes the sum of squares of the lengths of the perpendiculars when the perpendicular is drawn from each contour point on the silhouette.

Specifically, the main axis 405 to be obtained is a straight line having an inclination tan θ and passes through the center of gravity (g _x , g _y ), and can be expressed as follows.

で表される。 Next, when a perpendicular is drawn from each contour point (x _i , y _i ) to the straight line represented by (Equation 1), the length d _{i of the} perpendicular is

It is represented by

Therefore, the function J to be minimized is obtained by taking the sum of squares of the lengths of the perpendicular lines drawn from the contour points, and is expressed as follows using (Equation 1) and (Equation 2). The

より、

となる。 Here, in order to minimize (Equation 3)

Than,

It becomes.

From (Equation 5), the slope of the straight line can be expressed by (Equation 6). Then, the main axis 405 can be obtained by substituting the slope of the straight line into (Equation 1).

  Next, the contour feature extraction unit 1033 obtains the intersection point between each point on the contour of the subject silhouette and the main axis 405 in order to determine the starting point of the contour feature value. Here, when there are a plurality of intersection points 407 between each point on the contour and the obtained main axis, as shown in FIG. Let it be the start point 406. At this time, the start point 406 does not necessarily need to select the contour point located at the top. However, in particular, when the target is a person, it is effective to set the uppermost contour point intersecting with the main axis 405 as the start point 406. The reason is that the starting point 406 can easily correspond to the head. Specifically, the head is unlikely to deform unlike the arms and legs. Moreover, since the influence of a shadow is easy to come out with respect to the ground, it is mentioned that it is hard to receive the influence.

  Next, the contour feature extraction unit 1033 traces each point on the contour of the subject silhouette from the start point 406. Contour point tracing methods are described in detail in “Takeharu Aoi, Tomoharu Nagao,“ Processing and Recognition of Images ”, page 72, published by Shosho-do, 1992”.

  Next, the contour feature extraction unit 1033 calculates a distance l between each point on the traced contour and the center of gravity g, as shown in FIG. Here, the contour feature amount is expressed as a vector having a distance l between each point on the contour of the subject silhouette and the center of gravity as an element, as indicated by the contour feature amount 503 in FIG. Note that symbols A to I in FIG. 5 indicate a part of representative points on the contours of the subject silhouettes shown in the images 501 and 502, and corresponding points in the contour feature values.

  Here, as shown in FIG. 6A, a case where the number of contour points varies depending on the posture of the subject and the size in the image will be described. In this case, in each subject silhouette, a subject silhouette having n contour points and a subject silhouette having m contour points are obtained. Here, n <m.

  In this example, the number of elements of the contour feature quantity vector is n and m. Therefore, in order to prevent the number of vector elements of the contour feature quantity from differing, it is desirable that the number of vector elements is determined in advance and set to a fixed value Z. Specifically, as shown in FIG. 6B, representative points on the contour points to be used as the contour feature amount from all the contour points so that the number of contour points is Z for an arbitrary input image. Select every (n / Z) points or every (m / Z) points. In the present embodiment, a feature amount having the contour point selected in this way as a representative point is set as a contour feature amount.

Here, an example of the outline feature amount will be described. As shown in FIG. 5, after extracting a subject silhouette for each video obtained by the imaging unit 101 including a plurality of cameras 101 a to 101 h, an outline feature quantity 503 is extracted. The contour feature value l _c is composed of a vector having as an element the distance l between the center of gravity g of the subject silhouette and the representative point on the contour used as the contour feature value. Furthermore, in order to eliminate the influence of the difference in the size of the subject, the area (number of pixels) S of the subject silhouette may be calculated and normalized with the area S as in the following equation.

Here, l _{c — z} indicates the distance between the center of gravity g of the subject silhouette obtained from the image captured by the camera c and the representative point z on the contour. S _c is the area of the subject silhouette obtained from the image captured by the camera c. It is also effective to apply a low-pass filter to the contour feature value obtained in this way. This has the same effect as smoothing the subject silhouette as described above.

Next, the shape determination unit 104 calculates the degree of unevenness d _c of the resulting subject silhouette shape from the outline feature quantity l _c extracted for each image obtained by the camera c the contour feature extraction unit 1033 with the silhouette extraction unit 1032 (S306). Specifically, the degree of unevenness can be calculated by taking the difference along the contour point from the contour feature quantity l _c (contour feature quantity 503) as shown in FIG.

  A square value may be used instead of the absolute value, or a second-order difference may be performed instead of the first-order difference. Thus, for example, in the case of FIG. 5A, d = 4.10, and in the case of FIG. 5B, d = 5.81. As described above, when the limbs of a person are expressed in the silhouette, the unevenness degree of the silhouette shape is larger than in the case where the limbs of the person are not.

Further, when the case of determining the irregularities of the object silhouette shape from the series images to T images string obtained by the camera c, by calculating the time average D _c of asperity d _c, time-averaged D _c may be a new degree of unevenness.

Here, d _c (t) is the degree of unevenness obtained from the image at time t captured by the camera c.

  Further, when a plurality of persons are captured by one camera, the degree of unevenness is calculated based on (Equation 8) or (Equation 9) for each subject silhouette, and the calculated value is divided by the number of subjects. An average value of the unevenness per subject silhouette may be obtained, and the average value may be used as the unevenness. Further, the sum of a plurality of unevenness levels obtained from a plurality of subject silhouettes may be used as the unevenness level.

  Next, the camera selection unit 105 selects a camera image based on the unevenness degree of the subject silhouette shape calculated in S306 (S307).

  Here, the case where the time average of the unevenness degree of the subject silhouette shape is used as in (Equation 9) will be described, but the same applies to the case of (Equation 8).

  An example in which a certain area is imaged by a plurality of cameras 101a to 101h as shown in FIG. 7 will be described. Of course, each camera may be imaged by a cameraman as in sports broadcasting.

In order to obtain a camera image in which the user can easily recognize the shape of a person's limb, it is effective to select a camera image that maximizes D _c in (Equation 9). It is also possible to select the top N camera videos based on the value of D _c . Here, N is a natural number that is equal to or less than the number of cameras.

Further, in the present embodiment, since the contour feature amount is estimated from the subject silhouette, it may be difficult to distinguish between the front and back of the subject, such as the chest side and the back side of the subject. To avoid such a case, based on the camera image selected based on the value of D _c, it is installed in a position facing the imaging direction of the selected camera, also effective to select the captured camera images It is. Furthermore, it is possible to select a camera image in a direction orthogonal to the imaging direction of the selected camera.

  Also, as shown in FIG. 8 among all the installed cameras, a set of a plurality of cameras is determined in advance as A to D, and when one of the camera images is selected, the set becomes a set. One camera image may be selected.

  Next, the display unit 106 displays the video selected in S307 on the monitor 901 as shown in FIG. 9 (S308). Specifically, the image 902 selected based on the unevenness degree of the subject silhouette shape in S307 is displayed on the monitor 901. Here, N images may be displayed in descending order of unevenness.

  In addition to the selected image 902, a camera image 903 that is installed and imaged at a position facing the imaging direction of the camera is displayed. At this time, it is also possible to display the camera arrangement 906 and the installation position of the camera 904 capturing the currently displayed video 902 and the camera 905 installed at the position facing the camera.

  Note that the video display method is not limited to the above example as long as the video selected based on the unevenness degree of the subject silhouette shape in S307 is displayed.

  As described above, an image in which the user can easily recognize the shape of a person's limb can be automatically selected from images obtained by simultaneously capturing a scene including a person with a plurality of cameras. Therefore, it is not necessary for the supervisor or director to watch all the videos obtained from the plurality of cameras, and the number of camera videos to be watched can be reduced, thereby reducing the burden.

(Modification of Embodiment 1)
Next, a description will be given of a method of extracting a contour feature amount by selecting a representative point on the contour point of the subject silhouette by a method different from that of the first embodiment.

  The configuration of the video display device is the same as that shown in FIGS. The processing executed by the video display device is basically the same as the flowchart shown in FIG. 3, but the contour feature amount extraction method in the contour feature amount extraction process (S305) is different. Accordingly, the outline feature amount used in S306 to S308 is also different from that in the first embodiment. Other processes are the same as those described in the first embodiment.

Hereinafter, the outline feature amount extraction process (S305) will be described in detail.
FIG. 10 is a diagram for explaining a method for determining a representative point when extracting a contour feature amount.

  The method for determining the center of gravity g of the subject silhouette and the method of obtaining the subject silhouette principal axis 1002 are the same as those described in the first embodiment. Further, the method for obtaining the start point 1003 of the contour feature quantity is the same as that described in the first embodiment.

  Next, the contour feature extraction unit 1033 draws a straight line in a clockwise direction or a counterclockwise direction at a certain angle around the center of gravity g as an arrow in FIG. Ask. Then, the distance l between the reference point and the representative point is calculated using the center of gravity g as the reference point.

  At this time, as shown by points A and B in FIG. 10, when there are two or more intersections between the straight line drawn from the center of gravity g at a certain angle and the subject silhouette, the longer distance from the center of gravity g Is used as a representative point.

The contour feature amount can be defined as the following equation instead of (Equation 7) using the distance between the reference point and the representative point obtained in the above procedure.

Here, l _{c — y} indicates the distance between the reference point (center of gravity g) set on the main axis 1002 of the subject silhouette extracted from the video obtained by the camera c and the representative point 1004. Note that _Sc is the area of the subject silhouette extracted from the video obtained by the camera c.

  It is also effective to apply a low-pass filter to the contour feature value obtained in this way. This has the same effect as smoothing the subject silhouette as described above.

  Next, since the processing from S306 to S308 can be realized by using (Equation 10) instead of (Equation 7), detailed description thereof will not be repeated.

  The contour feature amount used in the present modification determines the representative point based on the angle of the straight line drawn from the center of gravity, so the number of contour points does not depend on the shape or size of the subject silhouette as shown in FIG. Therefore, as shown in FIG. 11, there is no difference in the number of contour points between the case where the foot portion of the contour extracted image is connected to the ground and the case where the foot portion is not connected to the ground due to the influence of the shadow generated on the foot portion. It is relatively insensitive to shadows.

(Embodiment 2)
Next, a video display apparatus according to Embodiment 2 of the present invention will be described.

  The video display apparatus according to the second embodiment calculates the unevenness degree of the subject silhouette in the shape determination unit 104 by a method different from the method described in the first embodiment.

  The configuration of the video display apparatus according to the second embodiment is the same as that shown in FIG. However, the internal configuration of the shape determination unit 104 is different from that of the first embodiment. Other than that, imaging unit 101, image holding unit 102, feature extraction unit 103, camera selection unit 105, and display unit 106 are the same as those in the first embodiment, and therefore, description thereof will not be repeated here.

FIG. 12 is a block diagram illustrating an internal configuration of the shape determination unit 104.
The shape determination unit 104 is a processing unit that determines the degree of unevenness of the subject silhouette shape obtained by the silhouette extraction unit 1032 from the contour feature amount extracted by the contour feature extraction unit 1033, and includes a peak detection unit 1041 and an limb determination unit 1042 and an unevenness degree determination unit 1043. This unevenness degree largely reflects the appearance of a person's limbs.

  The peak detection unit 1041 detects the peak position and the number of peaks of the contour feature amount extracted by the contour feature extraction unit 1033.

  The limb determination unit 1042 associates a hand or a foot with the peak position of the contour feature value detected by the peak detection unit 1041.

  The unevenness degree determination unit 1043 determines the unevenness degree of the subject silhouette shape by using the hand and foot information associated with the peak position of the contour feature value.

  Note that in this embodiment, an example in which the unevenness determination unit 1043 determines the unevenness by adding hand or foot information in the limb determination unit 1042 will be described, but the limb determination unit 1042 is not used. In the unevenness degree determination unit 1043, the peak number of the contour feature amount may be used as the unevenness degree of the subject silhouette.

  Hereinafter, a video display method by the video display apparatus according to the present embodiment will be described. FIG. 13 is a flowchart of processing executed by the video display apparatus according to this embodiment.

  Since the processing from S301 to S305 is the same as that in the first embodiment, description thereof is omitted.

  As shown in FIG. 14A, the peak detection unit 1041 detects the peak position of the contour feature amount extracted by the feature extraction unit 103 (S1301). Specifically, a point having a maximum value is selected by taking a difference in the contour feature amount along the contour point. That is, the point where the difference value changes from positive to negative is set as the peak position. A portion indicated by a black circle in FIG. 14 is a representative point on the contour point of the peak detection result 1402 and the corresponding object silhouette 1401 in the contour feature value.

  In S1301, as in the first embodiment, unnecessary irregularities can be reduced by detecting the peak position after applying a low-pass filter to the contour feature quantity.

Next, the limb determination unit 1042 labels each peak position detected in S1301 with a hand or a foot (S1302). Here, as shown in FIG. 14B, the contour feature quantities 1403 corresponding to the respective peak positions are set to l _p1 , l _{p2 ,.}

  Specifically, as shown in FIG. 15, cases can be classified according to the number of peaks detected in S1301, and a labeling method for hands and feet can be determined.

When the number of peaks is one, a foot is associated with the peak position.
When the number of peaks is two, it can be divided into a case where feet are associated with all peak positions and a case where hands and feet are associated with peak positions.

  When the number of peaks is three, it can be divided into a case where two feet and one hand are associated with each peak position, and a case where one foot and two hands are associated with each peak position.

When the number of peaks is 4, any one of the limbs is associated with each peak position.
As described above, it is determined which of the four limbs corresponds to the number of peaks.

Next, a method for labeling limbs to each peak position will be described.
When the number of peaks is 1, as shown in FIG. 15, if the head is the start point, the end point is also the head. Therefore, the position where the peak exists can be uniquely determined as a foot. When a peak is present at a position corresponding to the hand, two or more peaks are present.

When the number of peaks is two or more, contour feature quantities l _p1 , l _{p2 ,.} Here, M is the number of peaks. Hereinafter, for purposes of explanation, the outline feature quantity of the peak position and _{l p1> l p2>...>} l pM.

のように、決定する。すなわち、ｌ_p1／ｌ_p2の値が所定の閾値αよりも大きい場合には、ｌ_p1を足の輪郭特徴量と判断し、ｌ_p2を手の輪郭特徴量と判断する。それ以外の場合には、ｌ_p1およびｌ_p2がともに足の輪郭特徴量であると判断する。 When the number of peaks is 2, using the ratio of the contour feature quantity at the peak position,

To decide. That is, when the value of l _p1 / l _p2 is larger than the predetermined threshold value α, l _p1 is determined as the contour feature amount of the foot, and l _p2 is determined as the contour feature amount of the hand. In other cases, it is determined that l _p1 and l _p2 are both foot contour feature values.

When the number of peaks is three, it is determined that the maximum contour feature value l _{p1 at} the peak position is the contour feature amount of the foot, and the minimum contour feature value l _p3 is determined to be the contour feature amount of the hand. Also, which of the following corresponds to the intermediate value l _p2 of the contour feature value is determined according to (Equation 12). That is, if the ratio (l _p1 / l _p2 ) between the maximum contour feature value l _p1 and the intermediate value l _p2 of the contour feature value is larger than a predetermined threshold value α, it is determined that l _p2 is the contour feature value of the foot, In other cases, it is determined that l _p2 is the contour characteristic amount of the hand.

When the number of peaks is four, it is determined that l _p1 and l _p2 are the contour feature amounts of the feet, and l _p3 and l _p4 are determined as the contour feature amounts of the hand.

  Here, in (Equation 11) and (Equation 12), α = 1.5 was experimentally set. However, the value of the threshold value α is not limited to this, and varies depending on the installation position of the camera, the target scene, and the like.

Also, the method of associating the limb with the peak position of the contour feature quantity is not limited to the above example.
For example, hands and feet may be distributed based on the size of the contour feature amount at the peak position. If it can be assumed that the subject is standing on the ground, a corresponding point on the contour point of the subject silhouette is obtained for each peak position of the contour feature amount as in the example of FIG. It is also possible to label as a foot when the hand is close to the ground and as a hand when it is close to the center of gravity in the vertical axis direction on the image.

  Next, the unevenness degree determination unit 1043 calculates the unevenness degree of the subject silhouette shape using the label information associated in S1302 (S1303).

That is, irregularity determination unit 1043, using the outline feature quantity of labeled peak position for each image obtained by the camera c, calculating the asperity d _c of the object silhouette shape as the number 13.

  Here, M corresponds to the number of peaks of the outline feature quantity, that is, the number of peaks of the labeled outline feature quantity.

In addition, when it is necessary to pay attention to hand movements such as shoplifting movements in a surveillance system, or when importance is attached to movements of hands even in sports relaying etc., by weighting the contour feature amount labeled as a hand, It is possible to define the degree of unevenness d that places importance on the hand in the subject silhouette shape.

は、手としてラベル付けされた輪郭特徴量であり、

は足としてラベル付けされた輪郭特徴量である。このため、手の見えを重視した重み付け緒を行なうためには、１＞β＞０とすればよい。また、β＞１とすれば、足の見えを重視した凹凸度となる。 Here, K represents the number of peaks of the contour feature value labeled as a hand, and L represents the peak number of the contour feature value labeled as a foot. Also,

Is a contour feature labeled as a hand,

Is an outline feature labeled as a foot. For this reason, in order to perform weighting with emphasis on the appearance of the hand, 1>β> 0 may be set. Further, if β> 1, the degree of unevenness with an emphasis on the appearance of the foot is obtained.

  A mode in which the user can set such a value of β may be prepared in the video display device.

  Further, as shown in (Equation 9), it is also effective to calculate the time average of the unevenness for the T image rows obtained by the camera c.

  Using the unevenness degree of the subject silhouette shape obtained as described above, the processing of S307 and S308 is executed as in the first embodiment.

  As described above, according to the video display apparatus according to the present embodiment, a video that allows a user to easily recognize the shape of a person's extremities is automatically selected from videos obtained by simultaneously capturing scenes including a person with a plurality of cameras can do. Therefore, it is not necessary for the supervisor or director to watch all the videos obtained from the plurality of cameras, and the number of camera videos to be watched can be reduced. For this reason, it is possible to reduce the burden of video surveillance for the supervisor and director.

  Further, in addition to the first embodiment, since it is possible to select a video with a weight on the appearance of hands or feet, it is possible to select a camera video according to the application, such as sports broadcasting or monitoring.

(Embodiment 3)
Next, a video display apparatus according to Embodiment 3 of the present invention will be described.

  In addition to the video display method of the video display device shown in the first and second embodiments, the video display device according to the third embodiment simultaneously displays a camera video including a face image.

FIG. 16 is a block diagram showing a configuration of the video display apparatus according to Embodiment 3.
The video display device determines the degree of unevenness of the shape of the subject silhouette in an image including a person, thereby selecting an image that allows the user to easily recognize the shape of the limb of the person, performing face image detection processing, and detecting the detected face An apparatus that simultaneously displays a camera image including an image and the selected camera image, and includes an imaging unit 101, an image holding unit 102, a feature extraction unit 103, a shape determination unit 104, a face image detection unit 1601, A camera selection unit 1602 and a display unit 1603 are provided.

  The imaging unit 101, the image holding unit 102, the feature extraction unit 103, and the shape determination unit 104 are the same as those described in the first or second embodiment. The detailed configuration of the feature extraction unit 103 is the same as that shown in FIG.

  The face image detection unit 1601 detects a face image from the video extracted by the person area extraction unit 1031.

  The camera selection unit 1602 captures an image that allows the user to easily recognize the shape of the person's limb based on the degree of unevenness of the shape of the subject silhouette determined by the shape determination unit 104 from among a plurality of cameras. Select. At the same time, the camera selection unit 1602 selects a camera image capturing an image including the face image according to the face image detection result in the face image detection unit 1601.

A display unit 1603 displays the video selected by the camera selection unit 1602 on a monitor or the like.
Thereby, it is possible to select and display an image in which the user can easily recognize the shape and face of the person's limbs. Of course, it is also possible to record the selected video on a recording device (not shown).

  Hereinafter, a video display method by the video display apparatus according to the present embodiment will be described with reference to the flowchart of FIG. Here, the video selection method that allows the user to easily recognize the shape of the limbs of the person is described as being the same as in the second embodiment, but the video is selected in the same manner as in the first embodiment. You may do it.

  The processes of S301 to S305 and S1301 to S1303 are the same as in the second embodiment. Therefore, detailed description thereof will not be repeated.

  The face image detection unit 1601 performs face image detection processing on the person region extracted in S302 (S1701). Face image detection processing is described in KKSung and T. Poggio, "Example-Based Learning for View-Based Human Face Detection", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.20, No.1, pp.39-51, Techniques disclosed in 1998 can be used. In this method, a face image model is constructed by learning the face image in advance.

  Here, an example using the above technique will be described, but other face image detection methods may be used. When it is desired to select a camera image that mainly captures a front image or a face image close to the front, a face image captured from the front prepared in advance is used for model learning.

  Next, the camera selection unit 1602 selects a camera image based on the unevenness degree of the subject silhouette shape calculated in S1303 and the face image detection result performed in S1701 (S1702).

  Here, description will be made on the assumption that the time average of the unevenness degree of the subject silhouette shape is calculated using (Equation 9), but the same applies to the case where the unevenness degree is calculated based on (Equation 8).

  As in the first embodiment, description will be made using an example in which a certain region is imaged by a plurality of cameras as shown in FIG.

In order to obtain a camera image in which the user can easily recognize the shape of a person's limb, it is effective to select a camera image that maximizes D _c in (Equation 9). It is also possible to select the top N camera videos based on the value of D _c . Here, N is a natural number that is equal to or less than the number of cameras.

  In addition, the camera selection unit 1602 also selects the camera video from which the face image is detected (S1702).

  When the camera image selected based on the unevenness degree of the subject silhouette shape calculated in S1303 and the camera image selected based on the face image detection result performed in S1701 are different, the camera selection unit 1602 Select both images. Of course, the present invention is not limited to this example, and all camera images from which face images are detected may be selected. However, when there is a limitation in video communication capacity or the like in monitoring or the like, if a plurality of face images are detected, the camera video can be selected based on the size of the detected face images.

  In addition, if the camera video selected based on the unevenness degree of the subject silhouette shape and the camera video selected based on the face image detection result are the same, the camera selection unit 1602 selects a camera video that satisfies both. .

  Next, the display unit 1603 displays the video selected in S1702 as shown in FIG. 18 (S1703). Specifically, the monitor 1801 displays the video 1802 selected based on the unevenness degree of the subject silhouette shape in S1702 and the camera video 1803 in which the face image is detected. Note that, based on the face detection result, the face position may be enclosed by a square or the like, or the face image may be enlarged and displayed. At this time, if the installed camera has a zoom function, the face image may be zoomed and imaged.

  Here, N images may be displayed in descending order of the degree of unevenness, or all images from which face images are detected may be displayed. Further, if the camera video selected based on the unevenness degree of the subject silhouette shape is the same as the camera video selected based on the face image detection result, a camera video that satisfies both may be displayed. At this time, it is also possible to display the installation position of the camera 1804 selected based on the unevenness degree of the subject silhouette image that is the currently displayed camera and the camera 1805 that detected the face image, together with the camera arrangement 1806. .

  It is also possible to simultaneously record the video selected and displayed as described above in a recording device (not shown).

  Note that the video display method is not limited to the above example as long as the video selected based on the unevenness degree of the subject silhouette shape and the face image detection result in S1702 is displayed.

  As described above, according to the video display device according to the present embodiment, a video that allows a user to easily recognize the shape of a person's limb and the face of a person from a video obtained by simultaneously capturing a scene including the person with a plurality of cameras. Can be selected automatically. Therefore, it is not necessary for the supervisor or director to watch all the videos obtained from the plurality of cameras, and the number of camera videos to be watched can be reduced, thereby reducing the burden. Further, in particular for monitoring and other uses, it is also effective in proving crimes to simultaneously grasp a person's movement and face image.

  The video display device according to the embodiment of the present invention has been described above, but the present invention is not limited to this embodiment.

For example, in the above-described embodiment, the unevenness degree of the silhouette shape of the subject is calculated according to (Equation 8) or (Equation 9), but the unevenness degree may be calculated according to (Equation 17) shown below. good.

That is, the absolute value of the difference in the degree of unevenness in the time direction may be calculated for the T image sequences obtained by the camera c, and the added value may be used as the new unevenness degree. 19, the five image sequence obtained by the camera c, is a diagram schematically showing an example of obtaining a new degree of unevenness D _c. The graph shown in FIG. 19 shows the unevenness degree d _c (t) obtained from five image sequences from time t = 1 to time t = 5, and from two adjacent image sequences. The absolute value of the difference is calculated between the obtained unevenness degrees d _c (t + 1) and d _c (t), and the sum of them is the new unevenness degree D _c .

  According to the present invention, it is possible to select an image in which a user can easily recognize the shape of a person's limb. Further, by using the time information of the unevenness degree of the subject silhouette shape, it is possible to select an image in which the user can easily recognize the movement of the person's limbs. This eliminates the need for the user to pay attention to all the images obtained from the plurality of cameras, and can reduce the number of camera images to be watched, thereby reducing the user's monitoring burden. Therefore, the present invention can be applied to a shoplifting site monitoring apparatus using a plurality of cameras, a video selection apparatus for selecting an appropriate scene from a plurality of cameras, and the like.

It is a figure which shows the structure of the video display apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the structure of the feature extraction part which concerns on Embodiment 1 of this invention. It is a flowchart of the process which the video display apparatus which concerns on Embodiment 1 of this invention performs. It is a figure for demonstrating the extraction method of the outline feature-value which concerns on Embodiment 1 of this invention. It is a figure for demonstrating the extraction method of the outline feature-value which concerns on Embodiment 1 of this invention. It is a figure which shows the representative point of the outline feature-value which concerns on Embodiment 1 of this invention. It is a figure which shows the example of installation of the several camera which concerns on Embodiment 1 of this invention. It is a figure which shows the example of a combination of the several camera which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the video display which concerns on Embodiment 1 of this invention. It is a figure for demonstrating the extraction method of the outline feature-value which concerns on the modification of Embodiment 1 of this invention. It is a figure which shows the outline extraction result which concerns on the modification of Embodiment 1 of this invention. It is a figure which shows the structure of the shape determination part which concerns on Embodiment 2 of this invention. It is a flowchart of the process which the video display apparatus concerning Embodiment 2 of this invention performs. It is a figure which shows the example of a peak detection of the outline feature-value which concerns on Embodiment 2 of this invention. It is a figure which shows the example of matching of the limb which concerns on Embodiment 2 of this invention. It is a figure which shows the structure of the video display apparatus concerning Embodiment 3 of this invention. It is a flowchart of the process which the video display apparatus concerning Embodiment 3 of this invention performs. It is a figure which shows the example of the video display which concerns on Embodiment 3 of this invention. It is a figure which shows typically the other calculation methods of the unevenness | corrugation degree which concern on Embodiment 1-3 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Image pick-up part 102 Image holding part 103 Feature extraction part 104 Shape determination part 105,1602 Camera selection part 106,1603 Display part 1031 Person area extraction part 1032 Silhouette extraction part 1033 Contour feature extraction part 1041 Peak detection part 1042 Limb judgment part 1043 Irregularity Degree determination unit 1601 Face image detection unit

Claims (13)

A video selection method for selecting at least one video from a plurality of videos captured by a plurality of cameras,
A video acquisition step of acquiring a plurality of videos including a person imaged by a plurality of cameras;
A feature amount extraction step of extracting a feature amount obtained from a plurality of representative points provided on a contour line of a person's subject silhouette included in each of the plurality of videos;
An unevenness degree determining step for determining an unevenness degree indicating the degree of unevenness of the shape of the subject silhouette from the feature amount for each of the plurality of images,
And a video selection step of selecting at least one video for recognizing the shape of the person's hand or foot from the plurality of videos based on the degree of unevenness of the plurality of videos. The video selection method according to claim 1, wherein in the video selection step, a video having the maximum unevenness is selected. The video selection method according to claim 1, wherein in the feature amount extraction step, a distance between a predetermined reference point and the plurality of representative points is extracted as a feature amount. The unevenness degree determining step includes:
A curve generation step of generating a curve by connecting a plurality of feature quantities at the plurality of representative points;
A maximum value detecting step for detecting a maximum value in the curve;
The video selection method according to claim 3, further comprising: calculating a degree of unevenness based on the maximum value. The unevenness degree determining step includes:
A curve generation step of generating a curve by connecting a plurality of feature quantities at the plurality of representative points;
A maximum value detecting step for detecting a maximum value in the curve;
An association step of associating a person's hand or foot with the local maximum based on the local maximum and the number of local maximums;
The video selection method according to claim 3, further comprising: a concave / convex degree calculating step of calculating the concave / convex degree based on a result of association and the maximum value. The unevenness degree calculating step calculates the unevenness degree by weighting either the maximum value associated with the hand or the maximum value associated with the foot based on the result of the association. 5. The video selection method according to 5. 2. The unevenness determination step further calculates, for each of the plurality of videos, a time average of the unevenness of a plurality of temporally continuous images, and sets the calculation result as the unevenness of the subject silhouette shape. The video selection method according to any one of -6. In the unevenness degree determining step, for each of the plurality of videos, a difference value in the time direction of the unevenness degree of the plurality of temporally continuous images is calculated, and the calculation result is set as the unevenness degree of the subject silhouette shape. The video selection method according to claim 1. The video selection method according to any one of claims 1 to 8, wherein in the video selection step, videos selected from a direction opposite to the video are simultaneously selected on the basis of the selected video. 9. The video selection step simultaneously selects, based on the selected video, images capturing the subject from a direction orthogonal to the optical axis direction of the camera capturing the video. 9. Item 2. The video selection method according to item 1. The video selection method further includes a face image detection step of detecting a video including a face image of a person from the plurality of videos captured by the plurality of cameras.
The video selection step further selects at least one video for recognizing the person's face from the plurality of videos based on a detection result of the video including a face image. The video selection method according to item. A video selection device that selects at least one video from a plurality of videos captured by a plurality of cameras,
Video acquisition means for acquiring a plurality of videos including a person captured by a plurality of cameras;
Feature amount extraction means for extracting feature amounts obtained from a plurality of representative points provided on the contour line of the subject silhouette of a person included in each of the plurality of videos;
For each of the plurality of videos, an unevenness degree determining means for determining an unevenness degree indicating the unevenness degree of the shape of the subject silhouette from the feature amount;
A video selection device comprising: video selection means for selecting at least one video for recognizing the shape of the person's hand or foot from the plurality of videos based on the degree of unevenness of the plurality of videos. A program of a video selection method for selecting at least one video from a plurality of videos captured by a plurality of cameras,
A video acquisition step of acquiring a plurality of videos including a person imaged by a plurality of cameras;
A feature amount extraction step of extracting a feature amount obtained from a plurality of representative points provided on a contour line of a person's subject silhouette included in each of the plurality of videos;
An unevenness degree determining step for determining an unevenness degree indicating the degree of unevenness of the shape of the subject silhouette from the feature amount for each of the plurality of images,
A program that causes a computer to execute an image selection step of selecting at least one image for recognizing the shape of the hand or foot of the person from the plurality of images based on the degree of unevenness of the plurality of images.

Images