JP6615709B2 - Foreground region extraction apparatus, method, and program - Google Patents

Description

  The present invention relates to a foreground area extraction apparatus, method, and program, and in particular, a foreground area extraction apparatus, method, and program for extracting a foreground area representing an object from an input image based on a difference from a background image. About.

  In recent years, with the increase in the number of surveillance cameras installed, attention has been focused on services that automatically understand the meaning of video using image processing technology and reduce the cost of manual monitoring. To understand the meaning of a video, for example, processing such as detecting an object or person and recognizing the state of the detected object or person is necessary. However, if the entire image is scanned for all frames, the calculation cost increases. There is a problem that it ends up. Therefore, in the surveillance camera image, the camera is fixed at a fixed position for a long time, and by using the premise that the environmental objects hardly change, only the object / person part (foreground) entering and exiting the image is preprocessed. The approach of narrowing down the processing target is often used.

Z. Zivkovic and F. van der Heijden, Efficient adaptive density estimation per image pixel for the task of background subtraction, Pattern Recognit. Lett., Vol. 27, pp. 773-780, 2006

  The foreground area is obtained by taking a difference between an image obtained at a certain time and an image (background image) in which only a separately prepared background environment is shown. Due to the nature of such processing, there is a problem in that a shadow that is not an object of interest is included in the foreground area simply by taking a difference. As means for solving this, there is a method of estimating a shadow part by sequentially constructing a background using a Gaussian mixture model (see Non-Patent Document 1). However, in the technique as described in Non-Patent Document 1, although the illumination is indirectly applied to the subject and the shadow can be detected well in an environment where the shadow becomes thin, the sunlight is directly applied to the subject and the shadow In a dark and clear environment, shadows cannot be detected accurately.

  The present invention has been made in view of the above circumstances, and provides a foreground region extraction apparatus, method, and program capable of accurately extracting a foreground region even in an environment where sunlight directly strikes a subject. The purpose is to provide.

  In order to achieve the above object, a foreground region extraction device according to a first aspect of the present invention is a foreground region extraction device that extracts a foreground region representing an object based on a difference from a background image from an input image. A virtual foreground region generation unit that generates a virtual foreground region including a virtual region of the object or a virtual shadow region of the object based on the determined length of the object; and the background image from the image A foreground region extraction unit that extracts a real foreground region representing the object, a virtual foreground region generated by the virtual foreground region generation unit, and a real foreground region extracted by the foreground region extraction unit. Based on the overlapping area, an area matching unit that generates a map image that represents the object-like area or the object-like shadow area, and the map image generated by the area matching unit. , Wherein the real foreground region extracted by the foreground region extraction unit is configured to include a a selective foreground region processing unit for obtaining the foreground area.

  Further, in the foreground region extraction device according to the first invention, the virtual foreground region generation unit calculates the virtual region of the object based on a predetermined size of the object, and the virtual The virtual foreground region including the region of the object is generated, and the region matching unit overlaps the virtual foreground region generated by the virtual foreground region generation unit and the real foreground region extracted by the foreground region extraction unit. The selective foreground region processing unit is extracted by the foreground region extraction unit based on the map image generated by the region matching unit. A foreground area obtained by extracting the object area from the actual foreground area may be obtained.

  The foreground region extraction device according to the first aspect of the present invention further includes a date and time acquisition unit that acquires the date and time when the image was captured, wherein the virtual foreground region generation unit is a direction of the camera that captured the image obtained in advance. And the virtual foreground region including the shadow region of the object by calculating the shadow region of the object at the date and time based on the installation position, the date and time acquired by the date and time acquisition unit, and the length of the object And the region matching unit seems to be a shadow of the object based on a region where the virtual foreground region generated by the virtual foreground region generation unit overlaps the real foreground region extracted by the foreground region extraction unit. The map image representing an area is generated, and the selective foreground area processing unit extracts the map image based on the map image generated by the area matching unit. From the actual foreground regions which are, it may be obtained the foreground region to remove the region of the shadow.

  In the foreground region extraction device according to the first invention, the region matching unit obtains a degree of coincidence obtained by an area ratio of regions where the virtual foreground region and the real foreground region overlap each time the virtual foreground region is scanned. The map image may be generated by calculating.

  A foreground region extraction method according to a second invention is a foreground region extraction method in a foreground region extraction apparatus that extracts a foreground region representing an object from an input image based on a difference from a background image. A step of generating a virtual foreground region including a virtual region of the object or a virtual shadow region of the object based on a predetermined length of the object; and a foreground region extraction unit, A step of extracting a real foreground region representing the object based on a difference from the background image from an image; and a region matching unit including a virtual foreground region generated by the virtual foreground region generation unit and the foreground region extraction unit. A step of generating a map image representing an area that is likely to be the object or an area that is likely to be a shadow of the object based on an area that overlaps the extracted real foreground area; and a selective foreground area processing unit, On the basis of the map image generated by the serial area matching unit, wherein the real foreground region extracted by the foreground region extraction unit, and executes comprise the steps of obtaining the foreground area.

  Further, in the foreground region extraction method according to the second invention, the step of generating the virtual foreground region generator calculates the virtual region of the object based on a predetermined size of the object, The step of generating the virtual foreground area including the virtual object area and generating by the area matching unit is extracted by the virtual foreground area generated by the virtual foreground area generation unit and the foreground area extraction unit. The step of generating the map image representing the object-like area based on the area that overlaps the actual foreground area, and the selective foreground area processing unit obtaining the foreground area includes the map generated by the area matching unit. A foreground area obtained by extracting the object area from the actual foreground area extracted by the foreground area extraction unit may be obtained based on the image.

  Further, in the foreground region extraction method according to the second invention, the date and time acquisition unit further includes a step of acquiring the date and time when the image was captured, and the step of generating the virtual foreground region generation unit includes the previously obtained step Based on the direction and installation position of the camera that captured the image, the date and time acquired by the date and time acquisition unit, and the length of the object, the shadow area of the object at the date and time is calculated, and the shadow of the object is calculated. The step of generating the virtual foreground region including the region and generating the region matching unit includes: a virtual foreground region generated by the virtual foreground region generation unit; and a real foreground region extracted by the foreground region extraction unit. The step of generating the map image representing a shadow-like area of the object based on the overlapping area and the selective foreground area processing unit obtaining a foreground area includes the area matching On the basis of the map image generated by the parts, from the actual foreground region extracted by the foreground region extraction section, may be obtained the foreground region to remove the region of the shadow.

  A program according to the third invention is a program for causing a computer to function as each part of the foreground region extraction device according to the first invention.

  According to the foreground region extraction apparatus, method, and program of the present invention, a virtual foreground region including a virtual object region or a virtual object shadow region is generated based on a predetermined object length. The real foreground area representing the object is extracted from the image by the difference from the background image, and the object-like area or object shadow is based on the area where the generated virtual foreground area and the extracted real foreground area overlap. A foreground area is accurately generated even in an environment where sunlight directly hits the subject by generating a map image that represents a distinct area and obtaining the foreground area from the extracted actual foreground area based on the map image The effect that it can extract is acquired.

It is a block diagram which shows the structure of the foreground area | region extraction apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example which looked at the virtual person from the top, and an example of the virtual foreground area | region produced | generated from the point obtained by projecting the position of the both ends of a person. It is a figure which shows an example of the search frame with respect to a real foreground area | region. It is a figure which shows an example of the scanning of a virtual foreground area | region. It is a figure which shows an example of the production | generation of a mask image, and extraction of a foreground area | region. It is a flowchart which shows the foreground area extraction processing routine in the foreground area extraction apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the virtual foreground area | region production | generation routine based on the 1st Embodiment of this invention. It is a flowchart which shows the area | region matching processing routine which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the selective foreground area | region processing routine which concerns on embodiment of this invention. It is a block diagram which shows the structure of the foreground area | region extraction apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example which looked at the virtual person from the top, and an example of the virtual foreground area | region produced | generated from the point obtained by projecting the position of the both ends of a person, and the position of the both ends of the said person's shadow. It is a figure which shows an example of the scanning of a virtual foreground area | region. It is a flowchart which shows the foreground area extraction processing routine in the foreground area extraction apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the virtual foreground area | region production | generation routine based on the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline according to Embodiment of the Present Invention>

  First, an outline of the embodiment of the present invention will be described. In an embodiment of the present invention, in order to solve the above-described problem, based on environment calibration information such as camera calibration information performed at the time of camera installation and the direction of the camera and the latitude / longitude of the installation location, The foreground area is virtually generated by simulating the ratio of the lengths of the shadows, or by simulating the ratio of the shadow lengths and the direction in which the shadows appear, and the virtual foreground area and the actual foreground area are It is characterized in that processing for selectively obtaining a human region is performed by comparing. In the present embodiment, a case where a foreground region of a person is obtained as an object will be described. However, the present invention is not limited to this, and other objects (for example, animals, cars, etc.) may be used.

  By using the method of the present embodiment, it is obtained by the difference processing between an image acquired at a certain time and a background image in an environment where sunlight directly hits and shadows are dark and distinct, such as in an outdoor environment. An object region can be selectively left from the foreground region including the shadow.

<Configuration of Foreground Area Extraction Device According to First Embodiment of the Present Invention>

  Next, the configuration of the foreground region extraction apparatus according to the first embodiment of the present invention will be described. As shown in FIG. 1, a foreground area extraction apparatus 100 according to an embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a foreground area extraction processing routine to be described later, Can be configured with a computer including Functionally, the foreground region extraction apparatus 100 includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG.

  The input unit 10 receives an image shot by a camera installed at a fixed position. The input unit 10 receives calibration information related to the camera.

  The calculation unit 20 includes a calibration information storage unit 26, a camera information DB 28, a video acquisition unit 30, a shooting information acquisition unit 32, an illumination state determination unit 34, an existing foreground region extraction unit 36, and a virtual foreground region generation. A unit 40, a foreground region extraction unit 42, a region matching unit 44, and a selective foreground region processing unit 46.

  The calibration information storage unit 26 stores the camera calibration information received by the input unit 10 in the camera information DB 28. The camera calibration is performed when the surveillance camera is installed. For example, the calibration is performed by a method as described in Non-Patent Document 2. By calibration, camera internal parameters such as the distance between the camera and the floor, the angle of the camera with respect to the floor, and the focal length / distortion coefficient can be acquired. Further, in addition to the calibration information, the direction of the camera shooting direction and the latitude / longitude information of the camera installation position are separately acquired as calibration information. The calibration information storage unit 26 stores the acquired various types of calibration information in the camera information DB 28.

[Non-Patent Document 2] I. Miyagawa, H. Arai and H. Koike, Simple camera calibration from a single image using five points on two orthogonal 1-d objects, IEEE Trans. Image Process., Vol. 19, no. 6 , pp. 1528-1538, 2010.

  The video acquisition unit 30 acquires each of the images at each time for the video received by the input unit 10. It is assumed that a camera that can acquire information necessary for the shooting information acquisition unit 32 is used for image acquisition.

  The shooting information acquisition unit 32 acquires information related to exposure from each of the images acquired by the video acquisition unit 30. In the present embodiment, information related to exposure is acquired. For example, as information for estimating the illumination state in the illumination state determination unit 34 such as a backlight correction value and weather information available from the Internet. Other information that can be used may be obtained.

  The illumination state determination unit 34 determines whether or not the environment has a dark shadow that is difficult to estimate with existing technology. Therefore, based on the information obtained by the imaging information acquisition unit 32, bright light such as direct sunlight is emitted. It is determined whether or not the subject is hit. For example, the exposure state, the luminance value, and the illumination state information are accumulated in advance, and the determination can be made by a method of estimating the illumination state from the exposure state and the luminance value.

  As a result of the illumination state determination means, when it is determined that bright light is not hitting the subject (a cloudy state or the like, a dark shadow does not appear), the existing foreground region extraction unit 36 using the existing method is used. For each, a foreground region representing an object is extracted based on the difference from the background image using an existing shadow estimation method as in Non-Patent Document 1, and is output to the output unit 50. On the other hand, if it is determined that bright light is hitting the subject, the processing after the virtual foreground region generation unit 40 is performed.

  The virtual foreground area generation unit 40 calculates a virtual person area based on the predetermined size of the person and the calibration information stored in the camera information DB 28, and calculates the virtual person area. Generate a virtual foreground region that contains First, the virtual foreground region generation unit 40 uses the calibration information stored in the camera information DB 28 such as the distance between the camera and the floor, the angle, the focal length, and the distortion parameters to detect a person from the virtual camera position. The positions of both ends of the person are projected on the projection plane assumed to be taken. Next, based on the positions of both ends of the projected person, the person's area is represented by a volume approximated by a rectangle to generate a virtual foreground area. In the present embodiment, the human region is approximated by a rectangle, but may be approximated by another figure such as an ellipse. FIG. 2 shows an example of a virtual person viewed from above, and an example of a virtual foreground area generated from points obtained by projecting the positions of both ends of the person.

  The foreground area extraction unit 42 extracts an actual foreground area representing a person from each of the images acquired by the video acquisition unit 30 based on a difference from the background image. For extraction of the actual foreground area, an existing method may be used as in the existing foreground area extraction unit 36.

  As will be described below, the region matching unit 44 is based on a region in which a human region in the virtual foreground region generated by the virtual foreground region generation unit 40 and the real foreground region extracted by the foreground region extraction unit 42 overlap. Then, a map image representing a person-like area is generated.

The area matching unit 44 first calculates a frame (search frame) that is expanded by adding an offset o to a frame that touches each block of the actual foreground region. FIG. 3 shows an example of the search frame. Next, for each pixel (i, j), the degree of coincidence between the virtual foreground area and the actual foreground area when the virtual foreground area is scanned over the entire area inside the calculated search frame is calculated. The offset o may be arbitrarily specified, or a parameter learned so as to improve the accuracy of the calculation result may be used. For the pixel (i, j) at the position in the actual foreground region within the search frame, the integrated value S _h (i, j) of the degree of coincidence with the person is calculated. Thereby, a map image representing a human-like area can be obtained. The degree of coincidence is the area ratio of the area where the virtual foreground area and the real foreground area overlap with the area of the virtual foreground area, and is obtained each time the virtual foreground area is scanned. FIG. 4 shows an example of scanning of the virtual foreground area. Now, in the search frame, when the degree of coincidence when the upper left coordinate of the virtual foreground region is the coordinate (k, l) in the real foreground region is m (k, l), the pixel (i, The integrated value of the degree of coincidence in j) is calculated by the following equation (1).

Here, C represents a set of (k, l) in which the pixel (i, j) is included in the overlapping portion of the virtual foreground area and the real foreground area, and is obtained for each pixel (i, j). H represents a set of coordinates of the real foreground region that overlaps the human region in the virtual foreground region when the upper left coordinate of the virtual foreground region is at (k, l). The integrated value S _h obtained by the calculation, a map image S _h which represents the human likely area is generated. In addition, since the size of the reflection of the person varies depending on the image position, the degree of coincidence is calculated while enlarging / reducing the virtual foreground area in accordance with the position of the actual foreground area to be compared. The enlargement / reduction ratio is obtained by calculating the number of pixels on the image when the lower end of the region of the object having the height h comes to a certain image position using the parameters stored in the camera DB 28. be able to. In the present embodiment, the degree of coincidence m (i, j) is the area ratio, but the degree of coincidence may be calculated using another index. As for the integrated value S _h, an example has been shown to take a simple sum, such as divided by the number of integrating summing may be used another method.

  As described below, the selective foreground region processing unit 46 extracts an object region from the actual foreground region extracted by the foreground region extraction unit 42 based on the map image generated by the region matching unit 44. Get the foreground area.

In selective foreground region processing unit 46, first, among the map image S _h which represents the person likely area obtained by the area matching unit 44, 1 pixel positions of more than the threshold T, the following pixel position serving as the threshold value as 0, A mask image is generated. The generated mask image S _h, overlaps the real foreground area, only the location which is a 1 S _h, performs processing to leave the foreground area. As a result, it is possible to selectively obtain only the foreground region having a high probability as a human region. FIG. 5 shows an example of mask image generation and foreground region extraction.

<Operation of Foreground Area Extraction Device According to First Embodiment of the Present Invention>

  Next, the operation of the foreground region extraction apparatus 100 according to the first embodiment of the present invention will be described. When the input unit 10 receives a video taken by a camera installed at a fixed position and calibration information about the camera and stores them in the camera information DB 28, the foreground area extraction device 100 extracts the foreground area shown in FIG. Execute the processing routine.

  First, in step S100, each image at each time is acquired for the video received by the input unit 10.

  Next, in step S102, information related to exposure is acquired from each of the images acquired by the video acquisition unit 30.

  In step S104, it is determined based on the information obtained in step S102 whether or not bright light such as direct sunlight is hitting the subject. If it is determined that bright light is not hitting the subject, the process proceeds to step S106. If it is determined that bright light is hitting the subject, the process proceeds to step S108.

  In step S106, a foreground region representing an object is extracted from each image using a difference from the background image using an existing shadow estimation method such as that described in Non-Patent Document 1, and is output to the output unit 50 for processing. finish.

  In step S108, a virtual foreground area including a virtual person area is calculated based on a predetermined person size and calibration information stored in the camera information DB 28. Is generated.

  In step S110, an actual foreground region representing a person is extracted from each of the images acquired in step S100 based on the difference from the background image.

  In step S112, for each of the images, each time the virtual foreground region is scanned based on the region where the human region of the virtual foreground region generated in step S108 overlaps the real foreground region extracted in step S110, By calculating the degree of coincidence obtained by the area ratio of the area where the virtual foreground area and the real foreground area overlap with the area of the virtual foreground area, a map image representing an object-like area is generated.

  In step S114, the foreground area obtained by extracting the object area is obtained from the actual foreground area extracted in step S110 based on the map image generated in step S112 for each of the images, and is output to the output unit 50. The process is terminated.

  Step S108 is executed by the virtual foreground area generation processing routine shown in FIG.

  In step S200, it is assumed that a person is photographed from a virtual camera position using calibration information stored in the camera information DB 28 such as the distance between the camera and the floor, the angle, the focal length, and the distortion parameter. Project the positions of both ends of the person on the projection plane.

  In step S202, based on the positions of both ends of the projected person, the person's area is represented by a volume approximated by a rectangle, a virtual foreground area is generated, and the virtual foreground area generation processing routine ends.

  Step S112 is executed by the area matching processing routine shown in FIG.

  In step S300, an image is selected, and an extended frame (search frame) is calculated by adding an offset o to a frame that touches each block of the real foreground region in the image.

In step S302, each time a virtual foreground region is scanned by selecting a block of the real foreground region, the degree of coincidence calculated by the area ratio of the region where the virtual foreground region and the real foreground region overlap with the area of the virtual foreground region is calculated. For each pixel (i, j), according to the above equation (1), the integrated value S _h (i, j) of the degree of coincidence with the human region is obtained, and a map image representing the human-like region is generated.

  In step S304, it is determined whether or not processing has been completed for all chunks of the real foreground area. If not, the process returns to step S302 to select the next chunk of real foreground area, repeat the processing, and end. If yes, the process proceeds to step S306.

  In step S306, it is determined whether or not the processing has been completed for all the images. If not, the process returns to step S300 to select the next image and repeat the processing. If the processing has been completed, the region matching processing routine is executed. finish.

  Step S114 is executed by the selective foreground area processing routine shown in FIG.

At step S400, selects an image, generated for the image, of the map image S _h which represents the human likely region, 1 pixel positions of more than the threshold T, the following pixel position serving as the threshold value as 0, the mask image Generate.

In step S402, the mask image S _h generated in step S400, overlaps the real foreground region of the image, by performing the process of leaving the foreground area of only the position which is a 1 S _h, the foreground region And output to the output unit 50.

  In step S404, it is determined whether or not the processing has been completed for all images. If not, the process returns to step S400 to select the next image and repeat the processing. If the processing has been completed, selective foreground region processing is performed. End the routine.

  As described above, according to the foreground region extraction apparatus according to the first embodiment of the present invention, a virtual foreground region including a virtual object region is generated based on a predetermined object length. Then, a real foreground area representing a person is extracted from the image based on a difference from the background image, and a map image representing a person-like area based on an area where the generated virtual foreground area and the extracted real foreground area overlap , And obtaining the foreground area from the extracted actual foreground area based on the map image, the foreground area can be accurately extracted even in an environment where sunlight directly strikes.

<Configuration of Foreground Area Extraction Device According to Second Embodiment of the Present Invention>

  Next, the configuration of the foreground region extraction apparatus according to the second embodiment of the present invention will be described. In addition, about the location which becomes the same structure as 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The second embodiment is different from the first embodiment in that a virtual foreground area including a shadow-like area is generated and a foreground area is obtained.

  As shown in FIG. 10, the foreground area extraction apparatus 200 according to the embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a foreground area extraction processing routine described later, Can be configured with a computer including Functionally, the foreground region extraction apparatus 200 includes an input unit 10, a calculation unit 220, and an output unit 50 as shown in FIG.

  The calculation unit 20 includes a calibration information storage unit 26, a camera information DB 28, a video acquisition unit 30, a shooting information acquisition unit 32, an illumination state determination unit 34, an existing foreground region extraction unit 36, and a date acquisition unit 238. A virtual foreground region generation unit 240, a foreground region extraction unit 42, a region matching unit 244, and a selective foreground region processing unit 246.

  The date acquisition unit 238 acquires the date and time when the image was captured. The date and time when the image was taken may be obtained from the system time of a computer synchronized with the camera installation location via a network (not shown), or when using an IP camera or the like, the time of the camera device itself May be obtained.

  The virtual foreground area generation unit 240, based on the direction and installation position of the camera that captured the image obtained in advance, the date and time acquired by the date and time acquisition unit 238, and the length of the person, A human shadow region at the date and time is calculated, and a virtual foreground region including a virtual human region and a virtual human shadow region is generated. The virtual foreground region generation unit 240 first calculates the altitude and direction of the sun at the time and location from the information on the direction and installation position (latitude / longitude) of the camera and information on the date and time. In the present embodiment, latitude / longitude is used, but information on an installation position where the location can be known with a granularity of about the city may be used. For the altitude and direction of the sun, for example, data published by the National Astronomical Observatory of Japan and the like are used. When the altitude of the sun is α and the direction of the sun is obtained, the length of the shadow of the object of length h is

And the shadow length at that date and time can be obtained. Next, using the calibration information such as the predetermined person size and the camera-to-floor distance, angle, focal length, and distortion parameters stored in the camera information DB 28, the direction of the virtual camera Then, the positions of both ends of the person and the positions of both ends of the shadow of the person are projected on the projection plane assumed to shoot a virtual person and shadow from the installation position. Next, a human and shadow area is approximated by a rectangle to represent a volume, and this is used as a virtual foreground area. In the present embodiment, the human area is approximated by a rectangle, but may be approximated by another figure such as an ellipse. FIG. 11 shows an example of a virtual person and shadow viewed from above, and an example of a virtual foreground area generated from the points obtained by projecting the positions of both ends of the person and both ends of the shadow of the person.

  As described below, the region matching unit 244 is a region in which the shadow region of the virtual foreground region generated by the virtual foreground region generation unit 40 overlaps the real foreground region extracted by the foreground region extraction unit 42. Based on the above, a map image representing a shadow-like area is generated.

The area matching unit 244 first performs a process similar to that of the first embodiment, and adds a frame (search frame) that is expanded by adding an offset o to a frame that touches each block of the actual foreground region. calculate. Next, for each pixel (i, j) at a position in the actual foreground region within the search frame, an integrated value S _s (i, j) of the degree of coincidence with the shadow is calculated. As a result, a map image representing a shadow-like area can be obtained. The degree of coincidence is the area ratio of the area where the virtual foreground area and the real foreground area overlap with the area of the virtual foreground area, and is obtained each time the virtual foreground area is scanned. FIG. 12 shows an example of scanning. Now, in the search frame, when the degree of coincidence when the upper left coordinate of the virtual foreground region is the coordinate (k, l) in the real foreground region is m (k, l), the pixel (i, The integrated value of the degree of coincidence in j) is calculated by the following equation (3).

Here, D represents the set of coordinates of the real foreground area that overlaps the shadow area in the virtual foreground area when the upper left coordinate of the virtual foreground area is at (k, l), and for each pixel (i, j). Desired. Based on the integrated value S _s obtained by the calculation, a map image S _s representing a shadow-like region is generated.

  As described below, the selective foreground region processing unit 246 extracts an object region from the actual foreground region extracted by the foreground region extraction unit 42 based on the map image generated by the region matching unit 44. Get the foreground area.

In the selective foreground region processing unit 246, first, in the map image S _s representing the shadow-like region obtained by the region matching unit 44, the pixel position equal to or higher than the threshold T is set to 1, and the pixel position equal to or lower than the threshold is set to 0. A mask image is generated. The generated mask image S _s, overlaps the real foreground area, performs a process of removing the foreground region of only the position that is the 1 S _s. As a result, it is possible to selectively obtain only the foreground region having a high probability as a human region.

  Note that other configurations of the second embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

<Operation of Foreground Area Extraction Device According to Second Embodiment of Present Invention>

  Next, the operation of the foreground region extraction apparatus 100 according to the second embodiment of the present invention will be described. In addition, about the location which becomes the effect | action similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  When the input unit 10 receives a video photographed by a camera installed at a fixed position and calibration information relating to the camera and stores them in the camera information DB 28, the foreground region extraction device 200 performs the foreground region extraction shown in FIG. Execute the processing routine.

  In step S104, it is determined based on the information obtained in step S102 whether or not bright light such as direct sunlight is hitting the subject. If it is determined that bright light is not hitting the subject, the process proceeds to step S106. If it is determined that bright light is hitting the subject, the process proceeds to step S500.

  In step S500, for each image, the date and time when the image was captured is acquired.

  In step S502, based on the direction and installation position of the camera that captured the image stored in the camera information DB 28, the date and time acquired in step S500, and the length of the person, the area of the virtual person and the person at the date and time And a virtual foreground area including a virtual person area and a virtual person shadow area is generated.

  In step S504, a map image representing a shadow-like area is generated based on an area where the shadow area in the virtual foreground area generated in step S500 overlaps the real foreground area extracted in step S110.

  In step S506, for each image, based on the map image generated in step S504, the foreground region obtained by extracting the object region from the real foreground region extracted in step S110 is obtained and output to the output unit 50. The process is terminated.

  Step S502 is executed by the virtual foreground area generation processing routine shown in FIG.

  In step S600, the altitude of the sun at that time and place is selected based on the direction and installation position (latitude / longitude) of the camera that selected the image and obtained in advance, and the date and time acquired in step S500. And calculate the direction.

  In step S602, based on the altitude and direction of the sun calculated in step S600 and the person's length h, the direction in which the shadow appears is calculated from the relationship between the latitude / longitude of the camera and the direction of the sun. Also, the shadow length is calculated according to the above equation (2). Then, the positions of both ends of the shadow are determined based on the calculated direction and length of the shadow.

  In step S604, using the calibration information stored in the camera information DB 28 such as the distance between the camera and the floor, the angle, the focal length, and the distortion parameter, a virtual person is determined from the direction and the installation position of the virtual camera. The positions of both ends of the person and the positions of both ends of the shadow of the person are projected on the projection plane assuming that the shadow is photographed.

  In step S604, based on the positions of both ends of the projected person and the positions of both ends of the person's shadow, the person and the shadow area are represented by a volume approximated by a rectangle to generate a virtual foreground area.

  In step S606, it is determined whether or not the processing has been completed for all the images. If not, the process returns to step S600 to select the next image and repeat the processing. If the processing has been completed, the virtual foreground region generation processing is performed. End the routine.

In step S504, the process in step S302 of the region matching process routine shown in FIG. 8 is replaced with the following and executed. In step S302, each time a virtual foreground region is scanned by selecting a block of the real foreground region, the degree of coincidence calculated by the area ratio of the region where the virtual foreground region and the real foreground region overlap with the area of the virtual foreground region is calculated. For each pixel (i, j), the integrated value S _s (i, j) of the degree of coincidence with respect to the shadow area is obtained according to the above equation (3), and a map image representing the shadow-like area is generated.

Step S506 is executed by replacing steps S400 and S402 of the selective foreground region processing routine shown in FIG. 9 with the following. In step S400, a mask image is selected by selecting an image and setting a pixel position equal to or higher than the threshold T to 1 and a pixel position equal to or lower than the threshold to 0 in the map image S _s generated for the image and representing a shadow-like area. Generate.

In step S402, the mask image S _s generated in step S400 is overlapped with the actual foreground region of the image, and the foreground region is removed only at the position where S _s is 1, thereby performing the foreground region. And output to the output unit 50.

  In addition, since the other effect | action of 2nd Embodiment is the same as that of 1st Embodiment, description is abbreviate | omitted.

  As described above, according to the foreground region extraction device according to the second embodiment of the present invention, the direction and installation position of the camera that captured the image obtained in advance, the date and time acquired by the date and time acquisition unit, Calculate the shadow area of the object at the date and time based on the length of the person, generate a virtual foreground area that includes the virtual shadow area, and from the image, the real foreground area that represents the object by the difference from the background image Based on the area where the generated virtual foreground area overlaps with the extracted real foreground area, and generates a map image representing a shadow-like area, and from the extracted real foreground area based on the map image By obtaining the foreground area, the foreground area can be accurately extracted even in an environment where sunlight directly strikes.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

For example, in the first and second embodiments described above, the foreground area is obtained by using either the map image S _h representing a human-like area or the map image S _s representing a shadow-like area. However, the present invention is not limited to this. For example, the first and second embodiments are combined to generate both a map image S _h representing a human-like area and a map image S _s representing a shadow-like area, and the selective foreground area processing unit For each of the calculated map images S _h representing human-like regions and map images S _s representing shadow-like regions, which map image fits the actual foreground region by a statistical method. calculated, if you choose from the calculation results to use any of the map image, and select a map image S _h which represents the human likely region, only the position that is the 1 S _h, leaving the foreground area, When a foreground area is obtained and a map image S _s representing a shadow-like area is selected, the foreground area may be obtained by removing the foreground area only at the position where S _s is 1. .

  In the first and second embodiments described above, the illumination state determination unit 34 determines whether bright light is hitting the subject, generates a virtual foreground area as necessary, and obtains the foreground area. Although processing was performed, it is not limited to this. For example, as a configuration that does not include the shooting information acquisition unit 32, the illumination state determination unit 34, and the existing foreground region extraction unit 36, an image that has been determined in advance by an external device that bright light has hit the subject is input. A process of receiving, generating a virtual foreground area, and obtaining the foreground area may be performed.

DESCRIPTION OF SYMBOLS 10 Input part 20,220 Calculation part 26 Calibration information preservation | save part 30 Image | video acquisition part 32 Shooting information acquisition part 34 Illumination state determination part 36 Existing foreground area extraction part 40,240 Virtual foreground area generation part 42 Foreground area extraction part 44,244 Region matching unit 46, 246 Selective foreground region processing unit 50 Output unit 100, 200 Foreground region extraction device 238 Date and time acquisition unit

Claims (4)

A foreground region extraction device that extracts a foreground region representing an object from an input image based on a difference from a background image,
A date and time acquisition unit for acquiring the date and time when the image was captured;
Based on the direction and installation position of the camera that captured the image obtained in advance , the date and time acquired by the date and time acquisition unit, and the predetermined length of the object, and the direction and installation position of the camera, and Projecting a shadow calculated from the altitude of the object and the object onto the projection plane assumed from the direction and installation position of the camera with respect to the altitude and direction of the sun calculated from the date and time, A virtual foreground region generating unit that generates a virtual foreground region including the region of the object and a virtual shadow region of the object;
A foreground region extracting unit that extracts a real foreground region representing the object from the image based on a difference from the background image;
A map image representing a region that is likely to be the object or a region that is likely to be a shadow of the object, based on a region where the virtual foreground region generated by the virtual foreground region generation unit and the real foreground region extracted by the foreground region extraction unit overlap. A region matching unit for generating
A selective foreground region processing unit that obtains the foreground region obtained by removing the shadow region from the actual foreground region extracted by the foreground region extraction unit based on the map image generated by the region matching unit;
A foreground region extraction apparatus. The said area matching part produces | generates the said map image by calculating the coincidence degree calculated | required by the area ratio of the area | region where the said virtual foreground area | region and the said real foreground area | region overlap each time the said virtual foreground area | region is scanned. foreground region extraction apparatus according to 1. A foreground region extraction method in a foreground region extraction device that extracts a foreground region representing an object from an input image based on a difference from a background image,
The date and time acquisition unit acquires the date and time when the image was captured,
Virtual foreground area generation unit, camera and direction and the installation position of the captured the image previously obtained, and the date and time acquired by the date and time acquisition section, based on the length of the object to a predetermined, the camera And the shadow calculated from the altitude for the object and the object with respect to the projection plane assumed from the direction of the camera and the installation position with respect to the altitude and direction of the sun calculated from the date and time and the date and time of the sun. Projecting to generate a virtual foreground region including a virtual region of the object and a virtual shadow region of the object;
A foreground area extracting unit extracting an actual foreground area representing the object by a difference from the background image from the image;
The area matching unit is based on an area where the virtual foreground area generated by the virtual foreground area generation unit and the real foreground area extracted by the foreground area extraction unit overlap, or the area that seems to be the object or the shadow of the object Generating a map image representing the region;
A selective foreground region processing unit obtains the foreground region obtained by removing the shadow region from the actual foreground region extracted by the foreground region extraction unit based on the map image generated by the region matching unit. Steps,
Foreground region extraction method including: A program for causing a computer to function as each unit of the foreground region extraction device according to claim 1 or 2 .