JP4218368B2 - Human detection method and human detection device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method and apparatus for detecting the presence of a person, the position of a person, and the number of persons indoors by performing image processing on an observed image.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, the movement of people and the distribution of people in an observation area have been detected for the purpose of observation of human work and behavior, automatic control of the environment, and monitoring of intrusion of suspicious objects. The main conventional techniques for human detection include those using pyroelectric element sensors and those using image processing. In the case of using a pyroelectric element sensor, the presence or absence of a person in the observation region is detected by detecting infrared rays emitted from the human body using the pyroelectric element sensor. When detecting the location of a person, it is not possible to determine where a person exists by using only a single sensor. Therefore, it is necessary to measure using multiple sensors, which takes time to install the sensor. There is. In addition, since this method detects a person by detecting a temperature change in the observation space, it cannot be detected when the person is stationary.
[0003]
One that uses the image to detect people is one that uses the background subtraction method. The background subtraction method is a method of detecting a difference portion by taking a difference between a background image stored in advance and an input image. In this method, if the background image does not change, the changed part can be detected with high accuracy. However, in the actual use environment, when the ambient light changes, or an object is brought into or taken out of the observation field, False detection occurs. In order to cope with these environmental changes, automatic background image updating is generally performed.
[0004]
For example, in order to automatically update the background image, for example, to detect the number and distribution of people in the monitoring area as an indoor monitoring device, the background of the image captured in the monitoring area by specifying the person monitoring area in advance The person is detected based on the difference, and whether the person is a person is determined by the sum of the absolute values in the monitoring area, and the background image is updated when the change history of the background difference absolute value exceeds a predetermined amount. Is known (see, for example, Patent Document 1).
[0005]
In addition, a moving body identification device is known that improves the accuracy of detection of moving objects by performing background updating and excluding those that repeatedly operate at a specific position such as a door (movable background) (for example, , See Patent Document 2). This apparatus detects a moving object based on background difference, and when the difference value does not exceed the threshold value continuously, the image portion is set as a moving object. When the difference value equal to or greater than the threshold value continues for a certain period, the background image is updated assuming that the background has changed. Further, when the detected moving object shape repeatedly appears, it is distinguished from the moving object by determining that the moving object is a movable background.
[0006]
In addition, a person detection device that automatically updates the background when the surrounding situation changes and detects the number of passengers waiting in the elevator hall and the number of people in the elevator, and an elevator control system using the same are known ( For example, see Patent Document 3). In this apparatus, the moving object is detected based on the background difference. The area where the difference value exceeds the threshold is divided by the standard person area, or the number of pixels detected as moving objects is counted to determine the number of persons. If you have two background images with different background update timings and update the background image to update the background used for background difference while referring to the old background as seen from the current image, once you have been captured in the background It can prevent that the part which has not moved and is detected as a ghost. In addition, the threshold for human detection binarization is automatically adjusted by comparing the detection by area and the number of detections by counting the lump.
[0007]
In addition to the above example, in order to deal with the problem that a stationary person is captured in the background, as another detection method using an image, the luminance change in the time series direction is analyzed for each pixel to determine the pixel state. A device that detects and uses constraints of the state transition of the detected pixel is known (for example, see Non-Patent Document 1).
[0008]
[Patent Document 1]
JP-A-11-311682
[Patent Document 2]
JP 4-1111079 A
[Patent Document 3]
JP 10-31448 A
[Non-Patent Document 1]
Proceedings of the 7th Image Sensing Symposium (2001), C-20, pp 369-374, “Layer type detection method to understand the overlapping of multiple objects”, Hironobu Fujiyoshi, Takeo Kanade
[0009]
[Problems to be solved by the invention]
However, in the apparatus as shown in Patent Document 1 described above, the monitoring position cannot be set unless the place where the person exists is known in advance. In addition, when there are people who work while sitting, for example, in the case of a computer worker, it moves only about the fingertips, so most of the body is taken in the background, the difference integral value in the area becomes small, It is expected that people will not be detected.
[0010]
Further, in the device as shown in Patent Document 2 described above, the background is updated when a state different from the background image continues for a certain time or longer, so that a person sits on a chair or stops and is in a stationary state. Then, there is a problem that a person is taken into the background and only a moving person can be detected. In addition, the apparatus as disclosed in Patent Document 3 described above has a problem that a stationary person cannot be detected due to being captured in the background.
[0011]
In addition, in the detection method as shown in Non-Patent Document 1 described above, it is possible to identify whether the detected person is moving or stationary. Similarly, since the identification is based on the difference between the luminance values, there is a problem that when a person with a luminance value very similar to the background comes, it is determined as the background and no person is detected.
[0012]
The present invention solves the above-described problems, and uses image processing to detect not only a moving person but also a stationary person, and measure its position and number of people even in a situation where ambient light changes. An object of the present invention is to provide a human detection method and a human detection device.
[0013]
[Means for Solving the Problems and Effects of the Invention]
In order to achieve the above object, the invention of claim 1 is a human detection method for detecting a person by performing image processing on a picked-up image, and picks up a person's movement in a predetermined area sequentially in time series. A digital image storage process for storing digital images that have been captured in the imaging process and digitized in time series, and each of the digital images stored in the predetermined image area from the stored time series digital images. A pixel analysis process for reading out the luminance values of the pixels in a time series and calculating a temporal variation variance value of the luminance of each pixel at a predetermined time, and storing the variance values in correspondence with the pixels of the digital image in a time series. The pixel analysis result storage process to be performed and the time-series variation state of the variance value are discriminated, and the pixel state for which the variance value is calculated is identified as at least three states of “moving state”, “person candidate”, and “background”. Pixel state identification In addition, the pixel state identification result storage process for storing the pixel state identification result in correspondence with the pixel of the digital image in time series, the identified “person candidate” pixel, A candidate process for creating a candidate area by combining pixels, a candidate process for creating a candidate area, and storing a candidate result of creating a candidate area corresponding to the pixel of the digital image, and features of the candidate person area And detecting a human area, and a human detection data output process for outputting human detection data such as the position or the number of detected persons.
[0014]
In the human detection method, the luminance value of each pixel located in the predetermined image area is read out from the digital image that is continuous in time series, which is obtained by sequentially capturing the movement of the person in the predetermined area in time series. The time-varying variance value of the luminance of each pixel is calculated, the time-series variation state of the variance value stored in time series corresponding to the pixel of the digital image is determined, and the variance value of the pixel for which the variance value is calculated The state is classified into at least three states of “moving state”, “person candidate” and “background”, and the identified “person candidate” pixel is combined with the “moving state” pixel adjacent or continuous with this to create a candidate region. Because the human area is detected by distinguishing the characteristics of the human candidate area, the moving person is determined as “moving state”, the stationary person is determined as “human candidate”, and the stationary object is determined as “background”. Not only those who are moving, but also those who are stationary It can be detected to separate the person from the background Te. In addition, since the conventional human detection method determines the presence or absence of a person based on the difference between the luminance of the background image or the previous image and the luminance of the current image, the luminance of the human part in the background image and the current image is similar to that of the background. Even if there is a part that could not be detected due to the brightness of the person, or a part that could not be detected because the difference in brightness between the previous image and the current image disappeared when the person stopped, Since the determination is made by using the fact that the variance value representing the temporal variation in luminance is different, it is possible to detect a person with high accuracy. Further, since the person is detected by discriminating the time-series variation state of the dispersion values stored in time series, the person can be detected with high accuracy even under a situation where the ambient light changes.
[0015]
According to a second aspect of the present invention, in the human detection method according to the first aspect, in the human region detection process, when the shape or area of the human candidate region is within a predetermined range, the human candidate region is a human region. Judgment.
[0016]
In the human detection method, it is determined whether the human candidate region is a human region using the shape or area of the human candidate region, so that the process of the human region detection process can be simplified.
[0017]
According to a third aspect of the present invention, in the human detection method according to the first aspect, in the human region detection step, a ratio between the number of “moving state” pixels and the number of “human candidates” pixels included in the human candidate region is obtained. Using this ratio, it is determined whether the human candidate area is a stationary human area.
[0018]
In the above human detection method, by identifying whether the area determined to be a person is moving or stationary, it is possible to distinguish and recognize a stationary person having a long stay time and a moving person having a short stay time, Processing according to a stationary person with a long stay time, for example, air conditioning control can be performed.
[0019]
According to a fourth aspect of the present invention, in the human detection method according to the first aspect, in the human region detection step, the barycentric position of the partial region including the “moving state” pixels included in the human candidate region has a predetermined period. When located within a predetermined range, the person candidate area is determined not to be a human area.
[0020]
In the human detection method, when the position of the center of gravity of the partial region including the “moving state” pixels included in the human candidate region is located within a predetermined range for a predetermined period, the human candidate region is a human region. Because the brightness changes at a specific position on the captured image, for example, the flickering area of the PC display or TV screen can be excluded from the human area, and these can be mistakenly detected as people Can be eliminated.
[0021]
According to a fifth aspect of the present invention, in the human detection method according to the first aspect, in the human region detection process, the position and shape of a circumscribed rectangle circumscribing the human candidate region do not change for a predetermined period, and When the position of the point where the human candidate area touches the circumscribed rectangle does not change, it is determined that the human candidate area is not a human area.
[0022]
In the above human detection method, when the position and shape of the circumscribed rectangle circumscribing the human candidate area do not change for a predetermined period and the position of the point where the human candidate area touches the circumscribed square does not change, Since it is determined that the human candidate area is not a human area, a luminance change at a specific position on the captured image, for example, a flickering area of a personal computer display or a TV screen can be excluded from the human area. It is possible to eliminate detection as a person.
[0023]
According to a sixth aspect of the present invention, in the human detection method according to the first aspect, in the detection result output process, the detected human region and the human detection data are superimposed and displayed on the captured image.
[0024]
In the above human detection method, the detected human area and the human detection data are displayed superimposed on the captured image, so that the human detection data displayed superimposed on one output screen can be intuitively recognized, and the predetermined area This makes it easier to understand the trends of people within the company.
[0025]
According to a seventh aspect of the present invention, there is provided a human detection device for detecting a person by performing image processing on a picked-up image, an image pickup means for sequentially picking up a person's movement in a predetermined area in time series, and an image pickup by the image pickup means. A digital image storage means for storing a digitized digital image in time series and a luminance value of each pixel located in a predetermined image area is read out from the stored time series digital image, Pixel analysis means for calculating a temporal variation variance value of the luminance of each pixel for each time; pixel analysis result storage means for storing the variance value in time series corresponding to the pixels of the digital image; and A pixel state identifying means for discriminating a time-series variation state and identifying the pixel state for which the variance value has been calculated into at least three states of “moving state”, “person candidate”, and “background”; Digital A pixel candidate identification region is created by combining pixel state identification result storage means for time-series storage corresponding to the pixels of the image, the identified "person candidate" pixel, and a "moving state" pixel adjacent to or continuous with the pixel. A human candidate area creating means, a human candidate area creating result storing means for storing the human candidate area creation result in correspondence with the pixel of the digital image, and a human area for determining the characteristics of the human candidate area and detecting the human area It comprises a detection means and a person detection data output means for outputting person detection data such as the detected position or number of persons.
[0026]
In the above configuration, the luminance data of each pixel located in the predetermined image area is read out from the digital image continuously captured in time series that sequentially captures the movement of the person in the predetermined area in time series. Calculate the temporal variation variance value of the luminance of the pixel, determine the temporal variation state of the variance value stored in time series corresponding to the pixel of the digital image, and determine the state of the pixel for which the variance value was calculated At least three states of “moving state”, “person candidate”, and “background” are identified, and a person candidate region is created by combining the identified “person candidate” pixel and a “moving state” pixel adjacent or continuous with this pixel. Since the human area is detected by discriminating the characteristics of the candidate area, the moving person can be determined as “moving state”, the stationary person as “human candidate”, and the stationary object as “background”. Can do not only those who are moving but also those who are stationary It can be detected and separated from. In addition, since the conventional human detection method determines the presence or absence of a person based on the difference between the luminance of the background image or the previous image and the luminance of the current image, the luminance of the human part in the background image and the current image is similar to that of the background. In the human detection device, the brightness of the human part and the part of the object cannot be detected because the brightness difference between the previous image and the current image disappears when the person is stationary. Since the determination is made by utilizing the fact that the variance values representing the temporal fluctuations of are different, it is possible to accurately detect a person.
[0027]
According to an eighth aspect of the present invention, there is provided the human detection device according to the seventh aspect, further comprising environmental control means for controlling an environmental adjustment device such as lighting and / or air conditioning using the human detection data.
[0028]
In the above configuration, since the lighting and / or air conditioning is controlled using the person detection data, the lighting / air conditioning can be controlled according to the number of people in the room and the position of the person. The wasteful energy consumption to operate is eliminated, and the comfort of the office worker in the room can be improved. In addition, according to the present invention, control human detection data can be obtained by installing one imaging TV camera in the control target area.
[0029]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, a human detection method and a human detection device according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a configuration of a human detection device according to the present invention, and FIG. 2 shows a processing flow of a human detection method. The human detection device 1 detects a person 20 in the room 2 and outputs the number and position of the person 20 using a continuous image obtained by sequentially capturing images in a time series from the TV camera 3 installed so as to look down at the room 2. To do. In each process of the human detection method, the imaging process S101 is the TV camera 3, the digital image storage process S102 is the A / D conversion unit 4 and the storage unit 5, the pixel analysis process S103, the pixel state identification process S105, and the human candidate area creation Process S107 and human area detection process S109 are performed in the calculation unit 6, pixel analysis result storage process S104, pixel state identification result storage process S106, human candidate area creation result storage process S108 are stored in the storage unit 5, and human detection data output process S110 is performed in the communication unit 9 and the display 8.
[0030]
An overall flow of a human detection method for detecting a human by performing image processing on a captured image will be described. In the imaging process S101, the movement of a person in a predetermined area (room 2) is sequentially imaged in time series, and in the digital image storage process S102, digital images that are captured in the imaging process and digitized are continuous in time series. Images are stored. In the pixel analysis step S103, the luminance value of each pixel located in the predetermined image area is read from the stored time-series digital image, and the variance value for the time variation of the luminance of each pixel every predetermined time is obtained. In the pixel analysis result storing step S104, the variance value is stored in time series corresponding to each pixel of the digital image.
[0031]
In the pixel state identification step S105, the time-series variation state of the variance value is discriminated, and the pixel state for which the variance value is calculated is identified as four states of “moving state”, “elapsed state”, “person candidate”, and “background”. In the pixel state identification result storage step S106, the pixel state identification results are stored in time series corresponding to the pixels of the digital image. As will be described later, the “elapsed state” may be regarded as a part of the “moving state” or a part of the “person candidate”. In this case, the pixel state is changed to the “moving state” in the pixel state identifying process. ”,“ Person candidate ”, and“ background ”. In the human candidate region creation step S107, a human candidate region is created by combining the identified "human candidate" pixel, and a "moving state" pixel and an "elapsed state" pixel adjacent to or adjacent to the identified "human candidate" pixel. In the storing step S108, the human candidate area creation result is stored in correspondence with the pixel of the digital image. In the human region detection step S109, the human candidate region is detected to detect the human region, and in the human detection data output step S110, human detection data such as the detected position or number of people is output.
[0032]
A feature of the present invention is that a temporal change in luminance is obtained in a time series for a target pixel of a captured continuous image, and a luminance dispersion value at a certain time interval is calculated based on the time variation, and the luminance dispersion value is obtained. Thus, the state of the pixel of interest is classified into at least three states of “moving state”, “person candidate”, and “background” based on the time change. It is determined that the moving person is “moving state”, the stationary person is “person candidate”, and the stationary object is “background”. Since the conventional human detection method simply determines the presence or absence of a person by using the difference in luminance between the current image and the background image (or the previous image), a human part in the current image and the background image. Cannot be detected by distinguishing the person from the background. In addition, when a person is stationary, the difference between the current image and the previous image disappears and the person cannot be detected. According to the present invention, a stationary person and a stationary object can be distinguished and determined, and not only a moving person but also a stationary person can be detected separately from the background.
[0033]
The movement and stop state of the person who is the observation target and the object other than the person will be described. FIG. 3 shows an example of image change. As shown in FIG. 3A, when the person 20 moves from the left in the image 21 and stops at the center of the image 22 and then leaves to the right as in the image 23, and FIG. As shown in b), when the object 25 is moved and once placed at the center in the image 27 and then taken away, the comparison is made. In the stop state, a difference between the state of the person and the object appears. Normally, an abandoned object is in a stationary state, but there is a difference that a person does not completely stop even in a stopped state.
[0034]
The luminance change will be described. FIG. 4A shows a measurement example of the luminance of specific pixels such as the target pixels 24 and 29 indicated by X in the image, and FIG. 4B shows the state of luminance dispersion. In the present invention, two types of dispersion values, the dispersion value F and the dispersion value B, are defined and used. The f section 31 including n images from the past pixel analysis time point t (−n + 1) to the latest image capturing time point t (0) in the future direction with respect to the change curve 30 of the luminance Y shown in FIG. The variance value F is calculated using the luminance values at n points. In addition, the variance value B using the luminance values of m points in the b section 32 including m images from the pixel analysis time point t (−n + 1) to the pixel analysis time point t (−m−n + 2) in the past direction. Is calculated. The state of dispersion of the luminance values (frequency distribution) is as shown by a curve 33 and a curve 34 in the f section 31 and the b section 32, respectively. The example of the b section 32 shown in FIG. 4A is regarded as a moving state of a person or an object, and the example of the f section 31 is regarded as a stopped state.
[0035]
The dispersion value obtained from the luminance value distribution may be a numerical value that can quantify the state of dispersion. For example, the standard deviation σ obtained by fitting the frequency distribution with a normal distribution curve, or the square of the deviation from the average value It is calculated by the formula for sum. The variance values obtained by any of these methods are as follows: variance value 35 (variance value F), variance value 36 (variance value B), f-segment 31 corresponds to the stop state, and b-segment When 32 corresponds to the moving state, it can be seen that the state can be distinguished by the difference in the variance values. The variance value F is used for detecting the start of motion, and the variance value B is used for detecting the end of motion.
[0036]
The state identification will be described. FIG. 5 shows changes in luminance and dispersion value in the case of movement, stop, and movement of a person, and FIG. 6 shows changes in luminance and dispersion value in time in the case of movement, stop, and movement of an object. In FIG. 5A, the variance value F and the variance value B are obtained from the time change curve 40 of the luminance Y for each fixed interval, for example, from the f interval 31 and the b interval 32, and shown in FIG. 5B. The manner of obtaining the dispersion value F curve 41 and the dispersion value B curve 42 shown in FIG. 5C is shown.
[0037]
In order to identify the state of the pixel of interest using such a variance value F curve 41 and variance value B curve 42, three types of threshold values for the variance value are defined and used. These are the “motion state” detection threshold TFM, the elapsed state detection threshold TBS, and the “background” detection threshold TBB. The “moving state” detection threshold value TFM is used for “moving state” detection in comparison with the variance value F when a person or an object moves. The “elapsed state” detection threshold value TBS is used to detect “person candidate” or “background” by comparing with the variance value B in a state once the “moving state” is reached. Until the “moving state” is determined as “person candidate” or “background”, it is regarded as “elapsed state”, and this state is regarded as a part of the “moving state”.
[0038]
For example, in FIG. 5B, the pixel of interest is “moving state” after the intersection M1 where the “moving state” detection threshold value TFM intersects the rising slope of the variance value F curve 41 (curved portion having a positive time differential coefficient). ] (Section 44). Thereafter, in FIG. 5C, the pixel of interest is “elapsed” after the point S1 where the “elapsed state” detection threshold value TBS intersects the descending slope of the variance value B curve 42 (curve portion having a negative time differential coefficient). State (interval 45), and then when the state where the variance B is equal to or less than the threshold value TBS and equal to or greater than the threshold value TBB continues for a certain period of time, the state of the pixel of interest is determined as “person candidate”. Determine (section 46). Further, after the intersection S2 between the threshold value TBS and the dispersion value B curve 42 in FIG. 5C, the dispersion value B is changed as shown after the intersection point B1 between the threshold value TBB and the dispersion value B curve 42. When the threshold value is less than or equal to TBS and further becomes less than or equal to TBB within a certain time, the state of the pixel of interest is determined to be “background” (section 49). Here, instead of providing an “elapsed state”, the dispersion value is equal to or greater than the threshold value TBS or the threshold value TBS after a predetermined wait time has elapsed since the dispersion value became equal to or less than the threshold value TBS. The state of the target pixel may be determined as “moving state”, “person candidate”, or “background” depending on whether it is between the values TBB or below the threshold value TBB. In this case, it is assumed that the “dynamic state” continues during the wait time, and if the variance value is equal to or greater than the threshold value TBS after the wait time, the “dynamic state” is further continued, and the variance value is the threshold value. If it is between TBS and threshold value TBB, the state of the target pixel is determined as “person candidate”, and if the variance value is equal to or smaller than threshold value TBB, the state of the target pixel is determined as “background”.
[0039]
As described above, the “person candidate” section 46 and the “background” section 49 shown in FIG. 5A can be distinguished by the behavior after the intersections S1 and S2 with the threshold value TBS in the variance value B curve 42. it can. A person is not able to completely stop even if he / she stops, and is shaking, thereby distinguishing the stationary state (or background) of the object from the stationary state of the person.
[0040]
In the case of an object, as shown in FIG. 6, the “moving state” is detected by the intersections M3 and M4 with the variance F curve 51 as in the case of a person (sections 54 and 57). After the “elapsed state”, unlike a person, in the case of an object, the threshold value TBB and the dispersion value B curve after the intersections S3 and S4 between the threshold value TBS and the dispersion value B curve 52 in FIG. Intersection points B3 and B4 with 52 appear without much time, and the variance B becomes equal to or less than the threshold value TBB, so that the state of the target pixel can be determined as “background” (sections 56 and 59).
[0041]
With regard to the four states that can be taken by the pixel defined by the luminance and the luminance dispersion value, the transition between the states will be described. FIG. 7 shows, with arrows, the state of transition that can be normally taken between the four states of the pixel. The four states of the background 61, the moving state 62, the elapsed state 63, and the person candidate 64 can each maintain their own states as indicated by arrows returning from the self state and returning to the self state. Moreover, although it can become the background 61 or the person candidate 64 from the progress state 63, conversely, it does not become a progress state from the background 61 or the person candidate 64. In addition, the moving state 62 can be changed from the person candidate 64, but conversely, the moving candidate 62 does not become the moving candidate 62. When the moving state 62 changes to the background 61, the state transition usually takes place via the elapsed state 63. However, when the elapsed state is short, depending on the balance between the data interval of the distributed value and the elapsed state duration, There is a case where the moving state 62 as shown in FIG.
[0042]
Next, each process in the human detection method will be described. First, the digital image storage process will be described with reference to the flowchart of FIG. In this flow, as described above, the dispersion value F is stored from n images, and (m + n−1) images for determining the dispersion value B from m images are stored (one is common). First, initial processing for image storage and memory reservation are performed (S201 to S203). Next, an analog video signal that is sequentially read and generated from the image projected on the image sensor in the imaging process is received (S204), and this signal is D / A converted and stored in a predetermined memory (S205). To S208). Finally, the image memory area number is incremented in preparation for the next image input.
[0043]
The pixel analysis process and the pixel analysis result storage process will be described with reference to the flowchart of FIG. In this flow, the luminance of pixels located in a predetermined image area is read from the stored time-series digital image, and the luminance variance value F and variance value B for each predetermined time are calculated. First, initial processing and memory reservation are performed (S301, S302). Next, among the (m + n−1) images stored in the digital image storage process, the luminance value of the image at the target pixel is read from the newer n images, and the variance value F is calculated (S303 to S305). . Next, among the (m + n−1) images stored in the digital image storage process, the luminance value of the image at the target pixel is read out from the older m images and the variance value B is calculated (S306, S307). . The above processing is performed for all the pixels in the predetermined image area.
[0044]
The pixel state identification process and the pixel state identification result storage process will be described with reference to the flowchart of FIG. In this flow, the pixel state of the target pixel is classified into four, and the result is stored in the memory. First, initial processing and memory reservation are performed (S401 to S403). Next, the dispersion value F data, the dispersion value B data, and the pixel state identification result data at the time of the previous processing at the position of the target pixel are read (S404). If the previous result is “background” (Y in S405), the moving state detection threshold value TFM is compared with the variance value F. If TFM <F, the pixel state is moved (Y in S406, S407). If TFM <F, the pixel state is set to the background (N in S406, S408).
[0045]
If the previous result is a “person candidate” (Y in S409), the moving state detection threshold value TFM is compared with the variance value F. If TFM <F, the pixel state is moved (Y in S410, S411). ), If TFM <F, the pixel state is a human candidate (N in S410, S412).
[0046]
If the previous result is “elapsed state” (Y in S413), the elapsed state detection threshold value TBS is compared with the variance value B. If B ≦ TBS is not satisfied, the pixel state is changed to the moving state (N in S414, S415) ), B ≦ TBS, the background detection threshold value TBB and the variance value B are further compared. In this comparison, if B ≦ TBB, the pixel state is set to the background (Y in S416, S417). If B ≦ TBB is not satisfied, the elapsed state counter T is compared with the elapsed state count threshold value TM after the elapsed state counter is incremented (T = T + 1) in order to see the state change with time, and TM <T In this case, the elapsed state counter T is cleared to zero (T = 0) and the pixel state is set as a human candidate (N in S416, Y in S418, Y in S419, S420, S421). If TM <T is not satisfied, the pixel state is set to the elapsed state (N in S419, S422).
[0047]
If the previous result is “moving state” (Y in S423), the elapsed state detection threshold value TBS is compared with the variance value B. If B ≦ TBS is not satisfied, the pixel state is moved (N in S424, S425). ), B ≦ TBS, the background detection threshold value TBB and the variance value B are further compared. In this comparison, if B ≦ TBB is not satisfied, the elapsed state counter T is cleared to zero (T = 0) and the pixel state is set to the elapsed state (N in S426, S427, and S428). If B ≦ TBB, the pixel state is set to the background (Y in S426, S429).
[0048]
The pixel state identification result is stored in the memory in correspondence with the pixel position of the captured digital image (S430). The above state identification process is performed for all the pixels in the predetermined image area. In the above-described embodiment, a past period and a future period are determined from a past analysis time, and the two types of variance value B and variance value F obtained in each period are used to determine the target pixel. Although the state is identified, in addition to this method, one type of period consisting of only a period in the past direction, only a period in the future direction, or a period that spans both from a certain analysis point in the past was obtained. The state of the target pixel may be identified using one type of dispersion value. As described above, when two types of variance values are used, there is an effect that a state with good responsiveness can be identified, and when one type of variance value for one type of period as shown here is used. Has the effect of being able to be identified by reducing the storage capacity of the variance value.
[0049]
The human candidate area creation process and the human candidate area creation result storage process will be described with reference to the flowchart of FIG. In this flow, a “candidate person” pixel whose pixel state has been identified, a “moving state” pixel and an “elapsed state” pixel that are close to or continuous with this pixel are combined to create a candidate human region. First, initial processing and memory allocation are performed (S501, S502). Next, the pixel state identification result stored in the memory is read, and the first target pixel address is set (S503, S504). Next, the pixel state is inspected for the m1 × n1 pixel range separately defined in the vicinity of the target pixel, for example, the vicinity of 6 × 4 = 24, and any of “moving state” D, “elapsed state” T, and “person candidate” C is selected. These pixels are labeled as pixels in the human candidate area (S505). The above labeling process is performed for all the pixels in the predetermined image area (Y in S507 and S506). Next, in order to perform one labeling as a group for adjacent or consecutive human candidate regions, a label consistency check and a relabeling are performed, and finally the pixel position of the captured digital image is set. Correspondingly, the human candidate area creation result is stored in the memory (S508 to S511).
[0050]
FIG. 12 shows a candidate candidate area creation result. In this figure, each square represents one pixel, the plain pixel 71 is the background, the pixel 72 in which the letter D is entered is the moving state, the pixel 73 in which the letter T is entered is the elapsed state, and the letter C is entered. Pixel 74 represents a human candidate. In addition, regions 75, 76, and 77 including a set of pixels other than the background are labeled human candidate regions.
[0051]
The human area detection process will be described with reference to the flowchart of FIG. In this flow, the characteristics of the human candidate area are determined, and the human area is detected. For example, the area (number of pixels) of the area is used as the feature of the human candidate area. As another feature, the shape of the region such as the vertical and horizontal sizes of the region can also be used. First, the stored human candidate area creation result is read out, and the process proceeds in order of label numbers for each area (S601, S602). The area P of the labeled region is calculated (S603), and this area P is compared with the minimum area Pmin and the maximum area Pmax (S604). When the area P of the region satisfies Pmin <P ≦ Pmax, this region is determined to be a person, and the center of gravity of the region is set as a pixel position representing the person (Y in S604, S605, S606). If the area P of the region does not satisfy Pmin <P ≦ Pmax, it is determined that this region is not a person (S607). These processes are performed for all the labeled regions, and finally the human detection data is stored (S609).
[0052]
The human detection data output process will be described with reference to the flowchart of FIG. In this flow, human detection data such as the detected position or number of people is read from the memory (S11) and output on the display (S12). FIG. 15 shows an example of human detection data output. On the display screen 79, the detected number of people and the detected position are displayed.
[0053]
A human region detection process using the ratio of the sum of “moving state” and “elapsed state” pixels and the sum of “person candidate” pixels will be described with reference to the flowchart of FIG. In this flow, the ratio of the sum of the “moving state” pixel and the “elapsed state” pixel included in the human candidate region and the sum of the “human candidate” pixel is obtained, and the human candidate region is stationary using this ratio. It is determined whether it is an area. First, the stored person candidate area creation results are read out, and the process proceeds in order of label numbers for each area (S701, S702). The total number Ndt of “moving state” pixels and “elapsed state” pixels in the labeled region is calculated (S703), and the total number Nc of “human candidate” pixels in the region is calculated (S704). Next, the composition ratio Nc / (Ndt + Nc) of the human candidates obtained from the total numbers Ndt and Nc is compared with a predetermined stationary state detection threshold Ns (S705). As a result of comparison, if Ns <Nc / (Ndt + Nc), this area is determined to be a stationary person, and the center of gravity of the area is set as a pixel position representing the person (Y in S705, S706, and S707). If Ns <Nc / (Ndt + Nc), it is determined that this area is not a stationary person (N in S705, S708). These processes are performed for all the labeled areas, and finally the human detection data is stored (S710).
[0054]
A human region detection process using the gravity center position of the partial region that is a combination of the “moving state” and the “elapsed state” will be described with reference to the flowchart of FIG. According to this flow, when the position of the center of gravity of the partial area including the “moving state” pixel and the “elapsed state” pixel included in the human candidate area is within a predetermined range for a predetermined period, the human candidate area Can be determined not to be a human area. First, the stored person candidate area creation result and the previous person detection data are read out, and the process proceeds in the order of the label numbers for each area (S801 to S803). Next, when a “moving state” pixel or an “elapsed state” pixel exists in the attention area, the center of gravity of the “moving state” pixel and the “elapsed state” pixel in the area is calculated (Y in S804, S805). . Next, when there is no previous human candidate area within the predetermined range (N in S806), the center-of-gravity position counter Tg is assigned to this area and Tg = 1 is set (S807). If the previous candidate area is within the predetermined range (Y in S806), the barycentric position counter Tg assigned to this area is incremented (Tg = Tg + 1) (S808). Next, the center-of-gravity count threshold TG and the center-of-gravity position counter Tg are compared, and if TG <Tg, it is determined that this region is not a person (Y in S809, S810). These processes are performed for all the labeled areas, and finally the human detection data is stored (S812).
[0055]
According to the above-described method, it is possible to determine that a certain person candidate area is not a person. By doing so, as shown in FIG. 18, even if a television screen whose luminance changes with time enters the captured image 80, the position in the image that fluctuates due to the flickering of the television screen is the position of the television. Since this area is fixed, it is determined that this area is not a person, so that it is possible to prevent the flickering of the television screen from being erroneously detected as a person.
[0056]
A human area detection process using a circumscribed rectangle of a human candidate area and its contact points will be described with reference to the flowchart of FIG. According to this flow, when the position and shape of the circumscribed rectangle circumscribing the human candidate region do not change for a predetermined period and the position of the point where the human candidate region touches the circumscribed rectangle does not change, the human candidate region Can be determined not to be a human area. First, the stored human candidate area creation result, the position / shape of the circumscribed rectangle of the previous human candidate area, and the position data of the contact are read out, and the process proceeds in order of the label number for each area (S901 to S903). Next, the position / shape of the circumscribed rectangle of the person candidate area with the label number and the position of the contact are calculated (S904). Next, if there is a human candidate area within a predetermined range to determine whether or not a human area exists (Y in S905), the position / shape of the circumscribed rectangle of the previous human candidate area, the position of the contact point, and the current human candidate area The position / shape of the circumscribed quadrangle and the position of the contact are compared (S907). Further, when there is no previous candidate area within the predetermined range (N in S905), and when the difference between the previous time and this time is not within the predetermined range for the circumscribed rectangle (N in S907), the circumscribed rectangle counter Tq in this region And Tq = 1 (S906). If the difference between the previous time and the current time is within a predetermined range for the circumscribed rectangle (Y in S907), the circumscribed rectangle counter Tq assigned to this area is incremented (Tq = Tq + 1) (S908). Next, the circumscribed square frequency threshold value TQ and the circumscribed square counter Tq are compared, and if TQ <Tq, it is determined that this area is not a person (Y in S909, S910). These processes are performed for all the labeled areas, and finally the human detection data is stored (S912).
[0057]
According to the above-described method, it is possible to determine that a certain person candidate area is not a person. In this way, as shown in FIG. 20, even when a television screen 83 whose luminance changes with time enters the captured image 82, a circumscribed rectangle 84 that circumscribes the television screen 83 that fluctuates due to the flickering of the television screen. Since the position of is fixed at the position of the television, it is determined that this area is not a person, so that it is possible to prevent the flickering of the television screen from being erroneously detected as a person.
[0058]
The superimposed display of the captured image, the detected human region, and the human detection data will be described with reference to the flowchart of FIG. In this flow, the stored human detection data is read (S21), and the detected human region and human detection data are superimposed on the captured image and displayed (S22). FIG. 22 shows an example in which human detection data is output in an overlapping manner. Thus, by displaying the captured image 86 of the predetermined area on the display 85 and the position 88 and the number 89 of the person 87 in the detection area on each image, the person can be easily recognized. The human situation in the detection area can be grasped by watching.
[0059]
Hereinafter, a human detection device according to another embodiment of the present invention will be described. FIG. 23 shows a block diagram of a human detection device configuration. The image pickup means 11 of this human detection apparatus sequentially picks up the movement of a person in a predetermined area (room 2) in time series, and the digital image storage means 12 is picked up and digitized in the image pickup process. Continuous digital images are stored. The pixel analysis means 13 reads the luminance of the pixels located in the predetermined image area from the stored time-series digital image, calculates the luminance dispersion value for every predetermined time, and the pixel analysis result storage means 130 The dispersion values are stored in time series in correspondence with the pixels of the digital image.
[0060]
The pixel state identification means 14 discriminates the time-series variation state of the variance value, and identifies the pixel state for which the variance value is calculated as four states of “moving state”, “elapsed state”, “person candidate”, and “background”. Then, the pixel state identification result storage unit 140 stores the pixel state identification results in time series in correspondence with the pixels of the digital image. The human candidate area creating unit 15 creates a human candidate area by combining the identified “human candidate” pixel, and a “moving state” pixel and an “elapsed state” pixel adjacent to or continuous with the identified “human candidate” pixel. The storage unit 150 stores the human candidate region creation result in association with the pixel of the digital image. The human area detection means 16 discriminates the characteristics of the human candidate area and detects the human area, and the human detection data output means 17 outputs human detection data such as the detected position or number of persons.
[0061]
Illumination / air conditioning control based on human detection data output from such a human detection device will be described. FIG. 24 shows a human detection device provided with environment control means and an application example of the device. In order to detect the number of people 20 in the room 2, one TV camera 3 is installed on the ceiling so that the entire floor can be seen. The image from the TV camera 3 is sent to the human detection device main body 18 of the human detection device 10, and human detection is performed by the human detection device main body based on the image. An environmental control means 19 is connected to the human detection main body 18, and the environmental control means 19 controls illumination and air conditioning based on human detection data from the human detection apparatus main body 18. The room 2 includes, for example, four lightings 201, two air conditioning units 202, and one outside air inlet 203, and these lighting / air conditioning devices are controlled by the environment control means 19.
[0062]
The above-described human detection device main body 18 has a series of device configurations from the imaging means 11 to the human detection data output means 17 shown in FIG. The image pickup means in the human detection means 18 picks up time-series indoor images by the TV camera 3. The digital image storage means digitizes the analog video signal input from the imaging means in accordance with the flowchart of FIG. 8 described above and stores it as a continuous image in the memory. The pixel analysis means and the pixel analysis result storage means calculate the variance value F and variance value B for all the pixels on the image according to the flowchart of FIG. 9, and store the results in the memory. The pixel state identification unit and the pixel state identification result storage unit classify the state of each pixel into four states of “moving state”, “elapsed state”, “person candidate”, and “background” according to the flowchart of FIG. Save the result in memory. The human candidate area creation means and the human candidate area creation result storage means create a human candidate area using the pixels of “moving state”, “elapsed state”, and “human candidate” in accordance with the flowchart of FIG. Perform labeling.
[0063]
The human region detection means detects a person according to the flowchart of FIG. When distinguishing between a stationary person and a moving person, the processing shown in the flowchart of FIG. 16 is also performed. Further, when it is necessary to reduce false detection such as flickering on the TV screen, FIG. Alternatively, the processing shown in either or both of the flowcharts of FIG. 19 is performed. The human detection data output means displays the detected number of people and their position on the display according to the flowchart of FIG. 14 and transmits the human detection data to the environment control means 19. The environment control means 19 controls the lighting / air conditioning equipment based on the human detection data.
[0064]
By using the human detection device 10 of the present invention, it is possible to obtain human detection data for operating a plurality of lighting devices with a single human detection device. Conventionally, pyroelectric element sensors have been installed for each lighting device, and lighting / extinguishing of lighting devices is controlled in accordance with the presence of a person. By installing the TV camera, control person detection data can be obtained.
[0065]
The flow of the illumination control process will be described with reference to the flowchart of FIG. After the human detection apparatus main body executes the human detection process (S31), the environment control means receives the human detection data and determines whether there is a person in the area corresponding to each lighting device number (S32, S33). Areas determined not to have people are turned off (N in S33, S33), and areas determined to have people are lit (Y in S33, S34). Such processing is performed for all lighting devices. Further, after a predetermined time has elapsed, the entire steps S31 to S37 are repeated.
[0066]
In addition, regarding the air conditioning in the room, by using the person detection data, the air conditioning can be controlled according to the number of people in the room and the position of the person. For this reason, useless energy consumption can be eliminated and the comfort of the office worker in the room can be improved. Conventionally, air conditioning is controlled based on the output of the temperature sensor, so it is wasteful because it tries to maintain a constant temperature even when no one is present, and it has a constant strength regardless of how crowded it is. In many cases, it is not possible to provide a comfortable state for people. In addition, since the outside air intake operation is often performed based on a fixed number of people in the room determined by the size of the room, wasteful ventilation is performed even though there are few people, and energy is wasted. There are many. According to the present invention, such conventional problems are solved.
[0067]
The flow of the air conditioning control process will be described with reference to the flowchart of FIG. In the following description, it is assumed that a person in the room is performing a stationary work such as office work. After the human detection apparatus main body executes the human detection process (S41), the environment control means receives the human detection data, and first adjusts the outside air intake amount in accordance with the total number of detected persons (S42). Next, it is determined whether or not there is a stationary person in the area corresponding to each air conditioning equipment number (S44). In an area where there is no stationary person, it can be determined that there is no person, so the air conditioning equipment is stopped. (N in S44, S46) If there is a stationary person, the output of the air conditioner is adjusted to the number of stationary persons (Y in S44, S45). Such a process is performed for all the air conditioners. Further, after a predetermined time has elapsed, the entire steps S41 to S47 are repeated. Note that the above-described lighting / air-conditioning equipment, which is an environmental adjustment device, may be replaced with other environmental adjustment devices, and can be similarly controlled by the environmental control means.
[0068]
The present invention is not limited to the above-described configuration, and various modifications can be made. Further, the present invention is applied to a general animal other than a human being as a device for detecting the number and trend within a predetermined area. This apparatus is particularly effective for animals with little movement.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a human detection device according to an embodiment of the present invention.
FIG. 2 is a flowchart of a human detection method according to an embodiment of the present invention.
FIG. 3A is a diagram for explaining the movement of a person according to the person detection method, and FIG. 3B is a diagram for similarly explaining the movement of an object.
FIG. 4A is a luminance time change diagram illustrating a luminance dispersion value calculation section according to the above-described person detection method, and FIG. 4B is a luminance value frequency distribution diagram illustrating luminance dispersion.
5A is a time variation diagram of luminance in the case of a person according to the person detection method, and FIGS. 5B and 5C are time variation diagrams of luminance dispersion values. FIG.
6A is a time variation diagram of luminance in the case of an object according to the same person detection method, and FIGS. 6B and 6C are time variation diagrams of luminance dispersion values.
FIG. 7 is a state transition diagram of pixels according to the person detection method.
FIG. 8 is a flowchart of a digital image storage process according to the person detection method.
FIG. 9 is a flowchart of a pixel analysis process and a pixel analysis result storage process according to the person detection method.
FIG. 10 is a flowchart of a pixel state identification process and a pixel state identification result storage process according to the person detection method.
FIG. 11 is a flowchart of a human candidate area creation process according to the person detection method.
FIG. 12 is a pixel diagram showing a result of human candidate area creation according to the person detection method same as above.
FIG. 13 is a flowchart of a human area detection process according to the human detection method.
FIG. 14 is a flowchart of a human detection data output process according to the human detection method.
FIG. 15 is a diagram of a human detection data output example according to the human detection method.
FIG. 16 is a flowchart of a human area detection process using a ratio of a sum of “moving state” pixels and “elapsed state” pixels and a sum of “person candidates” pixels according to the person detection method;
FIG. 17 is a flowchart of a human region detection process using the center of gravity position of a partial region that is a combination of “moving state” and “elapsed state” according to the human detection method;
FIG. 18 is a diagram for explaining TV screen processing according to the person detection method;
FIG. 19 is a flowchart of a detection process using a circumscribing rectangle of a human candidate region and its contact points according to the person detection method same as above.
FIG. 20 is a diagram for explaining TV screen processing according to the person detection method;
FIG. 21 is a processing flow diagram of human detection data overlay display according to the human detection method.
FIG. 22 is a diagram of a human detection data overlay output example according to the person detection method same as above.
FIG. 23 is a configuration block diagram of a human detection device according to another embodiment of the present invention.
FIG. 24 is a configuration diagram of a human detection device according to still another embodiment of the present invention.
FIG. 25 is a flowchart of illumination control by the apparatus.
FIG. 26 is a flow chart of air conditioning control by the apparatus.
[Explanation of symbols]
1, 10 person detection device
11 Imaging means
12 Digital image storage means
13 Pixel analysis means
14 Pixel state identification means
15 candidate area creation means
16 human area detection means
17 Person detection data output means
19 Environmental control means
75,76,77 candidate area
130 Pixel analysis result storage means
140 Pixel state identification result storage means
150 candidate area creation result storage means
C candidate
D movement state
T elapsed state

Claims (8)

A human detection method for detecting a person by performing image processing on a captured image,
An imaging process for sequentially imaging the movement of a person in a predetermined area in time series;
A digital image storage process for storing digital images captured in the imaging process and digitized in a time-series manner;
A pixel analysis step of reading out the luminance value of each pixel located in a predetermined image area in a time series from the stored time-series digital image, and calculating a temporal variation dispersion value of the luminance of each pixel for every predetermined time; ,
A pixel analysis result storing process for storing the dispersion value in a time series corresponding to the pixels of the digital image;
A pixel state identification process for discriminating a time-series variation state of the variance value and identifying the pixel state for which the variance value is calculated into at least three states of “moving state”, “person candidate”, and “background”;
A pixel state identification result storing process for storing pixel state identification results in time series in correspondence with the pixels of the digital image;
A human candidate region creation process of creating a human candidate region by combining the identified "human candidate" pixel and a "moving state" pixel close to or continuous with the pixel,
A human candidate region creation result storage process for storing a human candidate region creation result corresponding to the pixel of the digital image;
A human area detection process for determining the characteristics of the human candidate area and detecting the human area;
And a human detection data output process for outputting human detection data such as the position or number of detected persons. The human detection method according to claim 1, wherein the human region detection step determines that the human candidate region is a human region when the shape or area of the human candidate region is within a predetermined range. In the human area detection process, a ratio between the number of “moving state” pixels included in the human candidate area and the number of “human candidates” pixels is obtained, and using this ratio, the human candidate area is a stationary human area. The human detection method according to claim 1, wherein the determination is made. In the human area detection process, when the gravity center position of the partial area including the “moving state” pixels included in the human candidate area is within a predetermined range for a predetermined period, the human candidate area is a human area. The human detection method according to claim 1, wherein it is determined that it is not. In the human area detection process, when the position and shape of a circumscribed rectangle circumscribing the human candidate area do not change for a predetermined period, and the position of the point where the human candidate area touches the circumscribed square does not change, The human detection method according to claim 1, wherein the human candidate area is determined not to be a human area. 2. The human detection method according to claim 1, wherein in the detection result output process, the detected human area and the human detection data are superimposed on the captured image. A human detection device for detecting a person by performing image processing on a captured image,
Imaging means for sequentially imaging the movement of a person in a predetermined area in time series;
Digital image storage means for storing digital images that are captured by the imaging means and digitized in time series; and
Pixel analysis means for reading out the luminance value of each pixel located in a predetermined image area in a time series from the stored time-series digital image and calculating a time variation variance value of the luminance of each pixel every predetermined time; ,
Pixel analysis result storage means for storing the variance values in time series in correspondence with the pixels of the digital image;
A pixel state identifying means for discriminating a time-series variation state of the dispersion value and identifying at least three states of a moving state, a human candidate, and a background as a state of the pixel for which the dispersion value is calculated;
Pixel state identification result storage means for storing pixel state identification results in time series in correspondence with the pixels of the digital image;
Human candidate area creating means for creating a human candidate area by combining the identified "human candidate" pixel and a "moving state" pixel adjacent or continuous thereto;
Human candidate region creation result storage means for storing the human candidate region creation result in correspondence with the pixel of the digital image;
Human area detection means for determining the characteristics of the human candidate area and detecting the human area;
And a human detection data output means for outputting human detection data such as the position or number of detected persons. 8. The human detection apparatus according to claim 7, further comprising environmental control means for controlling environmental adjustment equipment such as lighting and / or air conditioning using the human detection data.

Images