JP6332703B2 - Space recognition device - Google Patents

Description

  The present invention relates to a space recognition device that recognizes a measurement signal obtained by measuring a corresponding position from a plurality of measurement signals (for example, luminance signals) obtained by measuring a common space from different directions.

  For the purpose of crime prevention and the like, it is performed to detect unauthorized passers and intruders based on images taken by a surveillance camera. At that time, multiple surveillance cameras are installed to prevent a decrease in detection accuracy caused by the fact that some of the people have characteristics similar to the background or that there is occlusion between people. There is a case in which a common space is imaged from different directions and information that cannot be obtained from one monitoring camera is supplemented by another monitoring camera.

  For example, in the image monitoring apparatus described in Patent Document 1, the same position is imaged in the change area on the wall camera side in order to complement the change area that is missed on the ceiling camera side due to similarity with the background. It is described that the pixel on the ceiling camera side where the obtained pixel is obtained is set as the change region. At this time, between the monitoring image of the ceiling camera and the monitoring image of the wall camera, the pixels captured at the same position can be obtained by comparing the image features between the pixels, such as a difference in luminance value or correlation value, a difference in color component or correlation value, etc. Corresponding based on commonality / similarity.

  Such processing is normally performed using a monitoring image captured by the monitoring camera at a cycle of about 200 ms.

JP 2010-045501 A

  However, the feature value of a pixel changes over time due to fluctuations caused by disturbances such as noise in the camera, etc., and minute fluctuations in illumination conditions, etc. There is an error that cannot be matched because they do not match or that it is incorrectly matched to another nearby pixel.

  That is, in a normally used monitoring camera, an average value of luminance signals output from each image sensor for a certain time (for example, for 200 ms) is output as a luminance value of a pixel corresponding to the image sensor. The luminance value output in this way is unstable because it always fluctuates due to the disturbance generated within the predetermined time and the noise inside the camera. Therefore, as long as this luminance value is used, it is difficult to avoid an association error due to fluctuation.

  In addition, the correlation is performed by calculating feature quantities between pixels, but the calculation of feature quantities including fluctuations doubles the influence of fluctuations. Furthermore, since the difference or correlation value that is the calculation result between the feature quantities is a scalar, it is difficult to separate the non-fluctuation component from the calculation result between the feature quantities and use it for association.

  Therefore, the present invention has been made in view of the above problems, and even if fluctuation occurs in the measurement signal, the measurement signal obtained by measuring the corresponding position from the plurality of measurement signals obtained by measuring the common space is highly accurate. It is an object of the present invention to provide a space recognition device that can be recognized.

In order to achieve the above-described object, the space recognition device according to the present invention is configured such that one or more first measurement units arranged on the first measurement surface measure a physical quantity in an arbitrary space in an arbitrary time interval. From the first measurement signal output by each first measurement unit, a first feature amount distribution calculation unit that calculates a time frequency for each feature amount in the first small region set on the first measurement surface;
Each of a plurality of second measurement units arranged on a second measurement surface different from the first measurement surface measures a physical quantity of the space in the time interval and is output by each second measurement unit. A second feature amount distribution calculating unit that calculates a time frequency for each feature amount in each of the plurality of second small regions set on the second measurement surface from the signal;
A correspondence index value representing the difference or similarity between the feature quantities is obtained between the first small area and each of the plurality of second small areas, and the correspondence is obtained using the time frequency for each correspondence index value as an element. A corresponding index value distribution calculating unit for calculating the index value distribution;
Using the correspondence index value distribution of the plurality of second small areas, a second small area obtained by measuring a physical quantity at a position in the space corresponding to the first small area from among the plurality of second small areas. A corresponding area recognition unit for recognizing;
It is provided with.

  Further, in the space recognition device according to the present invention, the correspondence index value distribution calculation unit calculates the time frequency of the feature amount of the first small region and the feature amount of the second small region for each correspondence index value. The time frequency of the corresponding index value may be calculated by summing up the products of the time frequencies for each combination for which the corresponding index value is calculated among the combinations of time frequencies.

Furthermore, in the space recognition device according to the present invention, the corresponding index value distribution calculating unit calculates the corresponding index value distribution by obtaining the corresponding index value representing the difference between the feature quantities,
The corresponding area recognition unit calculates a product sum value of a corresponding index value and a time frequency in the corresponding index value distribution for each of the corresponding index value distributions, and the product sum value of the plurality of second small areas is predetermined. You may recognize as the 2nd small area which measured the physical quantity of the position in the said space which determines said 1st small area from the 2nd small area which is below a value.

Further, in the space recognition device according to the present invention, the corresponding index value distribution calculating unit calculates the corresponding index value distribution by obtaining the corresponding index value representing the similarity between the feature quantities,
The corresponding area recognition unit calculates a product sum value of a corresponding index value and a time frequency in the corresponding index value distribution for each of the corresponding index value distributions, and the product sum value of the plurality of second small areas is predetermined. You may recognize as the 2nd small area which measured the physical quantity of the position in the said space which determines said 1st small area from the 2nd small area which is more than a value.

Furthermore, in the space recognition apparatus according to the present invention, the plurality of first measurement units are arranged on the first imaging surface with the first measurement surface as the first imaging surface, It consists of a plurality of first photoelectric conversion units that output a first luminance signal as a first measurement signal,
The plurality of second measurement units are arranged on the second imaging surface with the second measurement surface different from the first measurement surface as a second imaging surface, and each of the second measurement units images the space and performs the second measurement. It is good also as a structure which consists of a some 2nd photoelectric conversion part which outputs the 2nd luminance signal as a signal.

  According to the space recognition device of the present invention, in the time frequency for each feature amount of the measurement signal in the first small area and the second small area, the correspondence index calculated between the first small area and the second small area In the time frequency for each value, a fluctuation component with a low time frequency and a non-fluctuation component with a high time frequency can be distinguished. Therefore, only the correspondence between the first small area and the second small area due to the non-fluctuation component can be correctly recognized, and the recognition accuracy is greatly improved.

It is a figure which shows schematic structure of the space recognition apparatus which concerns on this invention. It is a schematic diagram of a distribution image. It is a functional block diagram of the space recognition device concerning the present invention. (A) It is a figure which shows an example of the relationship between time and a luminance value. (B) It is a figure which shows an example of the relationship between a luminance value and time frequency. It is explanatory drawing which shows an example of luminance difference distribution. It is a flowchart which shows operation | movement of two cameras and a process part in the space recognition apparatus which concerns on this invention.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to FIGS. 1 to 6 of the accompanying drawings.

[Configuration of space recognition device]
As shown in FIG. 1, the space recognition device 1 according to the present embodiment includes an imaging unit 2, a storage unit 3, a processing unit 4, and an output unit 5, and measures a plurality of common spaces from different directions. Among the measurement signals (for example, luminance signals), the measurement signal obtained by measuring the corresponding position is recognized.

  The imaging unit 2 includes a camera 2a and a camera 2b. The camera 2 a and the camera 2 b capture a predetermined space (monitoring space) at high speed, generate a distribution image that is a collection of feature amount distributions as shown in FIG. 2 at a predetermined period, and output the distribution image to the processing unit 4. Note that each rectangular portion in the x direction (horizontal direction) and the y direction (vertical direction) in FIG. 2 corresponds to a pixel.

  The camera 2a and the camera 2b share at least a part of the field of view, and can be configured by, for example, a stereo camera in which two cameras 2a and 2b are installed at intervals imitating human eyes.

  Also, for example, it is composed of two cameras 2a and 2b installed on the wall and ceiling of the room, and two cameras (general view sharing cameras) 2a and 2b installed on the ceiling of the room several meters apart. You can also

  The camera 2a includes a plurality of photoelectric conversion units 20a (20a-1, 20a-2, ..., 20a-N) shown in FIG. Each of the plurality of photoelectric conversion units 20a-1, 20a-2,..., 20a-N is a first measurement unit (first imaging unit), and photoelectrically converts reflected light from an object existing in the monitoring space. A luminance signal corresponding to the luminance of the reflected light is output.

  Further, the camera 2a includes a feature amount distribution calculation unit 21a shown in FIG. The feature amount distribution calculation unit 21a holds the luminance signals from the plurality of photoelectric conversion units 20a-1, 20a-2,..., 20a-N for a certain period of time, and from the held luminance signals of a predetermined time width, one or more The time frequency for each feature amount with the photoelectric conversion unit 20a as a unit is calculated as the first feature amount distribution.

  Similarly to the camera 2a, the camera 2b includes a plurality of photoelectric conversion units 20b (20b-1, 20b-2,..., 20b-N) illustrated in FIG. Each of the plurality of photoelectric conversion units 20b-1, 20b-2,..., 20b-N is a second measurement unit (second imaging unit), and photoelectrically converts reflected light from an object existing in the monitoring space. A luminance signal corresponding to the luminance of the reflected light is output.

  Similarly to the camera 2a, the camera 2b includes a feature amount distribution calculation unit 21b shown in FIG. The feature amount distribution calculation unit 21b holds the luminance signals from the plurality of photoelectric conversion units 20b-1, 20b-2,..., 20b-N for a certain period of time, and from the held luminance signals of a predetermined time width, one or more The time frequency for each feature value with the photoelectric conversion unit 20b as a unit is calculated as the second feature value distribution.

  In this example, two cameras 2a and 2b are used. However, the present invention is not limited to this. For example, the measuring unit 2 may be configured by three or more cameras.

  In this example, the feature quantity distribution calculation unit 21a calculates a plurality of first feature quantity distributions with one or more photoelectric conversion units 20a of the camera 2a as a unit, and uses an image of these as a first distribution image. Similarly, the feature quantity distribution calculation unit 21b calculates a plurality of second feature quantity distributions using one or more photoelectric conversion units 20b of the camera 2b as a unit, and uses an image of these as a second distribution image.

  The storage unit 3 includes a memory device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores various programs and various data. The storage unit 3 is connected to the processing unit 4 and inputs / outputs such information to / from the processing unit 4.

  The processing unit 4 is configured by an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit).

  The processing unit 4 is connected to the storage unit 3 and operates as various processing / control units by reading out and executing a program from the storage unit 3. The processing unit 4 stores various data in the storage unit 3 and reads out the various data.

  Furthermore, the processing unit 4 is also connected to the camera 2a, the camera 2b, and the output unit 5, and processes the distribution image from the camera 2a and the camera 2b to capture a common space from different directions. Then, the luminance signal obtained by imaging the corresponding position is recognized, and space recognition information (for example, the number of passers and passer positions) based on the distance image generated from the recognition result is output to the output unit 5.

  The output unit 5 is connected to the processing unit 4 and outputs the space recognition information from the processing unit 4 to the outside. Specifically, the output unit 5 is composed of a communication device, and transmits space recognition information to a remote security center via a predetermined network.

  Further, the configuration of each unit will be described with reference to the functional block diagram of FIG.

  Each of the plurality of photoelectric conversion units 20a-1, 20a-2,..., 20a-N includes an image sensor, an electronic shutter, an amplifier, and an A / D converter. Similarly, each of the plurality of photoelectric conversion units 20b-1, 20b-2,..., 20b-N includes an image sensor, an electronic shutter, an amplifier, and an A / D converter. The electronic shutter, amplifier, and A / D converter may be shared by two or more photoelectric conversion units 20a and 20b.

  The imaging device is composed of a CCD (Charge Coupled Devices) or a C-MOS (Complementary Metal Oxide Semiconductor), and measures the intensity of visible light in the monitoring space and outputs a luminance signal having the measured intensity as a voltage value. .

  For example, each photoelectric conversion unit 20a, 20b operates an electronic shutter every 1/500 seconds, amplifies the voltage of each imaging device at the time of operation with an amplifier, and samples with an A / D converter, thereby 1 / A digital luminance signal in which the luminance value is discretized into 256 gradations from 0 to 255 every 500 seconds is output.

  The plurality of image sensors are arranged on a predetermined imaging surface to form a surface, and project the monitoring space onto the surface. For example, the image sensors are arranged in an array on a flat substrate such that 640 elements are equally spaced in the X direction of the substrate and 480 elements are equally spaced in the Y direction, and 640 × 480 pixels every 1/500 second. The gray image is taken.

  In addition, a plurality of image pickup devices each include a set of image pickup devices each having three image pickup devices each having a color filter suitable for each of the R, G, and B bands on a flat substrate in the X direction. 640 sets at intervals and 480 sets at equal intervals in the Y direction can be arranged in an array to pick up a color image of 640 × 480 pixels.

  Further, RGB luminance signals may be converted into another color system such as HSV and output.

  The imaging period (measurement period) of the photoelectric conversion units 20a and 20b only needs to be sufficiently shorter than the change to be detected. For example, when the person is the main recognition object, 1/100 second or 1/200 second may be used. Good.

  The number of gradations and the number of photoelectric conversion units 20a and 20b are not limited to the above example, and can be set according to the detection target and the size of the space. Further, the number of photoelectric conversion units 20a and the number of photoelectric conversion units 20b may be different.

  In the present embodiment, the imaging surface of the photoelectric conversion unit 20a is the first imaging surface, and the small region set on the first imaging surface is the first small region. Further, the imaging surface of the photoelectric conversion unit 20b is the second imaging surface, and the small region set on the second imaging surface is the second small region. In addition, at least a part of the space projected onto the first imaging surface and the space projected onto the second imaging surface are common.

  In the present embodiment, the cameras 2a and 2b are stereo cameras, and the first image pickup surface and the second image pickup surface are set on the same plane so as to be separated from each other by an interval of human eyes.

  In another embodiment, the cameras 2a and 2b can be general field-of-view sharing cameras. In this case, the first imaging surface and the second imaging surface are set on different planes, for example, at positions separated by several meters.

  As described above, the plurality of photoelectric conversion units 20 a-1, 20 a-2,..., 20 a -N arranged on the first imaging surface capture an arbitrary space in an arbitrary time interval, and the photoelectric conversion unit 20 a. -1, 20a-2, ..., 20a-N each output a luminance signal. That is, the plurality of first measurement units arranged on the first measurement surface measure a physical quantity in an arbitrary space in an arbitrary time interval, and each outputs a measurement signal.

  In addition, the plurality of photoelectric conversion units 20b-1, 20b-2,..., 20b-N arranged on the second imaging surface different from the first imaging surface images the space in the time interval and performs photoelectric conversion. Each of the units 20b-1, 20b-2, ..., 20b-N outputs a luminance signal. That is, the plurality of second measurement units arranged on the second measurement surface different from the first measurement surface measure the physical quantity of the space in the time interval, and each outputs a measurement signal.

  In the present embodiment, the camera 2a functions including the feature amount distribution calculating unit 21a, the camera 2b functions including the feature amount distribution calculating unit 21b, and the processing unit 4 includes the corresponding index value distribution calculating unit 23, The corresponding area recognition unit 24, the distance image generation unit 25, and the passer-by recognition unit 26 function, and the storage unit 3 includes the camera parameter storage unit 22.

  The feature quantity distribution calculation unit 21a is a first feature quantity distribution calculation unit in the present invention, and includes a buffer and an arithmetic circuit. The buffer is composed of, for example, a ring buffer, holds the luminance signals from the plurality of photoelectric conversion units 20a-1, 20a-2,..., 20a-N for a certain period of time, and cyclically stores the luminance values for each imaging cycle.

  The feature quantity distribution calculation unit 21b is a second feature quantity distribution calculation unit according to the present invention, and includes a buffer and an arithmetic circuit similarly to the feature quantity distribution calculation unit 21a. The buffer is composed of, for example, a ring buffer, holds the luminance signals from the plurality of photoelectric conversion units 20b-1, 20b-2,..., 20b-N for a certain period of time, and circulates and stores the luminance values for each imaging cycle.

  In this example, the number of buffers equal to or greater than a predetermined frequency calculation time width is provided for the number of photoelectric conversion units 20a and 20b.

  For example, if the imaging cycle is 1/500 seconds and the frequency calculation time width is 1/5 seconds, buffers having a buffer length of 100 or more are prepared as many as the number of photoelectric conversion units 20a and 20b.

  The arithmetic circuit calculates the time frequency for each feature amount from the luminance signal of the frequency calculation time width held in the buffer in the frequency calculation cycle, generates a feature amount distribution, and outputs the feature amount distribution to the corresponding index value distribution calculation unit 23.

  For example, the arithmetic circuit calculates a normalized histogram of feature amounts in each of a plurality of small regions set in advance on the imaging surface of the imaging unit 2.

  For example, the small area can be the pixel of the gray image, and the feature amount can be the luminance value of the gray image. That is, the photoelectric conversion units 20a and 20b and the small regions have a one-to-one correspondence, and the feature amount is a scalar.

  Further, for example, the small area may be the pixel of the color image, and the feature amount may be the R, G, and B luminance values. In this case, the photoelectric conversion units 20a and 20b and the small regions have a one-to-one correspondence, and the feature amount is a three-dimensional vector.

  Furthermore, for example, a plurality of neighboring pixels (2 × 2 pixels, 3 × 3 pixels, etc.) in the grayscale image or color image may be set as a small region. In this case, the original data for calculating the time frequency in each small area can be increased while performing resolution conversion.

  The frequency calculation cycle can be set to 1/5 second with respect to the frequency calculation time width of 1/5 second. In this case, there is no overlap or thinning.

  Note that the frequency calculation cycle may be set shorter than the frequency calculation time width to provide overlap, or the frequency calculation cycle may be set longer than the frequency calculation time width and thinned.

  As described above, the feature amount distribution calculating unit 21a calculates the time frequency for each feature amount from the measurement signal (luminance signal) in the first small area set on the first measurement surface (on the first imaging surface of the camera 2a). To calculate a feature amount distribution (first feature amount distribution).

  The feature amount distribution calculation unit 21b is characterized by measurement signals (luminance signals) in each of a plurality of second small regions set on a second measurement surface (on the second imaging surface of the camera 2b) different from the first measurement surface. A feature amount distribution (second feature amount distribution) is calculated by calculating a time frequency for each amount. In each feature quantity distribution, the time frequency of the true feature quantity that is a non-fluctuating component shows a significantly high value. Compared with this, the time frequency of the false feature quantity that is a fluctuation component is sufficiently low. Become. For example, even if sudden noise occurs and the amplitude is large, the time frequency of the false feature amount due to the noise is sufficiently low. Therefore, it is possible to distinguish a true feature value from a false feature value by using a feature value distribution.

  Here, FIG. 4A shows an example of the relationship between the luminance signal time and the luminance value in a certain pixel. As shown in FIG. 4A, it is assumed that the luminance value changes with time in the range of 0-255. When the number of luminance values of the frequency calculation time width T is 100, for example, the frequency value of each luminance value is counted, and when the time frequency is calculated by dividing each of the counting results by 100, the luminance value and time in the pixel are calculated. The frequency has a relationship like a normalized histogram shown in FIG.

  The camera parameter storage unit 22 has a camera parameter of the camera 2a and a camera parameter of the camera 2b in an XYZ coordinate system (world coordinate system) common to both the cameras 2a and 2b predetermined in a space captured by the camera 2a and the camera 2b. Is stored in advance.

  Each camera parameter of the camera 2a and the camera 2b includes an external parameter and an internal parameter, as will be described below.

  The external parameters of the camera 2a include the position (viewpoint) of the camera 2a in the world coordinate system, the shooting direction, and the angle of view. The external parameters of the camera 2b include the position of the camera 2b in the world coordinate system, the shooting direction, and the angle of view.

  The internal parameters of the camera 2a include the focal length, center coordinates, and distortion coefficient of the camera 2a. The internal parameters of the camera 2b include the focal length, center coordinates, and distortion coefficient of the camera 2b.

  In addition, by applying the camera parameters of the camera 2a to the pinhole camera model, coordinate conversion can be performed between the coordinates of the world coordinate system and the coordinate system of the photographing surface of the camera 2a. Similarly, by using the camera parameters of the camera 2b, coordinate conversion can be performed between the coordinates of the world coordinate system and the coordinate system of the imaging surface of the camera 2b.

  The correspondence index value distribution calculation unit 23 is a correspondence index value between small regions between the first feature amount distribution calculated by the feature amount distribution calculation unit 21a and the second feature amount distribution calculated by the feature amount distribution calculation unit 21b. Each time frequency is calculated and output to the corresponding region recognition unit 24.

  The correspondence index value is a value representing the difference or similarity between the feature quantities, and is a value serving as an index for recognizing the correspondence.

  For example, the feature amount can be a luminance value of a grayscale image, and the corresponding index value can be a luminance difference. In this case, the luminance difference that is the corresponding index value is a value representing the difference between the feature amounts. Therefore, the small areas having a small luminance difference have a lower difference and are more likely to correspond. Conversely, the small areas having a larger luminance difference have a higher difference and have a lower possibility of corresponding.

  The time frequency for each correspondence index value between the small areas is expressed in the form of a normalized histogram, for example. Hereinafter, this is referred to as a corresponding index value distribution. The correspondence index value distribution is information serving as a basis for recognizing the second small area corresponding to the first small area.

  The corresponding index value distribution calculation unit 23 sets a plurality of second small areas as candidates for each of the first small areas, and sets a search range for each of the first small areas in the second distribution image (that is, in the second measurement plane). Then, a corresponding index value distribution is calculated between the first small area and the second small area within the search range. The search range is, for example, a projection area of pixels in the first distribution image onto the second distribution image.

  Specifically, the correspondence index value distribution calculation unit 23 converts the coordinates of an arbitrary pixel in the first feature amount distribution expressed in the coordinate system of the first imaging surface to the world coordinate system using the camera parameters of the camera 2a. The epipolar line is derived by projection, and the epipolar line expressed in the world coordinate system is projected onto the coordinate system of the second imaging plane using the camera parameters of the camera 2b. Then, the projection image of the epipolar line and its vicinity are set as a search range for an arbitrary pixel in the first feature amount distribution.

  For example, in an example in which the cameras 2a and 2b are stereo cameras, a search range along the x-axis direction of the second imaging surface can be set, and the number n of pixels in the search range along the x-axis direction is an epipolar line. It can be determined from the projected image.

  When the corresponding index value is a luminance difference, the corresponding index value distribution calculating unit 23 calculates the time frequency Pa (i) of the luminance value i at any pixel (x, y) constituting the first feature amount distribution and the second A luminance difference distribution Ps (x, y, d, | i−j |) is represented by the following formula (1) based on the time frequency Pb (j) of the luminance value j in the corresponding pixel candidate (x + d, y) in the feature amount distribution. ).

  However, d is a shift width (parallax), and 0 ≦ d ≦ n. Further, ∫j∫ididj means summing up products of time frequencies for combinations of luminance values i and j having a luminance difference of | i−j |. Incidentally, the value ranges of the luminance values i, j and the luminance difference | i−j | between i and j are all 0-255.

  Hereinafter, the element Ps (x, y, d, 0) having a luminance difference of 0 in the luminance difference distribution Ps will be described.

  The luminance value j on the second feature value distribution side where the luminance difference | i−j | becomes 0 is 0, 1,..., With respect to the luminance value i = 0, 1,. , 255, the product of the time frequency Pa (x, y, 0) on the first feature quantity distribution side and the time frequency Pb (x + d, y, 0) on the second feature quantity distribution side, the first feature quantity distribution side. Product of the time frequency Pa (x, y, 1) of the second and the time frequency Pb (x + d, y, 1) on the second feature quantity distribution side, ..., the time frequency Pa (x, y, 255 on the first feature quantity distribution side) ) And the time frequency Pb (x + d, y, 255) on the second feature quantity distribution side are summed to obtain an element Ps (x, y, d, 0).

  Next, the element Ps (x, y, d, 1) having a luminance difference of 1 in the luminance difference distribution Ps will be described.

  The luminance value j on the second feature value distribution side where the luminance difference | i−j | becomes 1 is 1, 0 with respect to the luminance value i = 0, 1, 2,..., 255 on the first feature value distribution side, respectively. , 2, 1 and 3, ..., 254, the product of Pa (x, y, 0) and Pb (x + d, y, 1), Pa (x, y, 1) and Pb (x + d, y, 0) ) And the product of Pa (x, y, 1) and Pb (x + d, y, 2), the product of Pa (x, y, 2) and Pb (x + d, y, 1) and Pa (x, y, 2) and the product of Pb (x + d, y, 3), ..., the product of Pa (x, y, 255) and Pb (x + d, y, 254) is summed to obtain element Ps (x, y, d, 1) is obtained.

  Hereinafter, Ps (x, y, d, 255) is similarly obtained from Ps (x, y, d, 2).

  That is, for each first small region, the corresponding index value distribution calculating unit 23 determines, for each corresponding index value, the combination of the time frequency of the feature amount of the first small region and the time frequency of the feature amount of the second small region. The product of the time frequency for each combination for which the corresponding index value is calculated is summed to calculate the time frequency of the corresponding index value.

  FIG. 5 shows an example of the luminance difference distribution. In the example of FIG. 5, the shift width d = 0, 1, 2, 3, and 4, and for one pixel (x, y) in the first distribution image, there are five corresponding pixels in the second distribution image. There are (x, y), (x + 1, y), (x + 2, y), (x + 3, y), (x + 4, y), and the luminance difference distribution with each of these five candidates is Ps (x, y, 0, | i−j |), Ps (x, y, 1, | i−j |), Ps (x, y, 2, | i−j |), Ps (x, y, 3, | i−j |) , Ps (x, y, 4, | i−j |).

  In the above example, the small area is one pixel, but similar luminance values may be continuous on a plane in the monitoring space. Therefore, the luminance difference distribution may be calculated by setting a small region composed of a plurality of pixels including neighboring pixels so as to overlap each other, and counting the time frequency for each luminance difference over the plurality of pixels. By doing so, even if similar luminance values continue on the image, it becomes difficult to erroneously recognize the corresponding small region.

  For example, in the example in which the cameras 2a and 2b are stereo cameras, when both the first small area and the second small area are w × 1 pixels, the expression (1) is expanded as the following expression (2).

  Here, u is a search point, and −w / 2 ≦ u ≦ w / 2.

  As described above, the correspondence index value distribution calculation unit 23 represents the difference or similarity between the feature quantities between the first small area and each of the plurality of second small areas for each first small area. Corresponding index values are obtained, and a corresponding index value distribution having the time frequency for each corresponding index value as an element is calculated. At this time, as illustrated in Expressions (1) and (2), the correspondence index value distribution calculation unit 23 calculates the time frequency of the feature amount of the first small region and the feature of the second small region for each correspondence index value. The time frequency of the corresponding index value is calculated by summing up the products of the time frequencies for each combination for which the corresponding index value is calculated among the combinations of the time frequencies of the quantities.

  In the correspondence index value distribution calculated in this way, the relationship in which the time frequency of the true correspondence index value by the non-fluctuation component is significantly higher than the time frequency of the false correspondence index value is maintained. Therefore, the true correspondence index value can be used separately from the false correspondence index value.

  The corresponding area recognition unit 24 recognizes a position in the monitoring space corresponding to the first small area from among the plurality of second small areas, using the correspondence index value distribution of the plurality of second small areas.

  Specifically, the corresponding area recognition unit 24 calculates the norm of the corresponding index value distribution for each corresponding index value distribution calculated for the first small area, and sets each norm in advance. The second small area for which the lowest norm is calculated is determined as the second small area corresponding to the first small area from the second small areas whose norm is equal to or smaller than the threshold.

  The corresponding area recognition unit 24 determines that there is no corresponding second small area for the first small area for which the norm equal to or less than the threshold value has not been calculated. In addition, when there are a plurality of second small areas whose norms are equal to or less than the threshold value, the corresponding area recognition unit 24 selects the second small area having the smallest sum of norms of each second small area and the second small area adjacent thereto. By determining the second small area corresponding to the first small area, accidental misassociation may be prevented.

  Here, the norm of the luminance difference | i−j | in the luminance value distribution Ps (x, y, d, | i−j |) is a product sum value Σ | i obtained by summing up the products of the luminance difference and the time frequency. −j | · Ps (x, y, d, | i−j |). In the case of this definition, since a lower norm is more likely to be calculated in a distribution biased to the left side of the luminance difference distribution (the side where the luminance difference | i−j | is lower), the second small region in which the lower luminance difference appears more frequently. It becomes easy to be recognized as the corresponding second small region.

  In the example of FIG. 5, the second small area where d = 0 is recognized as the corresponding second small area.

  Also, for example, for each first small area, the mode value of the corresponding index value for each corresponding index value distribution calculated for the first small area (that is, for each combination of the first small area and the second small area) And the second small region in which the lowest mode value is detected is recognized as the second small region corresponding to the first small region.

  As described above, in the corresponding index value distribution, the time frequency of the true corresponding index value due to the non-fluctuating component is significantly higher than the time frequency of the false corresponding index value due to the fluctuation component. Therefore, the corresponding region recognition unit 24 recognizes small region correspondences selectively using the correspondence index value having a high time frequency, thereby effectively reducing erroneous recognition due to fluctuation components, and corresponding relationships with high accuracy. Can be recognized.

  The distance image generation unit 25 calculates, for each first small region, the distance from the corresponding second small region shift amount (parallax) dmin and the camera parameter to a point in the monitoring space projected on the small region. To generate a distance image.

  For example, when the pixel (x, y) in the first feature distribution corresponds to the pixel (x + dmin, y) in the second feature distribution shifted by dmin in the x-axis direction, the first feature from the camera 2a. The distance D to the point in the monitoring space projected on the pixel (x, y) in the quantity distribution is obtained by the following equation using the camera parameters.

D = f · h / | dmin | (3)
Here, f is the focal length of the camera 2a, and h is the distance between the camera 2a and the camera 2b.

  The passer-by recognition unit 26 recognizes the number of passers-by and the passer-by positions existing in the monitoring space from the distance image, and outputs the recognition result to the output unit 6.

  For example, a distance image generated when no person is present in the monitoring space is stored in the storage unit 3, and a difference area between the newly generated distance image and the distance image in the storage unit 3 is extracted. A difference area including an edge having a distance difference equal to or greater than the width of the person in the area is separated from the edge. And while counting the number of difference areas which are a person's size range among difference areas as a passer's number, the position of each said difference area is detected as a passer's position.

[Operation of space recognition device]
Next, operations of the camera 2a and camera 2b and the processing unit 4 in the space recognition apparatus configured as described above will be described with reference to the flowchart of FIG.

  In the camera 2a, each photoelectric conversion unit 20a periodically samples the luminance value by A / D converting the voltage of the image sensor every time the electronic shutter is operated, and the sampled luminance value is converted into a feature amount distribution calculation unit. The luminance signal is buffered by 21a writing to the buffer (S1).

  Similarly, in the camera 2b, each photoelectric conversion unit 20b periodically samples the luminance value measured by the image sensor, and the feature amount distribution calculation unit 21b buffers the luminance signal (also S1).

  The feature amount distribution calculating unit 21a and the feature amount distribution calculating unit 21b each determine whether or not the frequency calculation period has arrived based on the number of times of writing to the buffer or the latest writing position in the buffer (S2).

  For example, if the imaging cycle is 1/500 seconds and the frequency calculation cycle is 1/5 second, the frequency calculation cycle comes every time 100 luminance values are written.

  If the frequency calculation period has not arrived (NO in S2), the feature quantity distribution calculation unit 21a and the feature quantity distribution calculation unit 21b each return the process to step S1 and continue buffering.

  On the other hand, when the frequency calculation period arrives (YES in S2), each of the feature amount distribution calculating unit 21a and the feature amount distribution calculating unit 21b advances the process to step S3.

  The feature amount distribution calculation unit 21a reads a luminance signal having a frequency calculation time width that ends at the latest writing position from the buffer of the camera 2a for each first small region, and calculates the time frequency of the feature amount for each first small region. Then, a first feature amount distribution is calculated (S3).

  For example, if the imaging cycle is 1/500 seconds, the frequency calculation time width is 1/5 seconds, the luminance value as a feature value is an integer of 0 to 255, and the first small area is a pixel, 100 pixels are provided for each pixel. The frequency of luminance values 0, 1,..., 255 is counted from the luminance value, and a normalized histogram obtained by dividing each of the counting results by 100 is calculated.

  Similarly, the feature amount distribution calculation unit 21b reads out a luminance signal having a frequency calculation time width that ends at the latest writing position from the buffer of the camera 2b for each second small region, and sets the feature amount time for each second small region. The frequency is calculated to calculate the second feature amount distribution (S4).

  Steps S3 and S4 are described in order for explanation, but are processed in parallel in each camera.

  The camera 2a outputs a first distribution image that is a collection of the calculated first feature amount distributions to the processing unit 4. Similarly, the camera 2b outputs a second distribution image that is a collection of the calculated second feature value distributions to the processing unit 4.

  In the flowchart of FIG. 6, the processes of the camera 2a and the camera 2b and the processing unit 4 are expressed as a series of processes. However, the camera 2a and the camera 2b that output the feature amount distribution are processed by the processing unit 4 in steps S5 to S12. The process may be returned to step S1 without waiting for the end of the process.

  The corresponding index value distribution calculation unit 23 of the processing unit 4 sequentially sets each of the first small regions set in the input first distribution image as a target small region (S5), and performs steps S5 to S9. Perform loop processing.

  In the loop processing, the correspondence index value distribution calculation unit 23 first reads the camera parameters from the camera parameter storage unit 22, and calculates the search range by projecting the attention subregion onto the second distribution image using the camera parameters. Then, the second small region in the search range is set as a candidate for the second small region corresponding to the target small region (S6).

  Subsequently, the corresponding index value distribution calculating unit 23 calculates a corresponding index value distribution between the target small area and each candidate set in step S6 (S7).

  For example, the correspondence index value distribution calculation unit 23 calculates the formula (2) for each pixel in the first distribution image that is the target small region and a plurality of pixels in the second feature amount distribution set as a candidate pixel corresponding to the pixel. The luminance difference distribution is calculated by applying 1). For each pixel of the first distribution image, a normalized histogram (luminance difference distribution) composed of time frequencies of luminance differences of 0 to 255 is calculated for the number of candidates.

  Subsequently, the corresponding area recognition unit 24 of the processing unit 4 detects a second small area corresponding to the target small area from among a plurality of candidates based on the corresponding index value distribution calculated in step S7 (S8). .

  For example, the norm of each of the luminance difference distributions calculated in step S7 is calculated and compared with a threshold value, and the second small region in which the minimum norm is calculated among the norms equal to or less than the threshold value is detected.

  Then, the corresponding index value distribution calculation unit 23 checks whether all the first small areas have been processed (S9).

  If there is an unprocessed first small area (NO in step S9), the corresponding index value distribution calculation unit 23 returns the process to step S5 and processes the next first small area.

  On the other hand, if all the first small regions have been processed (YES in step S9), the distance image generation unit 25 of the processing unit 4 generates a distance image (S10).

  In other words, the distance image generation unit 25 reads the camera parameters from the camera parameter storage unit 22 and, for each first small region of the first distribution image, according to Equation (3) into which the camera parameters are substituted, A distance image is generated by converting a shift amount (parallax) dmin of the second small area in which the correspondence relationship with the small area is detected with respect to the first small area into a distance D from the camera 2a.

  Subsequently, the passer-by identification unit 26 of the processing unit 4 detects a difference area by the passer-by from the distance image generated in step S10, and counts the number of passers-by and detects the passer-by position (S11). The counted number of passers and the detected passer position (spatial recognition information) are output to the output unit 5 (S12).

  When the above process is completed, the process returns to step S1, the camera 2a and the camera 2b calculate the next feature quantity distribution, and the processing unit 4 waits for the input of the feature quantity distribution from the camera 2a and the camera 2b.

[Modification]
In the above-described embodiment, an example in which a luminance difference, that is, a difference between feature quantities is used as a corresponding index value representing a difference between feature quantities has been described. In another embodiment, a distance value between feature amounts can be used as a correspondence index value representing a difference between feature amounts. Also in this case, for each distance value, the correspondence index value distribution calculation unit 23 calculates the distance value from the combination of the time frequency of the feature amount of the first small region and the time frequency of the feature amount of the second small region. The product of the time frequencies for each combination is summed, and the time frequency of the distance value is calculated to obtain a distance value distribution having the time frequency for each distance value as an element. In addition, the corresponding area recognition unit 24 calculates the product sum value (norm) of the distance value and the time frequency for each of the distance value distributions, and among the plurality of second small areas, the product sum value is equal to or less than a predetermined value. A second small area obtained by measuring a position in the space corresponding to the first small area is determined from the areas.

  In another embodiment, it is possible to use a corresponding index value representing the similarity between feature quantities. In this case, the corresponding index value can be a similarity or a correlation value. For example, a value obtained by subtracting the above-described luminance difference from 255 is an example of similarity. Also in this case, the correspondence index value distribution calculation unit 23 calculates the similarity among the combinations of the time frequency of the feature amount of the first small region and the time frequency of the feature amount of the second small region for each similarity (or correlation value). The time frequency for each similarity (or correlation value) is calculated by summing the products of the time frequencies for each combination for which the degree (or correlation value) is calculated, and calculating the time frequency of the similarity (or correlation value). Find the similarity distribution (or correlation value distribution) as an element. In addition, the corresponding region recognition unit 24 calculates the product sum value of the similarity (or correlation value) and the time frequency for each of the similarity distribution (or correlation value distribution), and the product sum value of the plurality of second small regions is calculated. A second small region obtained by measuring a position in a space corresponding to the first small region is determined from the second small regions that are equal to or greater than a predetermined value.

  Furthermore, in the above-described embodiment, an example in which a corresponding small area is recognized using a camera in which camera parameters are set in advance has been described. However, a process for recognizing a corresponding small area in order to calculate camera parameters. It is also possible to apply to (so-called calibration). In this case, for example, the feature quantity distribution calculation units 21a and 21b calculate the first feature quantity distribution and the second feature quantity distribution in the same manner as described above, and the processing unit 4 accepts a setting input from the user or automatically sets feature points. A plurality of correspondence relationships between the first small area and the second small area are initially set by a detection process or the like. Corresponding index value distribution calculation unit 23 selects, for each first small region for which an initial value is set, a second small region that is initially set for correspondence with the first small region and a second small region in the vicinity thereof. The corresponding index value distribution is calculated as follows, and the corresponding area recognition unit 24 recognizes a second small area corresponding to each first small area from among a plurality of candidates. Then, the processing unit 4 applies a known calibration method such as a method using a homography matrix to the coordinates of the first small region and the second small region for which the corresponding region recognition unit 24 recognized the correspondence relationship, and sets camera parameters. calculate.

  Incidentally, in the above-described embodiment and each modification, the feature amount distribution calculation unit 21 may be realized in the processing unit 4. In that case, the imaging unit 2 outputs a captured image in which luminance values are arranged for each imaging cycle to the processing unit 4. The processing unit 4 operates as the feature amount distribution calculating unit 21, buffers the captured image from the imaging unit 2 in the storage unit 3, and generates a feature amount distribution for each frequency calculation period.

  In the above-described embodiment and each modification, the measurement unit outputs a luminance signal having an amplitude corresponding to the intensity of visible light in the monitoring space, but is not limited thereto.

  For example, a configuration in which the measurement unit outputs a luminance signal having an amplitude corresponding to the intensity of near infrared light in the monitoring space, a configuration in which the measurement unit outputs a luminance signal having an amplitude according to the intensity of infrared light in the monitoring space, and the measurement unit A configuration for outputting a luminance signal having an amplitude corresponding to the intensity of ultraviolet light in the monitoring space, a configuration for the measurement unit to output a luminance signal having an amplitude corresponding to the intensity of the X-ray in the monitoring space, and a distance sensor as the measurement unit A configuration in which the measurement unit measures a predetermined physical quantity in the space and outputs a measurement signal, such as a configuration that outputs a distance signal having an amplitude corresponding to the distance from the measurement unit to an object existing in the monitoring space, may be used. As the distance sensor, for example, a method of projecting and receiving laser light, a method of transmitting and receiving electromagnetic waves such as microwaves and millimeter waves, and a method of transmitting and receiving ultrasonic waves are conceivable.

  The best mode of the space recognition device according to the present invention has been described above, but the present invention is not limited by the description and drawings according to this mode. That is, it is a matter of course that all other forms, examples, operation techniques, and the like made by those skilled in the art based on this form are included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Space recognition apparatus 2 Imaging part 2a, 2b Camera 3 Memory | storage part 4 Processing part 5 Output part 20a (20a-1, 20a-2, ..., 20a-N), 20b (20b-1, 20b-2, ..., 20b) -N) Photoelectric conversion unit 21a, 21b Feature quantity distribution calculation unit 22 Camera parameter storage unit 23 Corresponding index value distribution calculation unit 24 Corresponding region recognition unit 25 Distance image generation unit 26 Passer recognition unit

Claims (5)

From the first measurement signal output by each of the first measurement units, one or more first measurement units arranged on the first measurement surface measure a physical quantity in an arbitrary space in an arbitrary time interval. A first feature amount distribution calculating unit that calculates a time frequency for each feature amount in the first small region set on the measurement surface;
Each of a plurality of second measurement units arranged on a second measurement surface different from the first measurement surface measures a physical quantity of the space in the time interval and is output by each second measurement unit. A second feature amount distribution calculating unit that calculates a time frequency for each feature amount in each of the plurality of second small regions set on the second measurement surface from the signal;
A correspondence index value representing the difference or similarity between the feature quantities is obtained between the first small area and each of the plurality of second small areas, and the correspondence is obtained using the time frequency for each correspondence index value as an element. A corresponding index value distribution calculating unit for calculating the index value distribution;
Using the correspondence index value distribution of the plurality of second small areas, a second small area obtained by measuring a physical quantity at a position in the space corresponding to the first small area from among the plurality of second small areas. A corresponding area recognition unit for recognizing;
A space recognition device comprising: The correspondence index value distribution calculating unit calculates, for each correspondence index value, a correspondence index value of a combination of a time frequency of the feature amount of the first small area and a time frequency of the feature amount of the second small area. The space recognition device according to claim 1, wherein the time frequency of the corresponding index value is calculated by summing up the products of the calculated time frequencies for each combination. The corresponding index value distribution calculating unit calculates the corresponding index value distribution by obtaining the corresponding index value indicating the difference between the feature quantities;
The corresponding area recognition unit calculates a product sum value of a corresponding index value and a time frequency in the corresponding index value distribution for each of the corresponding index value distributions, and the product sum value of the plurality of second small areas is predetermined. The space recognition apparatus according to claim 1, wherein the space recognition device recognizes a second small region obtained by measuring a physical quantity at a position in the space that determines the first small region from second small regions that are equal to or less than a value. The corresponding index value distribution calculating unit calculates the corresponding index value distribution by obtaining the corresponding index value representing the similarity between the feature quantities;
The corresponding area recognition unit calculates a product sum value of a corresponding index value and a time frequency in the corresponding index value distribution for each of the corresponding index value distributions, and the product sum value of the plurality of second small areas is predetermined. The space recognition apparatus according to claim 1, wherein the space recognition device recognizes a second small region obtained by measuring a physical quantity at a position in the space that determines the first small region from second small regions that are equal to or greater than a value. The plurality of first measurement units are arranged on the first imaging surface with the first measurement surface as a first imaging surface, and each of the first measurement units images the space and outputs a first luminance signal as the first measurement signal. It consists of a plurality of first photoelectric conversion units that output,
The plurality of second measurement units are arranged on the second imaging surface with the second measurement surface different from the first measurement surface as a second imaging surface, and each of the second measurement units images the space and performs the second measurement. The space recognition apparatus according to claim 1, comprising a plurality of second photoelectric conversion units that output a second luminance signal as a signal.

Images