JP7411539B2 - Ranging system and its coordinate calibration method - Google Patents

Description

The present invention relates to a distance measurement system that generates a distance image of an object using a distance measurement sensor and an imaging camera, and a method for calibrating coordinates between the distance measurement sensor and the imaging camera.

A TOF sensor (TOF = time of flight) is known as a distance measuring sensor that measures the distance to an object based on the transmission time of light. By displaying distance data acquired by a TOF sensor as a distance image and tracking changes over time, it is possible to obtain, for example, a moving route (flow line data) of a person or the like.

TOF sensors can accurately measure distances, but compared to imaging cameras, they have lower resolution and black-and-white data (IR brightness), so they are not suitable for image analysis such as facial recognition and attribute information of people. Therefore, a method has been proposed in which a TOF sensor and an imaging camera are combined, and attribute information obtained by the imaging camera is superimposed and displayed on a distance image obtained by the TOF sensor.

In order to operate the TOF sensor and the imaging camera in cooperation, the position data of the object acquired by both must match. That is, it is necessary to perform calibration to match deviations in coordinates and measurement directions between the TOF sensor and the imaging camera.

Regarding this, for example, Patent Document 1 describes an imaging device that searches for an occlusion area by integrating distance information acquired by a TOF camera (TOF sensor) and brightness information acquired by a CCD camera. In order to calibrate both, there is an extrinsic parameter estimator that estimates extrinsic parameters using the distance image obtained by the TOF camera and the brightness image obtained by the CCD camera, and an extrinsic parameter estimation section that estimates the extrinsic parameters using the distance image obtained by the TOF camera and the brightness image obtained by the CCD camera, and the It is stated that the apparatus includes a corresponding pixel determination unit that determines the correspondence between pixel positions in a distance image and pixel positions in a brightness image.

Japanese Patent Application Publication No. 2011-123071

In a conventional calibration method, an area of the object detected by a TOF sensor, for example, a human detection frame that is an area surrounding a person, is displayed superimposed on a camera image of the object obtained by an imaging camera. The operator manually performed calibration (changing parameters such as camera coordinates and direction) so that the human detection frame detected by the TOF sensor matched the position of the target person in the camera image. In reality, even if the human detection frames are matched once, they often deviate if the person moves, so calibration must be repeated while the person moves to search for optimal parameters. Therefore, a lot of time was required to perform the calibration work.

On the other hand, in the calibration method described in Patent Document 1, in order to superimpose the distance image of the TOF sensor and the captured image of the CCD camera, external parameters (rotation, translation) and internal parameters (focal length, lens distortion coefficient, optical center, The value of pixel size) is estimated and given. In this case, in order to match the distance image and the photographed image pixel by pixel, it is necessary to set these parameters with extremely high precision, and it is expected that the processing load will be large and more time will be required.

Further, since the technique of Patent Document 1 is based on the premise that the distance image and the photographed image match, the TOF sensor and the CCD camera must be installed at approximately the same position and direction with respect to the object. Therefore, for example, a TOF sensor and a CCD camera cannot be installed facing each other with the object in between.

An object of the present invention is to provide a distance measurement system and a calibration method that more easily and automatically perform coordinate calibration processing of a TOF sensor and an imaging camera in the distance measurement system.

The present invention provides a distance measurement system that generates a distance image of an object by installing a distance measurement sensor and an imaging camera, wherein the distance measurement sensor is a TOF sensor that measures the distance to the object based on the transmission time of light. The TOF image obtained by the TOF sensor and the camera image obtained by the imaging camera are aligned, and the object attribute obtained from the camera image is added to the movement trajectory of the object obtained from the TOF image. It is equipped with a cooperative processing device that displays information in a superimposed manner. In order to align the TOF image and the camera image, the cooperation processing device creates a person detection frame Fb surrounding the person detected from the TOF image, and a person detection frame Fa surrounding the person detected from the camera image. The installation parameters of the TOF sensor and the imaging camera are calibrated so that they match on a common screen.

The present invention also provides a method for calibrating the coordinates of the distance measurement sensor and the imaging camera when the distance measurement sensor and the imaging camera are installed to generate a distance image of an object, in which the distance measurement sensor has a light transmission time. A TOF sensor that measures a distance to an object based on the following steps: acquiring a TOF image with the TOF sensor, acquiring a camera image with the imaging camera, and detecting a person from the TOF image and identifying the person. creating a surrounding person detection frame Fb, detecting a person from the camera image and creating a person detection frame Fa surrounding the person, and matching the person detection frame Fb and the person detection frame Fa on a common screen. The method further includes the step of calibrating installation parameters of the TOF sensor and the imaging camera.

According to the present invention, calibration is performed by matching the positions of the human detection frames in the camera image and the TOF image. As a result, compared to the conventional case of matching camera images and TOF images pixel by pixel or matching camera images and human detection frames (TOF images), the processing load is smaller and the processing time is shorter. This has the effect that calibration can be automatically performed more easily.

FIG. 1 is a schematic diagram of a ranging system 100 according to the present embodiment. FIG. 3 is a block configuration diagram of the cooperation processing device 3. FIG. 2 is a diagram illustrating the principle of distance measurement by the TOF sensor 2. FIG. 2 is a diagram showing an example of how an imaging camera 1 and a TOF sensor 2 are installed. The figure which shows creation of a person detection frame from a camera image and a TOF image. FIG. 3 is a diagram illustrating coordinate transformation and calibration processing of a human detection frame. FIG. 6 is a diagram showing a specific example of parameter changes during calibration. A flowchart showing the procedure (first half) of calibration processing. Flowchart showing the procedure (second half) of calibration processing. FIG. 3 is a diagram showing an example of measurement by the distance measuring system of the present embodiment.

Embodiments of the present invention will be described below. In the calibration method for the ranging system of this embodiment, a human detection frame indicating the area of a person moving in the measurement space is acquired by both a ranging sensor (TOF sensor) and an imaging camera, and the acquired human detection The installation parameters of the imaging camera are automatically adjusted so that the positions of the frames match in a common coordinate system.

FIG. 1 is a schematic diagram of a ranging system 100 according to this embodiment. The distance measuring system 100 has a configuration in which an imaging camera 1 and a TOF sensor 2 are connected to a cooperative processing device 3 via a switching hub 4. A plurality of imaging cameras 1 and TOF sensors 2 are installed as necessary. The imaging camera 1 is an RGB camera such as a surveillance camera, and acquires image data of an object 9 (for example, a person). The TOF sensor 2 acquires distance data to the object 9 in a two-dimensional manner.

The acquired data is transmitted via the switching hub 4 to the cooperative display device 3, for example, a PC (personal computer). The cooperative display device 3 acquires the movement trajectory (flow line) of the object 9 from changes in distance data over time, and also analyzes image data to acquire attribute information such as the gender and age of the object (person). , the attribute information of the object is superimposed and displayed on the flow line of the object. The switching hub 4 is, for example, a switching hub compatible with PoE (Power over Ethernet), and performs data transmission with the imaging camera 1 and the TOF sensor 2, as well as supplies power thereto.

FIG. 2 is a block diagram of the cooperative processing device 3. As shown in FIG. The basic configuration of the cooperative processing device 3 (PC) is that a camera image is input from the imaging camera 1 and a distance image (hereinafter also referred to as a TOF image) is input from the TOF sensor 2, and the image synthesis unit 36 uses the image data of both. The flow line data and attribute information of the object are combined and displayed on the image display section 37. At this time, the calibration unit 34 automatically adjusts the position information of the imaging camera 1 and the TOF sensor 2, that is, the installation parameters, using automatic adjustment software installed in the device. Thereby, the flow line data and the attribute information can be displayed in the image display section 37 in a correctly correlated manner.

The cooperative processing device 3 performs arithmetic processing such as image composition and calibration, and achieves the above functions by storing a program used for this in a ROM (not shown), developing it in a RAM, and executing it by the CPU.

The configuration of the cooperative processing device 3 will be explained in detail. The cooperation processing device 3 includes two person detection units 31 and 32 for input camera images and TOF images. The person detection unit 31 performs image analysis on the camera image acquired by the imaging camera 1 and creates a person detection frame Fa. On the other hand, the person detection unit 32 performs person detection on the distance image (TOF image) acquired by the TOF sensor 2 and creates a person detection frame Fb.

The coordinate transformation section 33 coordinates transforms the human detection frame Fb created by the human detection section 32 into a human detection frame Fb' in a coordinate system with the imaging camera 1 as a viewpoint, and outputs the result. In the coordinate transformation, the installation parameters of the imaging camera 1 and the TOF sensor 2 stored in the parameter storage section 35 are read out, and calculations are performed using these as transformation coefficients. The calibration unit 34 detects the amount of deviation between the two input human detection frames Fa and Fb', and changes the installation parameters of the imaging camera 1 so that the two human detection frames Fa and Fb' match on the common screen. do. The changed parameters are stored in the parameter storage unit 35 and used as transformation coefficients when the human detection frame Fb is then coordinate transformed to the camera viewpoint.

Once the calibration is complete, actual measurement of the object (flow line display, etc.) is performed. Attribute information of the person's face image-analyzed by the person detection unit 31 and flow line data acquired by the person detection unit 32 and coordinate transformed by the coordinate conversion unit 33 are sent to the image synthesis unit 36. The image synthesis section 36 synthesizes both images and displays them on the image display section 37 (display). As a result, the image display unit 37 displays the person's attribute information (gender, age information, etc.) superimposed on the movement trajectory (flow line) of the person, for example. At that time, the flow line data and attribute information are associated and displayed in a linked manner using time stamps. Note that the image display section 37 may be a device externally connected to the cooperative processing device 3. Further, the cooperative processing device 3 includes a transmitter (not shown) that transmits an instruction signal for image acquisition to the imaging camera 1 and the TOF sensor 2.

Here, the adjustment of installation parameters performed by the calibration section 34 will be explained. For example, if the two human detection frames Fa and Fb' are shifted in the left-right direction (X direction), change the X coordinate and left-right angle φ of the imaging camera 1, and if the human detection frames are shifted in the vertical direction, changes the height Z and vertical angle θ of the camera. The parameters changed by the calibration section 34 are updated by writing them into a file in the parameter storage section 35 one by one. When the file is updated, the coordinate conversion unit 33 reads out new parameters and performs camera viewpoint conversion of the human detection frame Fb. With this operation, the human detection frame Fb' with new parameters is output.

This parameter update is repeated and fed back to correct the human detection frame Fb'. During this time, as the person walks around within the measurement space, the deviation of the person detection frame due to the difference in location is gradually absorbed. When the shift of the human detection frame can no longer be detected (or when it becomes less than the threshold), the automatic adjustment is ended.

Furthermore, in the calibration process of this embodiment, detection information is exchanged between the person detection section 31 and the person detection section 32 as follows. This improves the accuracy of calibration while complementing the shortcomings of the imaging camera 1 and the TOF sensor 2.

(1) The aspect ratio of the person may differ between the person detection frame Fa obtained from the imaging camera 1 (camera image) and the person detection frame Fb obtained from the TOF sensor 2 (TOF image). be. In this case, the angle of view (FOV=field of View). If you change this at the beginning of automatic adjustment, you will be able to adjust the positioning using either the vertical or horizontal direction of the human detection frame.

(2) When detecting a person on the TOF sensor 2 (TOF image) side, objects resembling a person (for example, a vertically long cart or shelf) may be mistakenly detected as a person. Therefore, by using the face detection function of the person detection unit 31 on the imaging camera 1 side (which can be performed simultaneously with human detection), if a face is not detected in the camera image, the person detection information on the TOF sensor 2 side is used. Delete (cancel). This makes it possible to avoid erroneous calibration processing based on erroneous detection of a person.

FIG. 3 is a diagram illustrating the principle of distance measurement by the TOF sensor 2. The TOF sensor 2 includes a light emitting unit 21 that emits pulsed infrared light from a light source such as a laser diode (LD) or a light emitting diode (LED), and a CCD sensor or CMOS sensor that receives the pulsed light reflected from an object 9. a light receiving section 22 for controlling the light emitting section 21, a light emission control section 23 for controlling the lighting/extinguishing of the light emitting section 21 and the amount of light emitted, a distance calculation section 24 for calculating the distance to the object 9 from the detection signal (light reception data) of the light receiving section 22, and a distance It includes a distance image generation unit 25 that generates a two-dimensional distance image (TOF image) in which values are displayed in different colors. The generated distance image is transmitted to the cooperative processing device 3.

In distance measurement, the light emitting section 21 emits irradiation light 26 toward the object 9 (for example, a person), and the light receiving section 22 receives the reflected light 27 reflected by the object 9 with the two-dimensional sensor 22a. If t is the time difference between when the light emitting unit 21 emits the irradiated light 26 and when the light receiving unit 22 receives the reflected light 27, then the distance D to the object 9 is D=c×t/2 (c is the speed of light ). The two-dimensional sensor 22a is a two-dimensional array of a plurality of pixels such as a CCD sensor, and the distance calculation unit 24 calculates two-dimensional distance data from the received light data at each pixel. Note that in practical distance measurement, an irradiation pulse of a predetermined width is emitted instead of the time difference t, and this is received by the two-dimensional sensor 22a while shifting the timing of the exposure gate. Then, the distance D is calculated from the values of the amount of light received at different timings.

FIG. 4 is a diagram showing an example of how the imaging camera 1 and the TOF sensor 2 are installed. An imaging camera 1 and a TOF sensor 2 are installed so as to cover a space in which a person 9 to be measured moves. The floor surface of the measurement space is defined as the XY direction, and the height direction is defined as the Z direction. As installation parameters, the installation coordinates of the imaging camera 1 and the TOF sensor 2 are (X1, Y1) and (X2, Y2), respectively, and the installation heights are Z1 and Z2, respectively. Further, regarding the installation direction (measurement direction), the left and right angles are φ1 and φ2, respectively, and the vertical angles are θ1 and θ2, respectively. Furthermore, the angle of view (FOV) of the imaging camera 1 is set. These parameters are stored as initial values in the parameter storage section 35. In the calibration process, the human detection frames Fa and Fb obtained from the imaging camera 1 and the TOF sensor 2 are corrected to optimal values so that they match.

Further, in this example, the imaging camera and the TOF sensor 2 are installed on the ceiling of the measurement space so as to face each other with the person 9 in between. The distance measuring system of this embodiment is characterized in that the positional relationship between the imaging camera 1 and the TOF sensor 2 is not subject to any restrictions.

FIG. 5 is a diagram showing creation of a human detection frame from a camera image and a TOF image.
(a) is a sketch of the measurement space, showing the positional relationship between the imaging camera 1, the TOF sensor 2, and the person 9. It is assumed that the person 9 moves in front of the desk 8 in the direction of the arrow.

(b) is a distance image (TOF image) acquired by the TOF sensor 2. In the distance image, the distance to the subject is displayed in different colors, and the person detection unit 32 can extract an area shaped like the person 9 in the same color. Then, a rectangular area surrounding the person is created, and this will be called a "person detection frame" Fb. The human detection frame Fb also moves as the person 9 moves, and flow line data can be generated by finding the trajectory. However, since it is not possible to identify a person's face in a TOF image, it is not possible to know what kind of person the person is.

(c) is a camera image acquired by an imaging camera. The person detection unit 31 can identify the face of the person 9 and obtain attribute information of the person by analyzing the camera image. Furthermore, in this embodiment, the person detection unit 31 creates a rectangular area surrounding the person in the camera image, that is, a "person detection frame" Fa. In the figure, the human detection frame Fa is indicated by a broken line. This technique is known as the object detection function (detect Multi Scale) of the Open CV (registered trademark) program, and can extract the whole body of a person using only a camera image. Even if the person is facing sideways or moving around, a detection frame Fa that surrounds the whole person can be created in real time.

In this embodiment, the human detection frame Fb from the TOF image and the human detection frame Fa from the camera image are used to automatically adjust their positions so that they match. The automatic adjustment process is performed by the coordinate conversion section 33 and the calibration section 34.

FIG. 6 is a diagram illustrating coordinate transformation and calibration processing of the human detection frame.
(a) is a TOF image similar to FIG. 5(b), in which a human detection frame Fb is shown. 5(b) is a camera image similar to FIG. 5(c), and the human detection frame Fa is shown.

In order to align the two human detection frames Fa and Fb in this state, the coordinate conversion unit 33 first corrects the deviation in the installation position between the TOF sensor 2 and the imaging camera 1. In other words, the human detection frame Fb obtained from the TOF image is coordinate-transformed into the coordinate system of the camera image. For this purpose, the coordinate transformation section 33 performs coordinate transformation by calculation using the installation parameters of the TOF sensor 2 and the imaging camera 1 stored in the parameter storage section 35. In (b), the human detection frame Fb' after the coordinate transformation is shown superimposed. In this state, the human detection frame Fa and the human detection frame Fb' are out of alignment, so a calibration process is performed.

(c) shows the calibration process. In the calibration, the installation parameters of the imaging camera 1 are changed, and the human detection frame Fb of the TOF image is coordinate-transformed again using the changed parameters, so that the human detection frame Fb' after the conversion is the human detection frame Fa of the camera image. Adjust to match. In the state shown in this figure, the human detection frame Fa and the human detection frame Fb' match, and the calibration has been completed.

In this manner, in this embodiment, calibration is performed by matching the positions of the human detection frames Fa and Fb in the camera image and the TOF image. As a result, compared to the conventional case of matching camera images and TOF images pixel by pixel or matching camera images and human detection frames (TOF images), the processing load is smaller and the processing time is shorter. Calibration can be automatically and easily performed.

Furthermore, in the calibration method of this embodiment, it is sufficient to extract a human detection frame from the camera image and the TOF image, and there is no need to match the measurement direction (photographing direction) for the person. Therefore, the positional relationship between the imaging camera 1 and the TOF sensor 2 can be set such that they face each other with the person 9 in between, as shown in FIG. be.

FIG. 7 is a diagram showing a specific example of parameter changes during calibration. That is, deviations between the human detection frames Fa and Fb' occur in the horizontal and vertical directions within the camera image, and the sizes and aspect ratios of the human detection frames may not match. The reason why the human detection frames Fa and Fb' deviate is because the initial setting values of the parameters deviate from the actual true values, and the deviation can be resolved by changing the setting values of the parameters. The following are specific types of parameters to be changed:

(a) shows a case where the human detection frames Fa and Fb' are shifted in the left-right direction, and in this case, the coordinates X1 of the imaging camera 1 are changed. (b) is a case where the size of the human detection frame is different (deviation in the depth direction), and the coordinate Y1 of the camera is changed. (c) is a case where the human detection frame is shifted in the left-right direction, and the left-right angle φ1 of the camera is changed. (d) is a case where the human detection frame is shifted in the vertical direction, and the height Z1 of the camera is changed. (e) is a case where the human detection frame is shifted in the vertical direction, and the vertical angle θ1 of the camera is changed. (f) is a case where the aspect ratios of the human detection frames are different, and the viewing angle FOV of the camera is changed.

Note that the cause of the shift is not necessarily single; for example, the shift in the left-right direction is caused by a combination of factors such as the coordinate X in (a) and the left-right angle φ1 in (c). Therefore, other related parameters are also changed.

Furthermore, since the TOF sensor 2 can accurately measure the height from the ground and the distance from a wall, etc., the TOF sensor 2 alone should be calibrated first. After that, by changing the parameters on the imaging camera 1 side, the position alignment can be efficiently calibrated for the deviation of the human detection frames Fa and Fb'.

FIGS. 8A and 8B are flowcharts showing the procedure (first half and second half) of the calibration process. It is assumed that the installation parameters (initial values) of the imaging camera 1 and the TOF sensor 2 are stored in the parameter storage section 35, and that the adjustment of the TOF sensor 2 alone has been completed.

FIG. 8A shows adjustment of the angle of view (FOV) of the imaging camera 1 performed in the first half process. Calibration starts with a person standing in the center of the screen of the imaging camera 1 and the TOF sensor 2, and with the person's face facing toward the imaging camera 1. Then, when a person is detected by both the imaging camera 1 and the TOF sensor 2, automatic parameter adjustment is started. First, the angle of view FOV of the imaging camera 1 is adjusted to match the aspect ratio of the human detection frame. At this time, if a person's face is not detected in the camera image, even if the TOF sensor 2 detects a person, it is a false positive and is canceled. The steps will be explained below.

S101: Start up the imaging camera 1 and the TOF sensor 2, and start a person detection operation.
S102: Based on the camera image of the imaging camera 1, it is determined whether the person detection unit 31 has detected a person (here, an object resembling a person). If detected, the process advances to S103; if not detected, the process waits until a person is detected.
S103: Based on the TOF image of the TOF sensor 2, it is determined whether the person detection unit 32 has detected a person (here, an object resembling a person). If detected, the process advances to S104; if not detected, the process waits until a person is detected.

S104: Based on the camera image of the imaging camera 1, it is determined whether the person detection unit 31 has also detected a person's face. If no face has been detected, the process advances to S105, and if a face has also been detected, the process advances to S106.
S105: Since the object detected by the TOF sensor 2 is not a person, the detection result is canceled and the process returns to S103. This is because the TOF sensor 2 may sometimes mistakenly detect objects that resemble a person (for example, a cart or a shelf) as a person, and this is to eliminate this. Then, the process is repeated until the person detected by the TOF sensor 2 becomes a real person.

S106: The person detection units 31 and 32 create person detection frames Fa and Fb from the camera image and the TOF image, respectively. The person detection frame is a rectangle that includes the entire body of the person.
S107: The coordinate conversion unit 33 performs coordinate conversion of the human detection frame Fb of the TOF image to the camera viewpoint. In this conversion, calculations are performed using the installation parameters of the imaging camera 1 and the TOF sensor 2 stored in the parameter storage section 35. Let Fb' be the human detection frame after conversion.

S108: The calibration unit 34 determines whether the aspect ratios of the human detection frames Fa and Fb' are equal. The aspect ratio here is the ratio of a person's height to width. If the aspect ratios are not equal, the process advances to S109, and if the aspect ratios are equal, the process advances to S110.
S109: The calibration unit 34 changes the angle of view (FOV) of the imaging camera 1, corrects the aspect ratio of the human detection frame Fa, and returns to S108. Then, the angle of view (FOV) of the imaging camera 1 is adjusted so that the aspect ratios of the human detection frames Fa and Fb' are equal.
S110: The adjusted angle of view (FOV) of the imaging camera 1 is stored in the parameter storage unit 35. After this, the process continues to the flowchart of FIG. 8B.

FIG. 8B shows adjustment of parameters other than the angle of view of the imaging camera 1 performed in the latter half of the process. Here, while the person freely moves back and forth and left and right in the measurement space, the person detection frame Fa in the camera image and the person detection frames Fa and Fb' in the TOF image are updated and adjusted so that there is no deviation between them. .

Specifically, the related parameters (X1, Y1, Z1, φ1, θ1) of the imaging camera 1 are changed in the + direction and - direction by a predetermined unit amount according to the mode of displacement of the human detection frame. . Then, the parameter value that reduces the deviation of the human detection frame is selected. This operation is repeated, and when the deviation becomes less than the threshold value, the automatic adjustment is ended. The steps will be explained below.

S111: When the person moves a predetermined distance (for example, 50 cm), this is used as a trigger to proceed to the following process.
S112: From the camera image and the TOF image, the person detection units 31 and 32 create and update person detection frames Fa and Fb, respectively.
S113: The coordinate conversion unit 33 performs coordinate conversion of the human detection frame Fb of the TOF image to the camera viewpoint. In this conversion, the latest installation parameters of the imaging camera 1 and the TOF sensor 2 stored in the parameter storage section 35 are used. Let Fb' be the human detection frame after conversion.

S114: The calibration unit 34 determines whether the deviation between the human detection frames Fa and Fb' is less than a threshold value. The threshold value is, for example, 5% of the detection frame size. If it is less than the threshold (determination is Yes), the process proceeds to S121, and if it is greater than or equal to the threshold (determination is No), the process proceeds to S115.
S115: Determine whether the human detection frame is shifted in the left-right direction. If the determination is Yes, the process advances to S116, and if the determination is No, the process advances to S117.
S116: The calibration unit 34 changes the left/right angle (φ1) parameter of the imaging camera 1 by a ± unit amount (for example, +1 deg, −1 deg), and selects the one with the smaller deviation of the human detection frame at that time. Further, the parameters of the camera coordinates (X1) are changed by a ± unit amount (for example, +100 mm, -100 mm), and the one with the smaller deviation of the human detection frame at that time is selected. After that, the process returns to S111.

S117: Determine whether the human detection frame is shifted in the vertical direction. If the determination is Yes, the process advances to S118, and if the determination is No, the process advances to S119.
S118: The calibration unit 34 changes the parameter of the vertical angle (θ1) of the camera by a ± unit amount (for example, +1 deg, −1 deg), and selects the one with the smaller deviation of the human detection frame at that time. Further, the camera height (Z1) parameter is changed by a ± unit amount (for example, +50 mm, -50 mm), and the one with the smaller deviation of the human detection frame at that time is selected. After that, the process returns to S111.

S119: Determine whether the sizes of the human detection frames are different. If the determination is Yes, the process returns to S120, and if the determination is No, the process returns to S111.
S120: The calibration unit 34 changes the parameter of the camera coordinate (Y1) by a ± unit amount (for example, +100 mm, -100 mm), and selects the one with the smaller difference in the size of the human detection frame at that time. Further, the camera height (Z1) parameter is changed by a ± unit amount (for example, +50 mm, -50 mm), and the one with the smaller difference in the size of the human detection frame at that time is selected. After that, the process returns to S111.

S121: If it is determined in S114 that the deviation between the human detection frames Fa and Fb' is less than the threshold value, all the parameters selected above (X1, Y1, Z1, φ1, θ1) are stored in the parameter storage unit 35. This completes the calibration. After this, actual measurements are performed using the distance measuring system.

According to the above flowchart, the calibration process can be performed automatically and easily by using the human detection frame extracted from the camera image and the TOF image. At this time, by using the face detection results from the camera image, it is possible to eliminate false human detection in the TOF image and improve the reliability of calibration. Furthermore, by matching the aspect ratios of the human detection frames, the angle of view (FOV) of the camera can be adjusted efficiently.

FIG. 9 is a diagram showing an example of measurement by the distance measuring system of this embodiment. Here, we will show a case where customers in a store are detected and their flow lines and attribute information are acquired and displayed. Two imaging cameras 1a and 1b and two TOF sensors 2a and 2b are installed in the store, and calibration between the imaging cameras and the TOF sensors has been completed in advance. Note that calibration is similarly completed between the two imaging cameras 1a and 1b and between the two TOF sensors 2a and 2b using the human detection frame.In this example, at the current time 10:11, Two customers P1 and P2 are staying in the store, and their movement trajectories within the store are displayed as flow lines L1 and L2. This flow line is created from distance images acquired by the TOF sensors 2a and 2b. On the other hand, customer attribute information (age, gender, repeat customers, etc.) is determined from the facial images of customers P1 and P2 acquired by imaging cameras 1a and 1b. The information from both is linked using time stamps, and attribute information is displayed according to the customer's current location. For example, information such as that customer P1 is a female (25 years old) and a repeat customer, entered the store at 10:10, and stopped by the display shelf (No. 4) is displayed.

In this embodiment, the customer is detected by the imaging cameras 1a, 1b and the TOF sensors 2a, 2b, but as a result of the calibration process, there is little deviation in the detection position, so the attribute information acquired at the time of entering the store will be used when the customer moves. The display will be displayed accurately according to the customer's current location. Therefore, even in a situation where a large number of customers are staying in the store, the displayed information is not confused with information of other customers, and the reliability of the displayed information is improved.

1, 1a, 1b: imaging camera,
2, 2a, 2b: Distance sensor (TOF sensor),
3: Cooperation processing device,
4: Switching hub,
9: Object (person),
21: Light emitting part,
22: Light receiving section,
23: Light emission control unit,
24: distance calculation section,
25: Distance image generation unit,
31, 32: person detection unit,
33: Coordinate transformation section,
34: Calibration section,
35: Parameter storage section,
36: Image synthesis section,
37: Image display section,
100: Ranging system,
Fa, Fb, Fb': human detection frame,
L1, L2: Flow line.

Claims (10)

In a distance measurement system that installs a distance measurement sensor and an imaging camera to generate a distance image of an object,
The distance sensor is a TOF (time of flight) sensor that measures the distance to the object based on the transmission time of light,
A TOF image obtained by the TOF sensor and a camera image obtained by the imaging camera are aligned, and attribute information of the object obtained from the camera image is superimposed on a movement trajectory of the object obtained from the TOF image. Equipped with a cooperative processing device that displays
The cooperative processing device includes:
a first person detection unit that detects a person from the camera image and creates a rectangular person detection frame Fa surrounding the person;
a second person detection unit that detects a person from the TOF image and creates a rectangular person detection frame Fb surrounding the person;
a coordinate conversion unit that converts the TOF image and the human detection frame Fb into a coordinate system of the camera image using installation parameters of the TOF sensor and the imaging camera;
In order to align the TOF image and the camera image, the human detection frame (referred to as Fb' ) converted by the coordinate conversion unit is aligned with the human detection frame F a on a common screen. a calibration unit that calibrates the installation parameters of the TOF sensor and the imaging camera by changing the installation parameters of the imaging camera ;
Equipped with
In order to match the human detection frame Fb' with the human detection frame Fa on the common screen, the calibration unit is configured to adjust the position of the human detection frame Fb' to be common to the position of the human detection frame Fa. A ranging system characterized in that the installation parameters of the imaging camera are changed so that they match on the screen . The ranging system according to claim 1 ,
The first person detection unit and the second person detection unit update the person detection frame Fa and the person detection frame Fb every time the position of the person to be detected moves by a predetermined distance,
The coordinate conversion unit converts the human detection frame Fb into the human detection frame Fb' using the changed installation parameters of the imaging camera,
The distance measuring system is characterized in that the calibration unit repeatedly changes installation parameters of the imaging camera based on the updated human detection frame Fa and the updated human detection frame Fb'. The ranging system according to claim 1 ,
When the aspect ratios of the human detection frame Fa and the human detection frame Fb' are different, the calibration unit matches the human detection frame Fb' with the human detection frame Fa on the common screen. In order to achieve this, the distance measuring system is characterized in that parameters of the angle of view of the imaging camera are changed so that the aspect ratio of the human detection frame Fa is matched with the aspect ratio of the human detection frame Fb'. The ranging system according to claim 1 ,
The first person detection unit has a function of detecting a person's face from the camera image,
If a person's face cannot be detected in the person detection frame Fa created by the first person detection unit, the person detection frame Fb created by the second person detection unit is canceled. distance system. In a method for calibrating the coordinates of the ranging sensor and the imaging camera when the ranging sensor and the imaging camera are installed to generate a distance image of a target,
The distance sensor is a TOF (time of flight) sensor that measures the distance to the object based on the transmission time of light,
acquiring a TOF image with the TOF sensor and acquiring a camera image with the imaging camera;
detecting a person from the TOF image and creating a person detection frame Fb surrounding the person; detecting the person from the camera image and creating a person detection frame Fa surrounding the person;
converting the human detection frame Fb into a coordinate system of the camera image using installation parameters of the TOF sensor and the imaging camera ;
By changing the installation parameters of the imaging camera so that the human detection frame (referred to as Fb') converted to the coordinate system of the camera image and the human detection frame Fa match on a common screen, calibrating installation parameters of the TOF sensor and the imaging camera;
Equipped with
In the calibration step , in order to match the human detection frame Fb' with the human detection frame Fa on the common screen, the position of the human detection frame Fb ' is changed to the position of the human detection frame Fa. A coordinate calibration method characterized by changing installation parameters of the imaging camera so that they match on a common screen . The coordinate calibration method according to claim 5 ,
The step of creating the human detection frame Fb and the human detection frame Fa is executed and updated every time the position of the person to be detected moves by a predetermined distance,
In the step of converting the human detection frame Fb to the coordinate system of the camera image, converting the human detection frame Fb into the human detection frame Fb' using the changed installation parameters of the imaging camera,
A coordinate calibration method characterized in that, in the calibration step, installation parameters of the imaging camera are repeatedly changed based on the updated human detection frame Fa and the updated human detection frame Fb'. The coordinate calibration method according to claim 5 ,
If the aspect ratios of the human detection frame Fa and the human detection frame Fb' are different, in the calibration step, the human detection frame Fb' is matched with the human detection frame Fa on the common screen. A coordinate calibration method comprising : changing a field angle parameter of the imaging camera so that the aspect ratio of the human detection frame Fa matches the aspect ratio of the human detection frame Fb'. The coordinate calibration method according to claim 5 ,
In the step of creating the person detection frame Fa from the camera image, detecting a person from the camera image and detecting the face of the person,
If a person's face cannot be detected in the human detection frame Fa created from the camera image, in the calibration step, the human detection frame Fb created from the TOF image is canceled. tion method. In a distance measurement system that installs a distance measurement sensor and an imaging camera to generate a distance image of an object,
The distance sensor is a TOF (time of flight) sensor that measures the distance to the object based on the transmission time of light,
The TOF image obtained by the TOF sensor and the camera image obtained by the imaging camera are aligned, and attribute information of the object obtained from the camera image is superimposed on the movement trajectory of the object obtained from the TOF image. Equipped with a cooperative processing device that displays
The cooperative processing device includes:
a first person detection unit that detects a person from the camera image and creates a person detection frame Fa surrounding the person;
a second person detection unit that detects a person from the TOF image and creates a person detection frame Fb surrounding the person;
a coordinate conversion unit that converts the TOF image and the human detection frame Fb into a coordinate system of the camera image using installation parameters of the TOF sensor and the imaging camera;
In order to align the TOF image and the camera image, the human detection frame (referred to as Fb') converted by the coordinate conversion unit is aligned with the human detection frame Fa on a common screen. , a calibration unit that calibrates installation parameters of the TOF sensor and the imaging camera by changing installation parameters of the imaging camera;
A ranging system characterized by comprising: In a method for calibrating the coordinates of the ranging sensor and the imaging camera when the ranging sensor and the imaging camera are installed to generate a distance image of a target,
The distance sensor is a TOF (time of flight) sensor that measures the distance to the object based on the transmission time of light,
acquiring a TOF image with the TOF sensor and acquiring a camera image with the imaging camera;
detecting a person from the TOF image and creating a person detection frame Fb surrounding the person; detecting the person from the camera image and creating a person detection frame Fa surrounding the person;
converting the human detection frame Fb into a coordinate system of the camera image using installation parameters of the TOF sensor and the imaging camera;
By changing the installation parameters of the imaging camera so that the human detection frame (referred to as Fb') converted to the coordinate system of the camera image and the human detection frame Fa match on a common screen, calibrating installation parameters of the TOF sensor and the imaging camera;
A coordinate calibration method comprising:

Images