JP7194015B2 - Sensor system and distance measurement method - Google Patents

Description

The present invention relates to a sensor system having a plurality of ranging sensors and a distance measuring method using the same.

2. Description of the Related Art A distance measuring sensor is known that measures a distance to an object based on light transmission time (hereinafter referred to as TOF method: time of flight) and acquires a distance image having distance information for each pixel. For example, a distance image camera disclosed in Patent Document 1 includes a plurality of camera units, and aims to obtain a distance image having a wider angle of view than that of a single imaging unit and high distance accuracy. A two-dimensional position correction unit for correcting the two-dimensional position information of each pixel based on the average distance information obtained by the distance information replacement unit and the two-dimensional pixel position of each pixel of each distance image; a distance image for obtaining a synthesized distance image obtained by synthesizing the distance images by converting the two-dimensional position information and the distance information of each pixel corrected by the two-dimensional position correction unit into a common three-dimensional coordinate system; It is disclosed that it has a synthetic part.

JP 2012-247226 A

In Japanese Patent Laid-Open No. 2004-100000, when the distance images of each camera unit are coordinate-transformed and synthesized, "each distance image is generated according to the camera parameters (internal and external) obtained by calibration at the time of installation of each camera unit 10. Each distance image is synthesized by coordinate-transforming the X-value, Y-value, and Z-value of the pixel into the camera coordinate system or the world coordinate system."

As a general method for this calibration (correction), a specific object (marker) is placed in the measurement space, the position of the marker is measured by each camera unit, and coordinate conversion is performed so that common coordinate values are obtained. It has been known. However, in reality, it may be difficult to properly place the markers. That is, since the measurement area covered by each camera unit is originally different (divided), the marker must be placed across each area, and the distance image obtained according to the installation of the camera unit can be confirmed. It becomes a troublesome and repetitive work of changing the placement of the marker or adjusting the installation of the camera unit. Alternatively, in order to avoid this, it is conceivable to increase the size of the marker or use a marker with a special shape. .

In view of the above, an object of the present invention is to provide a sensor system having a plurality of ranging sensors, which facilitates adjustment of installation of each ranging sensor and correction of distance data obtained from each ranging sensor. and to provide a distance measuring method.

In order to solve the above problems, the present invention provides a sensor system in which a plurality of sensors are installed to measure the distance to a detection object within a detection target area and generate a distance image, and the distance images from the plurality of sensors are synthesized. In order to do so, a sensor cooperation processing device that performs coordinate matching processing between sensors and a display device that displays the generated distance image are provided. In order to perform coordinate matching processing between sensors, the sensor cooperation processing device acquires trajectory data of a moving object moving within a detection target area using a plurality of sensors, and the trajectory of the moving object detected by each sensor is continuous or coincides. Transform the detection coordinates of each sensor as follows.

The present invention also provides a distance measurement method for installing a plurality of sensors and measuring the distance to a detection target within a detection target area to generate a range image, wherein the distance measurement method moves within the detection target area using the plurality of sensors. a step of acquiring trajectory data of a body; a step of converting coordinates detected by each sensor so that the trajectory of a moving object detected by each sensor is continuous or coincident to match the coordinates of the sensors; and synthesizing and displaying range images from a plurality of sensors in the state.

According to the present invention, since the movement trajectory of the worker is used for the calibration of a plurality of distance measuring sensors, there is no need to arrange dedicated markers for calibration, and the installation of the distance measuring sensors can be easily adjusted. can be done

1 is an external view of a sensor system 1 according to Example 1. FIG. 2 is a block diagram showing the configuration of the sensor system 1; FIG. FIG. 2 is a block diagram showing the internal configuration of sensors 11a and 11b; FIG. 2 is a block diagram showing the internal configuration of the sensor cooperation processing device 12; 4 is an overall flowchart showing an overview of calibration processing; Flowchart of floor surface detection processing (S200). A detailed flowchart of a plane detection process (S210). 4 is a detailed flowchart of plane detection processing (S215) for each voxel region; A detailed flowchart of a moving object trajectory detection process (S220). A detailed flowchart of a parallel determination process (S230). Flowchart of trajectory connection processing (S300). A detailed flowchart of a moving object trajectory detection process (S330). Detailed flowchart of coordinate conversion processing (S340). The figure which shows the method of displaying the imaging|photography space of one sensor. FIG. 10 is a diagram showing a photographing space in an installation state that does not require rotation conversion; FIG. 8B is a diagram showing a trajectory of a worker in the installation state of FIG. 8A; FIG. 10 is a diagram showing a photographing space in an installation state that requires rotation conversion; FIG. 9B is a diagram showing a trajectory of a worker in the installation state of FIG. 9A; FIG. 9B is a diagram showing a state after the trajectory of FIG. 9B is corrected by coordinate transformation; FIG. 4 is a diagram showing an example of an installation state that cannot be corrected; A screen that warns that there is a defect in the installation state (angle) of the sensor. A screen showing information 91 on the content of defects and correction information 92 . Screen showing correction results. A screen indicating that correction is not possible. A screen that warns that there is a problem with the installation state (height) of the sensor. A screen showing information 91 on the content of defects and correction information 92 . Screen showing correction results. A screen indicating that correction is not possible. FIG. 10 is a flowchart of floor surface detection processing (S400) of the second embodiment; FIG. A detailed flowchart of a parallel determination process (S420). FIG. 11 is an external view of a sensor system 1' according to Example 3; 2 is a block diagram showing the configuration of a sensor system 1'; FIG. Flowchart of locus superimposition processing (S500). Detailed flowchart of coordinate conversion processing (S540). FIG. 10 is a diagram showing a photographing space in an installation state that does not require rotation conversion; FIG. 15B is a diagram showing the trajectory of a worker in the installation state of FIG. 15A; FIG. 10 is a diagram showing a photographing space in an installation state that requires rotation conversion; FIG. 16B is a diagram showing the trajectory of the worker in the installation state of FIG. 16A; FIG. 16B is a diagram showing a state after the trajectory of FIG. 16B is corrected by coordinate transformation;

An embodiment of the present invention will be described below with reference to the drawings. Also, the ranging sensor is simply called a "sensor".

In the first embodiment, a case will be described in which a plurality of sensors are installed on the ceiling of a room to measure the range of a person in the room. Although a configuration using two sensors will be described below, the same applies to the case of two or more sensors.

FIG. 1A is an external view of a sensor system 1 according to Example 1. FIG. Two sensors 11a and 11b are installed on the ceiling 5 to photograph a detection target object (for example, the person 3) on the floor surface 4 and generate a distance image. The sensor positions and imaging directions are adjusted so that the imaging regions 2a and 2b by the respective sensors are adjacent to each other. Each sensor 11 a and 11 b is connected to the sensor cooperation processing device 12 and the display device 10 via the network 13 . The display device 10 displays a distance image of the person 3 present in the room. For example, by applying this system to a store, it is possible to grasp the movements of customers in the store. In Example 1, a plurality of sensors are installed at different positions so as to cover a large room. Therefore, in order to match the coordinate systems of the sensors, coordinate conversion processing is performed during the sensor installation work.

FIG. 1B is a block diagram showing the configuration of the sensor system 1. As shown in FIG. The sensor system 1 includes two sensors 11a and 11b, a sensor cooperation processing device 12 that performs processing in cooperation with the two sensors, and a network 13 that performs communication between the two sensors and the sensor cooperation processing device 12. and a display device 10 that visualizes and displays the output from the sensor cooperation processing device 12 and the execution result.

The sensors 11a and 11b emit irradiation light to the measurement space, receive reflected light from the measurement space, measure the distance to each object in the space from the delay time, and generate a distance image.
The sensor cooperation processing device 12 performs coordinate matching processing (calibration) between the sensors in order to combine the distance images of the two sensors 11a and 11b. For coordinate matching processing, floor detection processing and trajectory connection processing are performed. In the floor surface detection process, the two sensors 11a and 11b measure the coordinates (height) of the floor surface of the imaging regions 2a and 2b, respectively, and match the values. In the trajectory linking process, a moving object (for example, the worker 3) moves on the floor surface 4, the trajectories are measured by the sensors 11a and 11b, and coordinate transformation is performed so that the trajectories in the photographing areas 2a and 2b are continuously linked. .

FIG. 2 is a block diagram showing the internal configuration of the sensors 11a and 11b. Sensors 11a and 11b measure the distance to the object by the TOF method. The light emitting unit 21 emits irradiation light to the measurement space, and the light receiving unit 22 receives reflected light from the measurement space. The light receiving unit 22 has a plurality of pixel sensors arranged two-dimensionally, and generates two-dimensional information about the delay time and the intensity of the reflected light. The distance image generation unit 23 measures the distance to the object in the measurement space from the delay time from the light emission timing of the light emission unit 21 to the light reception timing of the light reception unit 22, and two-dimensionally displays the distance value by color-coding, for example. generate a range image. The luminance image generator 24 generates a luminance image from the intensity of the light reflected by the light receiver 22 .

The communication unit 25 communicates the distance image, luminance image, and control information with the sensor cooperation processing device 12 via the network 13 . The control unit 26 controls each unit such as the light emitting unit 21 and the light receiving unit 22 according to the control signal from the sensor cooperation processing device 12 received by the communication unit 25 . Buffer memory 27 is used to generate the range image, and buffer memory 28 is used to generate the luminance image. The two buffer memories 27 and 28 may share one memory by allocating addresses separately.

FIG. 3 is a block diagram showing the internal configuration of the sensor cooperation processing device 12. As shown in FIG. The communication unit 31 transmits execution mode instructions to the sensors 11a and 11b via the network 13, and receives measurement data from the sensors 11a and 11b. The buffer memory 32 is used as a temporary storage area for data to be transmitted and received.

The arithmetic processing unit 35 accumulates the distance data from the sensors 11a and 11b received by the communication unit 31 in the buffer memory 36, and based on the accumulated distance data, performs coordinate matching processing including floor surface detection processing and trajectory connection processing. I do. Also, the distance image and the luminance image generated by the sensors 11a and 11b are synthesized. The calculation support unit 33 improves performance by executing a part of the processing of the calculation processing unit 35 . Data transfer such as data input from the calculation processing unit 35 to the calculation support unit 33 and calculation result output from the calculation support unit 33 to the calculation processing unit 35 are performed via the shared memory 34 .

The display output unit 37 outputs the results of processing by the arithmetic processing unit 35 (coordinate conversion processing results and synthesized distance images) and causes the display device 10 to display them. The buffer memory 38 temporarily accumulates display data. The buffer memories 32, 36, 38 and the shared memory 34 may be shared by allocating one memory address separately.

Next, a calibration operation (coordinate alignment processing) of a plurality of sensors in the sensor system 1 will be described in detail.
FIG. 4 is an overall flowchart showing an overview of calibration processing. The operation of the installation operator (S101-S106) and the process of the sensor system 1 (S111-S120) are shown in correspondence. In addition, the number of a later-described drawing showing the details of each process is described.

When the operator installs a plurality of sensors and activates the processing operation (S101), the sensor system 1 starts floor surface detection processing. Each sensor acquires distance data (point cloud data) from the imaging space (S111), and detects planes (planes) distributed in a plane within the imaging space (S112). When the plane is detected, a movement start notification is sent from the sensor system (S113), the operator (moving object) moves on the floor surface in the imaging space (S102), and each sensor collects distance data (point cloud data) is obtained (S114). The sensor system detects the trajectory of the moving object from the point cloud data (S115), determines the parallelism between the plane detected in S112 and the plane formed by the trajectory of the moving object (S116), and determines the plane formed by the trajectory of the moving object. The plane most parallel to is detected as the floor surface. When data acquisition (floor surface detection) is completed for all sensors (S117), the worker ends movement (S103).

Next, the sensor system 1 proceeds to trajectory connection processing. From the point cloud data of the moving object acquired in S114, the trajectory spanning the adjacent imaging spaces 2a and 2b is acquired (S118). Then, the coordinates of each sensor are transformed so that the trajectory of the moving object is continuous at the connecting portion of the adjacent photographing spaces 2a and 2b (S119). The operator is notified of the connection result (S120). The content of notification indicates whether the initial installation state of the sensor is normal or defective (S104), and if defective, the content of coordinate conversion (correction) or the reason why correction is impossible (S105). The connection state after correction is indicated (S106), and if the worker approves, the calibration operation (installation work) is completed. If the correction is impossible or the operator does not approve, the process returns to S101 and the sensor is installed again.

Acquisition of the distance data of the plane in S111 and acquisition of the distance data of the moving object in S114 can be performed at the same time. In other words, the background and the foreground can be acquired when extracting a moving object from the shooting space, and processing can be performed at a higher speed than if they are acquired separately. Also, for the locus acquisition in S118, by using the point cloud data of the moving body acquired in S114, a series of walking motions of the moving body (worker) can be completed only once.

In the floor surface detection processing of S111 to S116, the distance data in the shooting space and the distance data of the moving object (worker) moving on the floor surface are acquired by each installed sensor, and the plane parallel to the plane formed by the trajectory of the moving object is obtained. is determined to be the floor surface. Then, from the result of the floor surface detection processing, it is possible to acquire the self-position such as the installation height and orientation of each sensor from the floor surface, and coordinate the installation height and imaging direction (elevation/depression angle) of each sensor.

In the trajectory linking process of S118 and S119, the worker as a mobile body walks all over the measurement target area, and each sensor captures a series of walking motions at least once or more. Each sensor acquires trajectory data in which the coordinate data of the walking worker is arranged in time series from the photographed distance data. Then, the head and tail of the trajectory data, that is, the vector of the trajectory when the worker enters the imaging range of the sensor and the vector of the trajectory when the worker exits the imaging range are calculated. Coordinate transformation of each sensor (transformation from local coordinates to world coordinates) is performed so that the vector at the time of appearance and the vector at the time of disappearance are continuously connected between the photographing spaces of adjacent sensors. As a result, the entire measurement target area covered by the sensor system can be grasped in one coordinate system (world coordinates).

As described above, in this embodiment, since the moving locus of the moving body (worker) is used for the calibration operation, there is no need to arrange a dedicated marker as in the conventional art, and the sensor installation work can be easily performed. be able to.

Individual processing will be described in detail below.
FIG. 5A is a flowchart of floor surface detection processing (S200). In this flow, the sensors 11a and 11b acquire the distance data in the shooting space and the distance data of the moving object moving on the floor surface upon receiving an instruction for floor surface detection processing from the sensor cooperation processing device 12. FIG. A plane parallel to the plane formed by the trajectory of the moving object is determined as the floor surface.

In S210, the sensors 11a and 11b acquire distance data in the imaging space, and detect surfaces (planes) distributed in a plane in the imaging space. Details are shown in FIGS. 5B and 5C.
In S220, the moving body (operator) moves on the floor surface in the imaging space, and each sensor acquires distance data of the moving body and detects the trajectory of the moving body. Details are shown in FIG. 5D.

In S230, the parallelism between the plane detected in S210 and the plane formed by the trajectory of the moving body detected in S220 is determined, and the plane most parallel to the plane formed by the trajectory of the moving body is detected as the floor surface. Details are shown in FIG. 5E.
This flow shows one operation of the floor surface detection process, and noise in the distance data acquired by the sensor can be reduced by combining it with methods such as repeated execution and averaging and low-level signal cuts. It can be removed to improve the accuracy of floor detection.

FIG. 5B is a detailed flowchart of the plane detection process (S210) in FIG. 5A.
At S211, the initial value of the angle of the vertical axis of the plane (that is, the axis parallel to the normal vector of the plane) is obtained. Here, the floor surface is used as a plane to be detected, and the sensor is installed vertically downward on the ceiling at a height away from the floor surface. When the sensor is oriented in the positive direction of the Z-axis as the orientation of the local coordinates of the sensor, the initial value of the vertical axis of the horizontal floor surface is the positive Z-axis direction as the reference orientation, and the difference from the reference orientation is: It can be expressed as X-axis rotation angle=180 degrees and Y-axis rotation angle=0 degrees. If the plane to be detected is not the floor surface but a vertical wall surface, the horizontal axis perpendicular to the wall surface is used as a reference, and detection can be performed by the same processing as the processing described below.

In S212, the distance data of the area including the detection target plane is acquired. At this stage, since it is unknown which area is flat, all the distance data that can be obtained by the sensor is obtained.
In S213, the distance data acquired in S212 is converted into 3D point cloud data in world coordinates. The point cloud data is a data set in which the positions of individual distance data are represented by XYZ coordinates. For example, the distance data that can be acquired by a TOF sensor with a resolution of 640 x 480 pixels is about 300,000 pixels at maximum. A data set in which 300,000 coordinate values are arranged is distance data. Furthermore, the positive direction of the Z-axis of the world coordinates is vertically downward, and the conversion of the coordinate axes of the world coordinates and the coordinate axes of the local coordinates is expressed by the rotation angle of the X-axis and the rotation angle of the Y-axis. conversion is possible. When converting one local coordinate to world coordinates, there is no problem even if the Z-axis rotation is fixed at 0 degrees, but when converting multiple local coordinates to world coordinates, conversion of Z-axis rotation is also necessary. becomes.

In S214, the three-dimensional space to be detected is divided into regions of a certain size. For example, it is divided so that cubes each side of which is 25 cm are adjacent to each other in the XYZ axis direction and fill the three-dimensional space. This cube is called a voxel region.
In S215, plane detection is performed for each point cloud data included in each voxel area. Details of the voxel region plane detection processing will be described with reference to FIG. 5C.
In S216, if the distance between the planes detected in each area detected in S215 is sufficiently small, they are regarded as the same plane.

In S217, it is determined whether plane detection has been performed in all voxel areas, and if plane detection has been performed in all voxel areas, the process ends. If there remains a voxel area that has not yet been plane-detected, the voxel area is subjected to the processing from S215 onwards.

FIG. 5C is a detailed flowchart of plane detection processing (S215) for each voxel region in FIG. 5B. In the plane detection process here, when the point cloud data in the voxel area is projected, the direction of the projection axis where the point cloud is most dense is obtained, and the direction of the projection axis is regarded as the direction of the normal vector of the plane. It performs plane detection.

In S2151, the angle of the projection axis of the voxel area targeted for plane detection is set. For the angle of the projection axis, for example, 72 X-axis rotation angles are set in 5-degree increments, and 72 Y-axis and Z-axis rotation angles are also set. A rough search is performed, and further, a combination of candidate angles among the results of the search is focused on and examined in increments of 1 degree.
In S2152, the point cloud data is projected onto the projection axis, and in S2153, the point cloud distribution on the projection axis is created as a one-dimensional histogram.

In S2154, the maximum point of the histogram created in S2153 is obtained, and in S2155, it is determined whether or not the maximum point has a clear peak. If it is determined that there is a peak (Yes), in S2156, the angle of the projection axis is listed as a candidate for the normal vector of the plane. If it is determined that it does not have a peak (No) in S2155, it is considered that the axis is not perpendicular to the plane, so it is excluded from the candidates and not listed.
In S2157, it is determined whether or not an unexecuted angle combination pattern remains, and if it remains (Yes), the processing is repeated from S2151. If there are no unexecuted angle combination patterns left (No), the process proceeds to S2158.

In S2158, among the directions of the projection axes and the maximum points listed as candidates for the normal vector of the plane by the processing from S2151 to S2157, the angle of the projection axis with the sharpest peak is determined as the direction of the normal vector of the plane. , and the point group forming the coordinates of the maximum is regarded as a point group forming a plane.

FIG. 5D is a detailed flowchart of the moving body trajectory detection process (S220) in FIG. 5A. Here, the trajectory of the mobile object is obtained from the distance data of the mobile object.
In S221, the distance data of the moving object moving on the floor is acquired. If the mobile object is a worker, distance data is acquired by measuring how the robot walks on the floor with two sensors. A moving object (worker) walks all over the area to be measured, and each sensor measures a series of walking motions at least once or more to obtain distance data.

In S222, the point cloud data of the moving object (worker) is extracted from the distance data acquired in S221, and the positions of the moving object are arranged in time series on the XYZ coordinates to acquire the trajectory data of the moving object. Note that point cloud data of a moving object can be extracted by extracting a moving object using a stationary point cloud as a background and a moving point cloud as a foreground.

FIG. 5E is a detailed flowchart of the parallel determination process (S230) in FIG. 5A. Here, the planes detected in FIGS. 5B and 5C are compared with the trajectory of the moving object detected in FIG. 5D, and a plane that satisfies the condition that the trajectory and the plane are parallel is regarded as the floor surface.

In S231, the trajectory of the moving object detected in FIG. 5D is projected onto the vertical axis of the plane detected in FIGS. 5B and 5C. In S232, the distribution of trajectories on the vertical axis in S231 is created as a one-dimensional histogram. At S233, the maximum point of the histogram created at S232 is acquired.

In S234, it is determined whether or not the maximum point acquired in S233 has a sharp peak. If the peak is sharp (Yes), the trajectory and the plane are considered parallel in S235, and the plane is considered as the floor surface. If the peak is not sharp in S234 (No), the processing from S231 to S234 is repeated for other planes.

It should be noted that the parallel determination processing between the locus and the plane in FIG. You can also Furthermore, when walking along a wall, it is possible to distinguish between the wall and the floor based on the vertical movement of the center of gravity and the left-right swaying pattern of the top of the head during movement so that the wall is not mistakenly detected as the floor. That is, since the direction in which the center of gravity moves up and down is the vertical direction, and the direction in which the center of gravity moves left and right is the horizontal direction, it is possible to distinguish between a vertical wall surface and a horizontal floor surface. In addition, when the moving object is a worker, the area where the point cloud does not exist between the plane and the moving object is assumed based on the premise that the worker's feet are on the ground so as not to misdetect the wall surface as the floor surface. By regarding the smaller one as the floor surface, it is possible to distinguish between them.

FIG. 6A is a flow chart of the trajectory linking process (S300). In this flow, an instruction for trajectory connection processing is received from the sensor cooperation processing device 12, and the sensors 11a and 11b acquire the trajectory of the moving object moving on the floor surface. Then, coordinate transformation is performed so that the trajectory of the moving object acquired by the sensors 11a and 11b is continuous.

S310 is a physical sensor installation work, and the operator installs a plurality of sensors 11a and 11b so as to cover the imaging area.
In step S320, using the result of the floor surface detection process (S200) performed in FIG. 5A, the self position such as the installation height and orientation from the floor surface of each sensor is acquired. That is, this makes it possible to calibrate the installation height and orientation (elevation/depression angle) of each sensor.

In S330, moving object trajectory detection that straddles adjacent imaging regions is performed. Details of the processing will be described with reference to FIG. 6B.
In S340, processing for connecting the trajectories obtained in S330, that is, coordinate conversion between sensors is performed. Details of the processing will be described with reference to FIG. 6C.

FIG. 6B is a detailed flowchart of the moving body trajectory detection process (S330) in FIG. 6A. Note that this process can also be used as the moving object trajectory detection process (S220) shown in FIG. 5D.
In S331, the moving body (worker) walks and moves so as to cover the flow line so that each sensor can measure the moving body.

In S332, the distance to the worker is measured by each sensor, moving object extraction is performed from the obtained distance data, and the point group of the worker who is a moving object is obtained.
In S333, a trajectory is obtained from the point cloud of the worker obtained in S332.

FIG. 6C is a detailed flowchart of the coordinate conversion process (S340) in FIG. 6A. The trajectories from each sensor obtained in FIG. 6A are rearranged in time series to obtain parameters of coordinate transformation that most smoothly connect the trajectories.

In S341, the trajectories of the moving bodies (workers) acquired in FIG. 6B (S333) are sorted in the order in which the moving bodies appeared. At this time, a continuous trajectory is counted as one, and as a result of sorting, the trajectory moves back and forth between the imaging spaces of the sensors with different orders. In addition, you may sort by the order which disappeared instead of the order which appeared.

In S342, attention is focused on the adjacent Nth and N+1th trajectories sorted in S341. These are trajectories acquired by different sensors. Then, the N+1th trajectory is coordinate-transformed so that the vector when the Nth trajectory disappears and the vector when the N+1th trajectory appears are continuous by extrapolation. Coordinate transformation includes movement of the coordinate origin and rotation of the coordinate axes.

In S343, it is determined whether or not the N-th trajectory and the N+1-th trajectory are continuous by the coordinate conversion in S342. If the result of determination is that they are not continuous (No), an error warning is issued in S344. In other words, if it is difficult to connect trajectories, or if the time from the disappearance of the Nth sorted trajectory to the appearance of the N+1th trajectory is long and the continuity of the trajectories can be considered low, the trajectory linking process (coordinate conversion process) is performed. Issue an error warning to the effect that it has failed.

If it is determined in S343 that the N-th and N+1-th trajectories are continuous (Yes), it is determined in S345 whether or not there remains a trajectory that has not been subjected to the connection processing. If there is a remaining trajectory, the processing from S341 is repeated, and if there is no remaining trajectory, the process ends.
It should be noted that, even if the determination in S343 indicates that the coordinates have been transformed so that the trajectory is continuous (Yes), the parameters at the time of coordinate transformation may be displayed on the display device 10 .

Next, the coordinate conversion process (S342) will be described in detail, taking a specific installation state of the plurality of sensors as an example. Coordinate transformation is related to the positional relationship of the imaging space of each sensor, so the notation of the imaging space of the sensor will be described.

7A and 7B are diagrams showing how the imaging space of one sensor is displayed. FIG. 7A is a side view of the imaging space viewed from the lateral direction, and FIG. 7B is a plan view of the imaging space viewed from the ceiling. be. The coordinate axes (local coordinates) for the sensor 11 are the forward direction as the X axis, the horizontal direction as the Y axis, the vertical direction as the Z axis, and the floor surface as the XY plane. The sensor 11 is installed so as to obliquely look down on the space.

A ridge indicated by reference numeral 41, which is a combination of a sector and an arc, indicates the angle of view of the sensor 11. FIG. A range of dotted lines indicated by reference numeral 42 indicates a plane on the floor photographed by the sensor 11 . Therefore, the space surrounded by reference numerals 41 and 42 is the imaging space of the sensor 11 (hereinafter referred to as sensor imaging space).

First, as a specific installation state of the two sensors, a case where coordinate rotation conversion is unnecessary in the coordinate conversion processing (S342) will be described. In other words, this is the case where the trajectories of two adjacent sensors are continuous even without rotation conversion processing.
FIG. 8A is a diagram showing an imaging space in an installation state that does not require rotation conversion. (a) is a side view, and (b) is a plan view. The display here is in world coordinates. The XY plane is the floor surface, and the Z-axis direction is the height direction.

The two sensors 11a and 11b are installed at the same height from the ceiling and at the same angle, and are parallelly moved by a predetermined distance Xo in the photographing direction. In other words, if the installation positions of the two sensors 11a and 11b are expressed in world coordinates, the X coordinate is the position shifted by the distance Xo, the Y coordinate is the same position, and the Z coordinate is the same position (ceiling surface). Also, the installation directions (X-axis direction) of the two sensors 11a and 11b match in the world coordinates and the local coordinates.

As a result, each sensor imaging space (the space surrounded by reference numerals 41a and 42a and the space surrounded by reference numerals 41b and 42b) is also formed by parallel movement in the X-axis direction by a distance Xo. Here, by making the X-direction distance Xo between the two sensors 11a and 11b equal to the X-direction length of the imaging planes (reference numerals 42a and 42b) of one sensor, the imaging planes 42a and 42b of the sensors 42b are arranged continuously with their boundaries touching.

On the other hand, a rectangular parallelepiped spatial region indicated by reference numeral 40 (one-dot chain line) indicates a detection target region in which a detection target (person) exists. That is, the range (X-axis and Y-axis directions) of the detection target area 40 is the movement range of the detection target, and the height (Z-axis direction) is, for example, the distance between the fingertips when a person in a standing posture raises his hand. The size includes the height from the floor.

Here, in the state shown in FIG. 8A, the sensor imaging space (area where the angles of view 41a and 41b are overlapped in space) by the sensor 11a and the sensor 11b completely covers the detection target area 40 to be detected. state.

Next, coordinate conversion processing (generation of installation information) performed based on the trajectory of the moving body (worker) moving in the detection target area 40 will be described.
FIG. 8B is a diagram showing the trajectory of the worker in the installation state of FIG. 8A. (a) is a plan view of an imaging space, and (b) is an enlarged view of a worker's trajectory. The display here follows the local coordinate system of each sensor. Trajectories on the imaging planes 42a and 42b when the operator moves in the detection target area 40 are indicated by reference numerals 50a and 50b. Here, the operator appears from the left end boundary of the imaging plane 42a by the sensor 11a, moves in the X-axis right direction, and disappears from the right end boundary of the imaging plane 42a, as indicated by the arrow. Next, it appears from the left end boundary of the imaging plane 42b of the sensor 11b, moves in the X-axis right direction, and disappears from the right end boundary of the imaging plane 42b. Suppose that the object moves in this order while slightly meandering in the Y-axis direction.

As shown in the figure, since the local coordinate axes (Y coordinate, Z coordinate and X axis direction) of the sensors 11a and 11b are aligned, the trajectories 50a and 50b are smoothly connected at the boundary between the imaging planes 42a and 42b. You can see that there is. In the drawing (b), the trajectories 50a and 50b are shown shifted in the Y-axis direction to distinguish them.

In the above-described coordinate transformation step (S342 in FIG. 6C), the rotation angle of the coordinate system of the sensors 11a and 11b is calculated from the direction of the vector 51a when the right end of the trajectory 50a disappears and the vector 51b when the left end of the trajectory 50b appears. is generated as quantitative information. That is, the sensor system 1 presents information (angular deviation) of the vectors 51a and 51b of the connecting portion of the sensors 11a and 11b to the installation operator. In this example, both vectors 51a and 51b match, and the angular deviation is 0 (coordinate rotation is unnecessary). Accordingly, the installation worker quantitatively confirms that the installation state of the sensors 11a and 11b is as intended (no deviation in position and angle), and completes the installation work.

Next, an installation state that requires coordinate rotation transformation will be described.
FIG. 9A is a diagram showing a photographing space in an installation state that requires rotation conversion. (a) is a side view, and (b) is a plan view. The display here is in world coordinates. Unlike the installation state shown in FIG. 8A, the installation positions and installation directions (shooting directions) of the two sensors 11a and 11b are deviated. That is, the installation positions of the two are shifted in the X-axis direction by a distance Xo and in the Y-axis direction by a distance Yo. Moreover, the installation directions are also different, and the sensor 11a is in a state where the X axis is rotated by an angle θ about the Z axis with respect to the sensor 11b. As a result, the directions of the imaging planes 42a and 42b are not aligned, and the boundaries between the two are non-parallel.

FIG. 9B is a diagram showing the trajectory of the worker in the installation state of FIG. 9A. (a) is a plan view of an imaging space, and (b) is an enlarged view of the trajectory. The display here is in the direction of the local coordinates of each sensor, and the photographing planes 42a and 42b are arranged in parallel. As is clear from the figure, the two trajectories 50a and 50b are not smoothly connected at the boundary of the imaging plane, and the directions of the vector 51a when the trajectory 50a disappears and the vector 51b when the trajectory 50b appears deviate by an angle θ. ing.

At this time, the sensor system 1 informs as the sensor installation information that there is an angular deviation θ between the vectors 51a and 51b of the connecting portion, and automatically corrects this angular deviation in the coordinate conversion step (S342). If the coordinates cannot be corrected by coordinate conversion, an error warning to that effect is issued.

FIG. 9C is a diagram showing a state after the trajectory in FIG. 9B is corrected by coordinate transformation. (a) is a plan view of an imaging space, and (b) is an enlarged view of the trajectory. Here, the world coordinates common to the two sensors are displayed. By rotating the local coordinates of the sensor 11a by the angle θ shown in FIG. 9B, that is, by rotating the imaging plane 42a to the imaging plane 42a′, the vanishing vector 51a′ of the trajectory 50a′ on the sensor 11a is obtained in the world coordinates. , the vector 51b at the time of appearance of the trajectory 50b on the sensor 11b can be made parallel. Furthermore, in the coordinate transformation, the trajectory 50a' and the trajectory 50b can be smoothly connected by correcting the deviation Yo in the Y-axis direction.

In this example, a state in which two sensors are tilted around the Z axis has been described, but even in a state in which the sensors are tilted with the X or Y axis as the rotation axis, correction can be similarly performed by coordinate transformation. . Here, an installation state that cannot be corrected will be described.

FIG. 9D is a diagram showing an example of an installation state that cannot be corrected. For example, when the deviation (angle θ) between the installation directions of the two sensors is excessive, even if the trajectory 50a' and the trajectory 50b can be connected by the coordinate conversion, the gap 43 is formed at the boundary between the adjacent photographing planes 42a' and 42b. may occur. In the region of this gap 43, none of the sensors can perform the detection operation, which is inappropriate. In this case, an error warning is displayed.

10A-D and 11A-D are display examples of error warnings and sensor installation assistance information. The screen 10a of the display device 10 displays the result of the coordinate conversion process, and the result of the automatic correction, or when the automatic correction is impossible, the user can display support information for manually adjusting the sensor position. offer.

10A to 10D show display examples when there is a deviation in the installation angle of the sensor. Here, six sensors S1 to S6 are installed, and their imaging planes 42a to 42f are formed adjacent to each other. Assume that the operator moves on each of the photographing planes 42a to 42f, and trajectories 50a to 50f are obtained.

FIG. 10A is a screen warning that there is a defect in the installation state (angle) of the sensor. On the screen 10a, trajectories 50a to 50f obtained by the six sensors S1 to S6 are displayed, and a message 90 is displayed to the effect that there is a defect in the installed state of the sensors.
FIG. 10B is a screen showing defect content information 91 and correction information 92 . The defect is caused by the angle of the sensor S1, and it indicates that the correction of -10 degrees rotation around the Z-axis is performed. It also shows that the imaging plane 42a is rotated as indicated by reference numeral 42a' due to this correction.

FIG. 10C is a screen showing correction results. The sensor S1 is replaced by the corrected imaging plane 42a'. The user confirms the corrected trajectory shown on this screen, and if it is determined that there is no problem, presses the OK button 93 to complete the installation. If it is determined that there is a problem, the NG button 94 is pressed and the sensor is installed again.

FIG. 10D is a screen indicating that correction is not possible. Along with the information 91 on the details of the defect, the reason why correction is impossible and installation support information 95 are displayed. Here, the reason why correction is not possible is that "the amount of correction exceeds the range of rotation correction (±15 degrees), and a gap is generated between adjacent areas" (the state in FIG. 9D). Reinstall so that it rotates +15 degrees or more around the Z axis." is displayed. The user can easily adjust the installation of the sensor by correcting the installation direction of the sensor based on the provided information.

11A to 11D show display examples when there is a deviation in the installation height of the sensor. Six TOF sensors S1 to S6 are arranged, and an operator moves on each imaging plane 42a to 42f. Trajectories 60a to 60f indicating the height of the worker are obtained and displayed on respective imaging side surfaces (XZ planes) 45a to 45f. In this case, since the same worker is moving, if there is no deviation in the installation height of the sensor, the locus of height should be the same on any of the imaging side surfaces 45a to 45f.

FIG. 11A is a screen warning that there is a defect in the installation state (height) of the sensor. On the screen 10a, trajectories 60a to 60f of the height of the worker acquired by the six TOF sensors (S1 to S6) are displayed.
FIG. 11B is a screen showing defect content information 91 and correction information 92 . The defect is caused by the height of the sensor S1, and it indicates that -200 mm correction is performed in the height direction. Also, this correction causes the imaging side surface 45a to be translated as indicated by reference numeral 45a'.

FIG. 11C is a screen showing correction results. The sensor S1 is replaced by the corrected imaging plane 45a'. The user confirms the corrected trajectory shown on this screen, and if it is determined that there is no problem, presses the OK button 93 to complete the installation. If it is determined that there is a problem, the NG button 94 is pressed and the sensor is installed again.

FIG. 11D is a screen indicating that correction is not possible. Along with the information 91 on the details of the defect, the reason why correction is impossible and installation support information 95 are displayed. Here, the reason why correction is not possible is that the amount of correction exceeds the height correction range (500 mm), making it impossible to sense the whole body of a person. Fix it" is displayed. The user can easily perform sensor installation adjustment work by correcting the installation position of the sensor based on the provided information.

When the coordinate conversion is automatically corrected in this manner, an error warning is issued if the angle or position correction range is exceeded or if there is a gap between the photographing areas of the sensors installed adjacently. Then, sensor installation support information such as which sensor and which sensor joint has an error (especially when using three or more sensors) and how much the orientation and position of those sensors should be corrected is provided. , is displayed on the display device 10 . In accordance with this, the installation worker can confirm the indicated point and promptly correct it according to the support information.

Alternatively, a warning is issued when the movement (walking) of the moving body (worker) is not smooth and interrupted. For example, if a pedestrian stops, suddenly changes course, or reverses, the system issues a warning. In addition, there is a risk of errors in linking trajectories at places where the direction of pedestrian movement is likely to change, such as the corners of corridors and doorways. As a result, the trajectories can be connected in sections where the amount of change in the trajectory is small, and the accuracy of coordinate alignment can be improved.

After the sensor is installed, if the orientation or position of the sensor is misaligned during system operation, the distance image and luminance image displayed between adjacent imaging areas will become discontinuous. In order to deal with this, it is desirable to periodically check the installation state of the sensor and, if there is a deviation in the direction or position of the sensor, perform automatic correction or warn the user. As a confirmation method at that time, the trajectory of the moving object (worker) may be acquired again and the trajectory connection process described above may be performed. The trajectory linking process described above can also be performed.

According to the first embodiment, since the movement trajectory of the moving body (worker) is used for the calibration (coordinate matching) of a plurality of sensors, there is no need to arrange a dedicated marker in the measurement space, and the sensors can be installed. There is an effect that the work can be easily performed.

Here, a method for more reliably detecting a moving object will be described. If the moving object is a person, the person to be detected can be identified by identifying the person's posture or gesture by moving while maintaining a specific posture or gesture. Therefore, it is possible to acquire the trajectory of the moving body and coordinate matching processing even in a situation where a plurality of persons are captured in the shooting range. Specific postures and gestures are, for example, moving with both hands raised, moving with hands on the waist, moving while looking at feet, and the like. These can be arranged and set as the initial state at the start of movement, and can be handled flexibly.

Alternatively, instead of the moving body (person) making a gesture, a method of having or pasting a marker identifying the moving body may be used. In other words, a marker with a specific surface shape (texture) can be used. Lifting and moving, etc. For example, in the case of a TOF sensor, infrared light is emitted and the reflected light is detected. Therefore, a helmet attached with a tape that reflects infrared light with high reflectance or a marker with a pattern made of high reflectance tape is used. Good. The pattern may be, for example, a bar code, a QR code (registered trademark), or letters or symbols such as alphabets.

Embodiment 2 describes a modification of the floor surface detection process in Embodiment 1. FIG. The components of the sensor system 1 are the same as in FIG. 1B, and the block diagrams of the sensors 11a and 11b and the sensor cooperation processing device 12 are the same as in FIGS. The floor surface detection processing (FIGS. 5A to 5E) of the first embodiment is replaced as follows.

FIG. 12A is a flowchart of floor surface detection processing (S400). This replaces the floor detection process (S200) in the first embodiment, and omits the plane detection process (S210) in FIG. 5A.

S410 is processing for detecting the trajectory of the moving object, which is the same as the trajectory detection processing (S220) in FIG. 5A. Details thereof have been described in FIG. 5D of the first embodiment.
In S420, it is determined whether or not the trajectory of the moving object acquired in S410 and the point group are parallel.

FIG. 12B is a detailed flowchart of the parallel determination process (S420) in FIG. 12A. This corresponds to the parallel determination process (S230) in FIG. 5E of the first embodiment.
In S421, the trajectory of the moving object obtained in S410 is translated in a certain direction by a certain distance.

In S422, the trajectory moved in S421 is matched with the point group, that is, the distance between the point group and the trajectory is calculated.
In S423, it is determined whether or not the steps of S421 and S422 have been calculated for all combinations (patterns) of all directions and all distances. If all patterns are completed (Yes), the process proceeds to S424. If not completed (No), return to S421 and repeat the above steps.

In S424, the direction of parallel movement with the best matching is regarded as the vertical direction, and the region of the point cloud with the best matching is regarded as the floor surface.

In the first embodiment, the floor is regarded as being detected by detecting a flat surface. In the second embodiment, however, even if the floor is not flat, based on the feature that a person moves parallel to the floor, the vertical direction is detected. and the floor surface can be detected.

In the third embodiment, a case will be described in which a plurality of sensors are installed in a vehicle such as an automobile, and distances to pedestrians and other vehicles outside the vehicle are measured.

FIG. 13A is an external view of a sensor system 1' according to Example 3. FIG. In the sensor system 1', two sensors 11a and 11b are installed on the front of the vehicle 7 to capture an object to be detected (for example, the pedestrian 3) on the road surface 6 in front to generate a range image. The directions of the sensors are adjusted so that the areas 2a and 2b photographed by each sensor are substantially aligned. Each of the sensors 11 a and 11 b is connected to a sensor cooperative processing device 12 and a display device 10 mounted on the vehicle 7 via a network 13 . While the vehicle 7 is running, the display device 10 displays a distance image of pedestrians and other vehicles outside the vehicle and provides it to the driver.

FIG. 13B is a block diagram showing the configuration of the sensor system 1'. The basic configuration is the same as that of the sensor system 1 of FIG. 1B, but in the sensor cooperation processing device 12, detection of the road surface instead of the floor surface and trajectory superimposition processing are performed. In the third embodiment, since the regions 2a and 2b photographed by the respective sensors are substantially the same, coordinate conversion is performed in the process of coordinate matching so that the entire trajectory of the moving object is the same. This is called "trajectory superimposition processing". The configurations of the sensors 11a and 11b and the sensor cooperation processing device 12 are the same as those shown in FIGS. 2 and 3 of the first embodiment.

Next, the coordinate matching process in the sensor system 1' of the third embodiment, that is, the road surface detection process and the trajectory superimposition process will be described.
First, the road surface detection process is performed in the same manner as the floor surface detection process in the first embodiment. That is, in the flowcharts of FIGS. 5A to 5E, "floor surface" should be read as "road surface". Alternatively, as a modification thereof, the flow charts of FIGS. 12A to 12B of the second embodiment may be followed.

Next, in the trajectory superimposing process, coordinate transformation is performed so that the positions and directions of the trajectories simultaneously detected by the two sensors 11a and 11b are matched. Specifically, a vector at the time of appearance and a vector at the time of disappearance of one detected trajectory are obtained, and coordinate transformation is performed so that the respective vectors match between the two sensors.

FIG. 14A is a flow chart of the trajectory superimposition process (S500). Of these, S510 to S530 are the same as in FIG. 6A, and will be briefly described.
In S510, the operator installs a plurality of sensors 11a and 11b on the vehicle so that the photographing areas substantially match. Get self-location such as installation height and orientation from

In S530, the moving body (worker) walks on the road surface and the moving body trajectory is detected. Details of moving object trajectory detection are as described in FIG. 6B, but in the third embodiment, the moving object moves beyond the imaging range of each sensor, and the obtained trajectory is divided into a plurality of pieces.
In S540, a process of superimposing the trajectory obtained in S530, that is, a coordinate conversion process is performed. This process differs from S340 of FIG. 6A (FIG. 6C).

FIG. 14B is a detailed flowchart of the coordinate conversion process (S540). Here, parameters for coordinate transformation for superimposing the trajectories of each sensor obtained in S530 so as to minimize the shift are acquired.

In S541, for two trajectories acquired at the same time by the two sensors 11a and 11b, a vector paired with a vector when the trajectory appears and a vector when the trajectory disappears (hereinafter referred to as a feature vector) is generated. call). That is, the feature vector is the set of vectors at both ends of the segmented trajectory.
In S542, one of the trajectories is coordinate-transformed so that the feature vectors of the two sensors 11a and 11b obtained in S541 match in position and direction. Coordinate transformation involves translation and rotation of coordinate axes.

In S543, it is determined whether or not the two feature vectors match due to the coordinate transformation in S542. If they match (Yes), the process proceeds to S545. If they do not match (No), the process advances to S544 to display an error warning indicating that the coordinate matching process has failed and sensor installation support information. The error warning and sensor installation support information are the same as those described in the first embodiment.
In S545, it is determined whether or not there remains a trajectory that has not been superimposed. If there is a remaining trajectory, the processing after S541 is repeated, and if there is no remaining trajectory, the process ends.

Next, the coordinate conversion process (S542) will be described in detail, taking a specific installation state of a plurality of sensors as an example.
First, a case where the sensor installation state does not require coordinate rotation conversion in the coordinate conversion process (S542) will be described. In other words, this is the case where the trajectories of the two sensors are superimposed even without performing the rotation conversion process.

FIG. 15A is a diagram showing an imaging space in an installation state that does not require rotation conversion. (a) is a side view, and (b) is a plan view. The display here is in world coordinates. The X axis is the width direction of the vehicle body, the Y axis is the front of the vehicle body, the Z axis is the vertical direction of the vehicle body, and the XY plane is the road surface.

The two sensors 11a and 11b are installed on the vehicle body so as to face each other at the same height and are parallel-moved by a predetermined distance Xo in the X-axis direction. In other words, if the installation positions of the two sensors 11a and 11b are expressed in world coordinates, the X coordinate is the position shifted by the distance Xo, and the Y and Z coordinates are the same position. The installation directions (imaging directions) of the two sensors 11a and 11b are directions facing each other symmetrically with respect to the YZ plane, and the Y axes of the world coordinates and the local coordinates match. As a result, the imaging space of the sensor 11a (the space surrounded by the angle of view 41a and the imaging plane 42a) and the imaging space of the sensor 11b (the space surrounded by the angle of view 41b and the imaging plane 42b) overlap. The imaging planes 42a and 42b are coincident. The imaging space of both sensors 11a and 11b covers the detection target area 40 (one-dot chain line) in which the detection target exists.

Next, coordinate conversion processing (generation of installation information) will be described.
FIG. 15B is a diagram showing the trajectory of the worker in the installation state of FIG. 15A. (a) is a plan view of an imaging space, and (b) is an enlarged view of a worker's trajectory. The display here follows the local coordinate system. When the operator moves in the detection target area 40, trajectories on the imaging planes 42a and 42b are indicated by reference numerals 50a and 50b. As indicated by the arrow, the operator emerges from the left end boundary of the imaging plane 42a by the sensor 11a, moves in the right direction of the X axis, and disappears from the right end boundary of the imaging plane 42a. At the same time, it appears from the left end boundary of the imaging plane 42b by the sensor 11b, moves in the X-axis right direction, and disappears from the right end boundary of the imaging plane 42b. Here, it is assumed that the object slightly meanders in the Y-axis direction.

As shown in the figure, since the local coordinate axes (Y coordinate, Z coordinate and X axis direction) of the sensors 11a and 11b are aligned, the trajectories 50a and 50b completely match on the imaging planes 42a and 42b. You can see the situation. (In the drawing, the two trajectories 50a and 50b are shown slightly shifted in order to distinguish them, but they actually match).

In the step of coordinate transformation described above (S542 in FIG. 14B), the direction of the feature vector of the trajectory 50a (a pair of 52a at the time of appearance and 53a at the time of disappearance) and the feature vector of the trajectory 50b (52b at the time of appearance and 53a at the time of disappearance 53b), the rotation angle of the coordinate system of the sensors 11a and 11b is generated as quantitative information. That is, the sensor system 1' presents information (angular deviation) of the feature vectors 52a and 53a and the feature vectors 52b and 53b at both ends of the imaging planes of the sensors 11a and 11b to the installation operator. In this example, both feature vectors match, and the angle deviation is 0 (no coordinate rotation required). Accordingly, the installation worker quantitatively confirms that the installation state of the sensors 11a and 11b is as intended (no deviation in position and angle), and completes the installation work.

Next, an installation state that requires coordinate rotation transformation will be described.
FIG. 16A is a diagram showing an imaging space in an installation state that requires rotation conversion. (a) is a side view, and (b) is a plan view. The display here is in world coordinates. Unlike the installation state shown in FIG. 15A, the installation positions and installation directions (shooting directions) of the two sensors 11a and 11b are shifted. That is, the installation positions of the two are shifted in the X-axis direction by a distance Xo and in the Y-axis direction by a distance Yo. Moreover, the installation directions are also different, and the sensor 11a is in a state where the X axis is rotated by an angle θ about the Z axis with respect to the sensor 11b. As a result, the directions of the photographing planes 42a and 42b are not aligned, and the peripheral lines of both are non-parallel.

FIG. 16B is a diagram showing the trajectory of the worker in the installation state of FIG. 16A. (a) is a plan view of an imaging space, and (b) is an enlarged view of the trajectory. The display here is in the direction of the local coordinates of each sensor, and the areas of the imaging planes 42a and 42b are arranged so as to be parallel and coincide. The two trajectories 50a and 50b do not match in the imaging plane, and the directions of the feature vectors 52a and 53a of the trajectory 50a and the feature vectors 52b and 53b of the trajectory 50b are shifted by an angle θ.
At this time, the sensor system 1′ notifies as sensor installation information that there is an angular deviation θ between the feature vectors at both ends, and automatically corrects this angular deviation in the coordinate conversion step (S542).

FIG. 16C is a diagram showing a state after the trajectory in FIG. 16B is corrected by coordinate transformation. (a) is a plan view of an imaging space, and (b) is an enlarged view of the trajectory. Here, the world coordinates common to the two sensors are displayed. By rotating the local coordinates of the sensor 11a by the angle θ shown in FIG. ') can be matched to the trajectory 50b (feature vectors 52b, 53b) of the sensor 11b.

However, if correction by coordinate conversion is not possible, an error warning as shown in FIGS. 10D and 11D of the first embodiment and support information for manual adjustment are provided. The user can easily perform sensor installation adjustment work by correcting the installation direction and position of the sensor based on the provided information.

One of the reasons why the coordinate matching process fails in the third embodiment is that the photographing ranges of the two sensors are largely out of alignment. At that time, a warning is issued to the effect that the deviation of the field of view (shooting range) of the installed sensor is too large. Alternatively, there may be a problem in how the moving body (worker) moves (walks). For example, if you stop for a long time or run in a straight line at high speed, you will be warned to move at an appropriate speed. In addition, by performing this process at places where the direction of movement of people is likely to change, such as road corners and intersections, characteristic trajectories can be obtained, facilitating the process of matching the trajectories. Therefore, unlike the case of the first embodiment, it is possible to reduce the error when superimposing the trajectory by instructing the movement to be performed under the condition that the amount of change in the trajectory of the movement is large.

In the examples of FIGS. 15A and 16A, the imaging directions of the two sensors 11a and 11b are set to face each other in the X-axis direction, and the detection target area 40 is an area sandwiched between the two sensors. However, when actually detecting a pedestrian or the like in front of the vehicle, the detection target area 40 is located away from the position of the sensor in the Y-axis direction. In this case, the photographing directions of the two sensors 11a and 11b may be tilted in the Y-axis direction by a predetermined angle so that they face each other obliquely. Then, the coordinates of the two sensors may be adjusted based on the state in which the sensors are installed symmetrically with respect to the YZ plane.

In the third embodiment, as in the first embodiment, the movement trajectory of the moving body (worker) is used for the calibration (coordinate matching) of a plurality of sensors, so that the sensor installation work can be easily performed. has the effect of being able to

Furthermore, since the sensor system of Example 3 is configured to detect the same measurement space with a plurality of sensors, it is possible to detect an object existing in the measurement space without blind spots. In addition, it becomes possible to combine different types of detection methods as sensors. In other words, not only the TOF sensor but also any combination of sensors capable of distance measurement, such as scanning laser radar, stereo camera, millimeter wave radar, etc., can be used to coordinate coordinates and generate a range image. can.

Furthermore, even if one of the sensors to be combined is not a distance measuring sensor, if spatial coordinate information corresponding to non-distance data obtained from that sensor is held in advance, it can be converted into distance data, so there is no problem. Alternatively, since distance information can be estimated by methods such as machine learning and deep learning for non-distance data obtained from non-ranging sensors, coordinate matching is performed and range images are generated in the same way as when combining ranging sensors. can be generated.

In this way, by combining sensors with different types of detection methods, it is possible to switch to the optimum sensor according to the type of target object and the measurement environment (temperature, brightness, distance), improving the measurement accuracy of the sensor system. has the effect of

1, 1' ... sensor system,
10... display device,
11... sensor (ranging sensor),
12 ... sensor cooperation processing device,
21... Light-emitting part,
22... Light receiving part,
23 ... distance image generator,
24... Luminance image generator,
35... Arithmetic processing unit,
40 ... detection target area,
42... Imaging plane,
50... Trajectory of moving body,
51 to 53... vector,
90 to 92... error information,
95: Installation support information.

Claims (11)

In a sensor system that installs multiple sensors and measures the distance to a detection target within a detection target area to generate a range image,
a sensor cooperation processing device that performs coordinate matching processing between the sensors in order to synthesize distance images from the plurality of sensors installed so that the respective detection target areas are adjacent to each other;
a display device for displaying the generated distance image,
a plurality of imaging planes, which are areas of the floor surface included in the detection target area of each of the plurality of sensors, are arranged continuously with boundaries between the imaging planes adjacent to each other being in contact with each other;
In order to perform coordinate matching processing between the sensors, the sensor-coordinated processing device performs a coordinate matching process between the sensors, so that when the person moves on the floor surface within the detection target area by the plurality of sensors, the sensor-coordinated processing device detects the distance to the person detected by the sensors. By arranging the predetermined positions of the person based on the point cloud data obtained from the distance in time series, trajectory data indicating the trajectory representing the movement of the predetermined position of the person is acquired, and at the boundary of the adjacent detection target area, A sensor system characterized by transforming coordinates detected by each sensor by rotating and translating the coordinates so as to eliminate deviations in the positions of the ends of the trajectory and deviations in the direction of the trajectory on the imaging plane detected by each sensor. . The sensor system of claim 1, wherein
The sensor cooperation processing device is
At the boundaries of the adjacent detection target areas, the detection coordinates of each sensor are parallel to each other so as to eliminate positional deviation of the end of the locus representing the movement of the predetermined position defining the height of the person on the side surface of the image. A sensor system characterized by converting by movement. The sensor system of claim 1, wherein
A sensor system, wherein the sensor cooperation processing device detects a floor surface within the detection target area by each sensor in order to perform coordinate matching processing between the sensors. In the sensor system of claim 3,
In order to detect the floor surface, the sensor-coordinated processing device acquires distance data within the detection target area and the trajectory data of the person moving within the detection target area using each sensor, and extracts the distance data. and determining a plane parallel to the plane formed by the locus data as the floor surface. In the sensor system of claim 3,
In order to detect the floor surface, the sensor cooperation processing device obtains the trajectory data of the person moving within the detection target area using each sensor, and detects a plane parallel to a plane formed by the trajectory data. A sensor system characterized by determining a plane. The sensor system of claim 1, wherein
The sensor cooperation processing device displays, on the display device, the name of the sensor to be corrected and its rotation angle or translation amount as correction information when performing coordinate matching processing between the sensors. The sensor system of claim 1, wherein
When the coordinate matching process between the sensors cannot be performed, the sensor cooperation processing device displays the name of the sensor to be adjusted and its rotation angle or translation amount as sensor installation support information when the user performs manual adjustment. A sensor system characterized by displaying on a device. A plurality of sensors installed so that respective detection target areas are adjacent to each other and installation positions are different, wherein a plurality of imaging planes, which are areas of the floor surface included in the respective detection target areas, are adjacent to each other. A sensor cooperation processing device that cooperatively processes distance data within the detection target area measured by the plurality of sensors that are continuously arranged with the boundaries between the planes in contact with each other,
Having an arithmetic processing unit that performs coordinate matching processing between the plurality of sensors and synthesizes distance data from each sensor,
In order to perform coordinate matching processing between the sensors, the arithmetic processing unit calculates the distance to the person detected by the sensors when the person moves on the floor surface in the detection target area by the plurality of sensors. By arranging the predetermined positions of the person based on the point cloud data obtained from from the Coordinate transformation parameters of each sensor when transforming the coordinates detected by each sensor by rotating and translating the coordinates so as to eliminate the positional deviation of the end of the trajectory and the deviation of the direction of the trajectory on the imaging plane detected by the sensor. A sensor-coordinated processing device characterized by defining In the sensor cooperation processing device according to claim 8,
When performing coordinate matching processing between the sensors, the arithmetic processing unit is configured to generate a tangent vector of an end of the trajectory detected by each sensor when the trajectory of the person appears in the detection target region, or a tangent vector of the detection target region. a tangential vector at the end of the trajectory when the trajectory disappears from the sensor, and determining coordinate transformation parameters of each sensor so that the directions of the obtained vectors match between the sensors. The detection target areas of the plurality of sensors are adjacent to each other, and the plurality of imaging planes, which are the areas of the floor surface included in the detection target areas, are continuous with the borders between the adjacent imaging planes being in contact with each other. In the distance measurement method for generating a distance image by measuring the distance to the detection target in the detection target area so as to be arranged in the detection target area,
Predetermined positions of the person based on point cloud data based on the distance to the person detected by the sensors when the person moves on the floor within the detection target area are arranged in chronological order. obtaining trajectory data indicating a trajectory representing the movement of the person at a predetermined position;
Coordinate rotation and parallel translation of the detection coordinates of each sensor so as to eliminate deviation of the position of the end of the trajectory on the imaging plane detected by each sensor and deviation of the direction of the trajectory on the boundary of the adjacent detection target areas. a step of performing coordinate alignment between the sensors by transforming by
a step of synthesizing and displaying distance images from the plurality of sensors in a state in which coordinates are aligned;
A distance measurement method comprising: In the distance measurement method according to claim 10 ,
At the boundaries of the adjacent detection target areas, the detection coordinates of each sensor are parallel to each other so as to eliminate positional deviation of the end of the locus representing the movement of the predetermined position defining the height of the person on the side surface of the image. A distance measuring method, comprising a step of performing coordinate matching between the sensors by transforming by movement.

Images