JPWO2020246202A1 - Measurement system, measurement method, and measurement program - Google Patents

Abstract

It is a measurement system used for a moving body traveling on a road surface, and includes an image pickup device and an information processing device. The image pickup device is configured to be able to image a road surface or an obstacle included in an angle of view in a time series. The information processing apparatus includes a communication unit, an IPM conversion unit, an obstacle determination unit, an empty area setting unit, and a measurement unit. The communication unit is connected to the image pickup device and is connected by the image pickup device. The IPM conversion unit is configured to be able to receive the captured image, and the IPM conversion unit converts the image into a back-perspective projection to generate an IPM image (an image drawn so as to overlook a predetermined plane including the road surface or obstacles). The obstacle determination unit is configured to be able to determine the presence / absence or position of the obstacle in the IPM image, and the empty area setting unit is based on the presence / absence or position of the determined obstacle. The empty area (the area in the IPM image that does not include the obstacle) can be variably set, and the measuring unit can measure the absolute speed of the moving body based on the empty area. ..

Description

The present invention relates to a measurement system, a measurement method, and a measurement program.

In the industrial world, it is required that moving objects such as automobiles and mobile robots can accurately grasp their own absolute speed. For example, in a general automobile, the number of revolutions per unit time of a tire is measured by a vehicle speed sensor (which can output a predetermined pulse according to the rotation), and the absolute speed is multiplied by the length of the outer circumference of the tire. Can be estimated.

Japanese Unexamined Patent Publication No. 2015-21221

However, such a method obtains an approximate value for reference when driving a human, and lacks accuracy. Prior to the development of autonomous driving technology in the future, a more accurate measurement method different from the conventional method is required. Further, similar needs have arisen not only in cars but also in mobile robots and the like.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide a measurement system, a measurement method, and a measurement program which are used in a moving body and can measure the absolute speed of the moving body with high accuracy. ..

According to the present invention, it is a measurement system used in a moving body traveling on a road surface, and includes an image pickup device and an information processing device, and the image pickup device captures the road surface or obstacles included in the angle of view in a time series. The information processing apparatus includes a communication unit, an IPM conversion unit, an obstacle determination unit, an empty area setting unit, and a measurement unit, and the communication unit is connected to the image pickup device. The IPM conversion unit is configured to generate an IPM image by performing a back-perspective projection conversion of the image, wherein the image captured by the image pickup apparatus can be received. An image drawn so as to overlook a predetermined plane including the road surface or an obstacle, the obstacle determination unit is configured to be able to determine the presence or absence or position of the obstacle in the IPM image, and the empty area setting unit. Is configured to be able to variably set an empty area based on the presence or absence or position of the determined obstacle, wherein the empty area is a region in the IPM image that does not include the obstacle, and the measurement is performed. The unit is provided with a measurement system configured to be able to measure the absolute speed of the moving body based on the empty area.

In the measurement system according to the present invention, an empty area can be variably set in an IPM image, and by measuring the absolute velocity of a moving object based on such an empty area, highly accurate measurement can be realized. , Has an advantageous effect.

Schematic diagram of the system according to the embodiment. FIG. 2A is a schematic configuration diagram of the information processing apparatus, and FIG. 2B is a functional block diagram of a control unit in the information processing apparatus. Schematic diagram of the back-perspective projection transformation. The schematic diagram which shows the flow from the imaging of the 1st and 2nd cameras (left and right) to the acquisition of a difference image. FIG. 5A is a diagram showing an aspect of setting a variable empty area, and [FIG. 5B] a diagram showing an aspect of setting a fixed empty area. [FIG. 6A] A first histogram obtained from a difference image, [FIG. 6B] a second histogram obtained from a difference image. An activity diagram showing the flow of measurement methods.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The various features shown in the embodiments shown below can be combined with each other. In particular, as used herein, the term "part" may include, for example, a combination of hardware resources implemented by a circuit in a broad sense and information processing of software specifically realized by these hardware resources. .. Further, various information is handled in this embodiment, and these information are represented by high and low signal values as a bit set of binary numbers composed of 0 or 1, and communication / calculation is executed on a circuit in a broad sense. Can be done.

Further, the circuit in a broad sense is a circuit realized by at least appropriately combining a circuit, a circuit, a processor, a memory, and the like. That is, an integrated circuit for a specific application (Application Specific Integrated Circuit: ASIC), a programmable logic device (for example, a simple programmable logic device (Simple Programgable Logic Device: SPLD), a composite programmable logic device (Complex Programg)). It includes a programmable gate array (field programmed gate array (FPGA)) and the like.

1. 1. Overall Configuration Section 1 describes the overall configuration of the measurement system 1. FIG. 1 is a diagram showing an outline of the configuration of the measurement system 1 according to the present embodiment. The measurement system 1 includes an image pickup device 2 and an information processing device 3, and is a system in which these are electrically connected. It is preferable that the measurement system 1 is installed and used on a moving body traveling on a road surface (which may include not only a general road but also a defined floor surface in a room). The moving body is assumed to be, for example, a car, a train (including not only public transportation but also a game), a ship, a mobile robot, a human, an animal, and the like. In this specification, an automobile is taken as an example to explain, and an automobile equipped with the measurement system 1 is defined as "the present automobile". That is, the measurement system 1 is configured to be capable of measuring the absolute speed of the vehicle.

1.1 Imaging device 2
The image pickup device 2 is a so-called vision sensor (camera) configured to be able to acquire information on the outside world as an image, and it is particularly preferable to use a high-speed vision image pickup device 2 having a high image pickup rate. The imaging rate is, for example, 100 fps or more, preferably 250 fps or more, and more preferably 500 fps or 1000 fps. Specifically, for example, 100, 125, 150, 175, 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, 600, 625, 650, 675, 700, 725, 750, 775, 800, 825, 850, 875, 900, 925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1175, 1200, 1225, 1250, 1275, 1300, 1325, 1350, 1375, 1400, 1425, 1450, 1475, 1500, 1525, 1550, 1575, 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, It is 1900, 1925, 1950, 1975, 2000 fps (hertz), and may be in the range between any two of the numerical values exemplified here. More specifically, the image pickup device 2 is a so-called two-eye image pickup device including the first camera 21 and the second camera 22. Further, it should be noted that the angle of view of the first camera 21 and the angle of view of the second camera 22 have regions that overlap each other. Further, in the image pickup apparatus 2, a camera capable of measuring not only visible light but also a band that cannot be perceived by humans, such as an ultraviolet region and an infrared region, may be adopted. By adopting such a camera, it is possible to carry out measurement using the measurement system 1 according to the present embodiment even in a dark field, which is an advantageous effect.

<First camera 21>
The first camera 21 is provided in parallel with the second camera 22 in the measurement system 1, for example, and is configured to be capable of taking an image of the front left side of the vehicle. Specifically, the angle of view of the first camera 21 captures a road surface extending in front of the vehicle or an obstacle (for example, a vehicle in front, a pedestrian, an animal, etc.) located in front of the vehicle. Further, the first camera 21 is connected to the communication unit 31 in the information processing device 3 described later by a telecommunication line (for example, a USB cable or the like), and is configured to be able to transfer the captured image to the information processing device 3.

<Second camera 22>
The second camera 22 is provided in parallel with the first camera 21, for example, in the measurement system 1, and is configured to be capable of capturing an image of the front right side of the vehicle. Specifically, the angle of view of the second camera 22 captures the road surface extending in front of the vehicle or the vehicle in front (that is, an obstacle) located in front of the vehicle. Further, the second camera 22 is connected to the communication unit 31 in the information processing device 3 described later by a telecommunication line (for example, a USB cable or the like), and is configured to be able to transfer the captured image to the information processing device 3.

In other words, the road surface or obstacles included in the angles of view of the first and second cameras 21 and 22 can be captured in time series as the first and second images IM1 and IM2.

1.2 Information processing device 3
FIG. 2A shows a schematic configuration diagram of the information processing apparatus 3, and FIG. 2B shows a functional block diagram of the control unit 33 in the information processing apparatus 3. As shown in FIG. 2A, the information processing device 3 has a communication unit 31, a storage unit 32, and a control unit 33, and these components are inside the information processing device 3 via a communication bus 30. It is electrically connected. Hereinafter, each component will be further described.

<Communication unit 31>
Although wired communication means such as USB, IEEE1394, Thunderbolt, and wired LAN network communication are preferable, the communication unit 31 requires wireless LAN network communication, mobile communication such as 3G / LTE / 5G, Bluetooth (registered trademark) communication, and the like. It may be included according to the above. That is, it is more preferable to carry out as a set of these plurality of communication means. In particular, the first camera 21 and the second camera 22 in the above-mentioned image pickup apparatus 2 are configured to be able to receive images by communicating with a predetermined high-speed communication standard (for example, USB3.0, camera link, etc.). It is preferable to be done. In addition, a monitor (not shown) for displaying measurement results such as the absolute speed of the vehicle and an automatic control device (not shown) for automatically controlling (automatically driving) the vehicle based on the measurement results are connected. You may.

<Memory unit 32>
The storage unit 32 stores various information defined by the above description. This is, for example, as a storage device such as a solid state drive (SSD), or a random access memory (Random Access Memory:) that stores temporarily necessary information (arguments, arrays, etc.) related to program operations. It can be implemented as a memory such as RAM). Further, these combinations may be used.

In particular, the storage unit 32 stores the first image IM1 and the second image IM2 (image IM) imaged by the first camera 21 and the second camera 22 in the image pickup apparatus 2 and received by the communication unit 31. do. The storage unit 32 stores the IPM image IM'. Specifically, the first IPM image IM1'converted from the first image IM1 and the second IPM image IM2'converted from the second image IM2 are stored. Here, the image IM and the IPM image IM'are, for example, array information including pixel information of 8 bits each for RGB.

Further, the storage unit 32 stores an IPM conversion program for generating an IPM image IM'based on the image IM. The storage unit 32 calculates the difference D between the first IPM image IM1'and the second IPM image IM2', and the first histogram HG1 based on the angle (direction) and the second histogram based on the distance. It stores a histogram generation program for generating HG2. The storage unit 32 stores an obstacle determination program for determining the presence / absence or position of an obstacle such as a vehicle in front in the IPM image IM'(or difference D) based on the first histogram HG1 and the second histogram HG2. ing. The storage unit 32 stores an empty area setting program for setting an empty area R (described later) in the IPM image IM'(or difference D). The storage unit 32 stores a measurement program for measuring the absolute speed of the vehicle based on the empty area R. When there is an obstacle such as a vehicle in front, the storage unit 32 measures the relative speed of the obstacle with respect to the vehicle based on the time-series change of the position, and by extension, the absolute speed of the vehicle and the relative of the obstacle. It stores a measurement program for measuring the absolute velocity of an obstacle based on the velocity. Further, the storage unit 32 also stores various programs and the like related to the measurement system 1 executed by the control unit 33.

<Control unit 33>
The control unit 33 processes and controls the overall operation related to the information processing device 3. The control unit 33 is, for example, a central processing unit (CPU) (not shown). The control unit 33 realizes various functions related to the information processing device 3 by reading out a predetermined program stored in the storage unit 32. Specifically, it corresponds to an IPM conversion function, a histogram generation function, an obstacle determination function, an empty area setting function, a measurement function, and the like. That is, the information processing by the software (stored in the storage unit 32) is specifically realized by the hardware (control unit 33), so that the IPM conversion unit 331 and the histogram generation unit are as shown in FIG. 2B. It can be executed as 332, an obstacle determination unit 333, an empty area setting unit 334, and a measurement unit 335. In addition, although it is described as a single control unit 33 in FIGS. 2A and 2B, it is not actually limited to this, and it may be implemented so as to have a plurality of control units 33 for each function. Further, it may be a combination thereof. Hereinafter, the IPM conversion unit 331, the histogram generation unit 332, the obstacle determination unit 333, the empty area setting unit 334, and the measurement unit 335 will be described in more detail.

[IPM conversion unit 331]
In the IPM conversion unit 331, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The IPM conversion unit 331 is configured to be capable of executing a back-perspective projection conversion process for an image IM transmitted from the first camera 21 and the second camera 22 in the image pickup apparatus 2 and received by the communication unit 31. The back-perspective projection transformation will be described in detail in Section 2 with reference to FIGS. 3 and 4.

In other words, the first IPM image IM1'is generated by the back-perspective projection conversion of the first image IM1, and the second IPM image IM2' is generated by the back-perspective projection conversion of the second image IM2. Generated. Further, by comparing the first IPM image IM1'and the second IPM image IM2', the correspondence between these coordinates is estimated, and the true IPM image IM'is based on the estimated correspondence of the coordinates. It is good to correct the error with the value.

Further, it is preferable to carry out such a back-perspective projection conversion in a limited area (ROI) to shorten the processing time and increase the control rate of the entire measurement system 1. Such a predetermined area may be determined by processing a frame in the past (usually one before). As for the measurement system 1 as a whole, the lower of the image pickup rate of the first camera 21 and the second camera 22 and the operation rate of the control unit 33 acts as the control rate related to the measurement of the absolute speed of the vehicle. In other words, by increasing the imaging rate and the operating rate to the same extent, it is possible to measure the absolute speed of the vehicle even with feedback control alone.

[Histogram generator 332]
In the histogram generation unit 332, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The histogram generation unit 332 calculates the difference D between the first IPM image IM1'and the second IPM image IM2', and subsequently generates a plurality of histogram HGs generated based on different parameters. Such a histogram HG may also be limited to a predetermined region determined in the past frame. Specifically, a first histogram HG1 based on an angle (direction) and a second histogram HG2 based on a distance are generated. Further, the histogram generation unit 332 may determine a predetermined region to be used in the processing in the next frame based on the generated first histogram HG1 and the second histogram HG2. See FIGS. 6A and 6B for an example of a histogram.

[Obstacle determination unit 333]
In the obstacle determination unit 333, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The obstacle determination unit 333 determines the presence or absence of an obstacle or its position based on the difference D calculated by the histogram generation unit 332 and the first histogram HG1 and the second histogram HG2. If it is determined that there is a vehicle in front as an obstacle, the position may be appropriately presented to the driver of the vehicle via a monitor (not shown). Further, an appropriate control signal may be transmitted to the automatic control device for automatically controlling (automatically driving) the vehicle based on the determination result.

[Empty area setting unit 334]
In the empty area setting unit 334, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The empty area setting unit 334 can variably set the empty area R based on the presence / absence or position of an obstacle determined by the obstacle determination unit 333. Here, the empty region R means an region in the IPM image IM'that does not include obstacles. For example, the empty area R is set to be rectangular (rectangular or square). In particular, by adopting a rectangular shape, particularly a square shape, there is an advantageous effect that the calculation can be made more efficient. Further, the empty area setting unit 334 sets the empty area R so that the number of pixels is large in a range not including obstacles. Basically, it should be noted that the larger the number of pixels in the empty area R, the higher the measurement accuracy of the absolute speed of the vehicle, which will be described later, within the range below the predetermined upper limit of the number of pixels. Further, the empty area setting unit 334 sets the empty area R variably for each frame of the IPM image IM'. See FIG. 5A for this.

[Measurement unit 335]
In the measurement unit 335, information processing by software (stored in the storage unit 32) is specifically realized by hardware (control unit 33). The measuring unit 335 can measure the absolute speed of the vehicle with high accuracy based on the empty area R set by the empty area setting unit 334. When the obstacle determination unit 333 determines that there is an obstacle, the measurement unit 335 measures the relative speed of the obstacle with respect to the vehicle based on the time-series change in the position of the obstacle. Further, the measuring unit 335 measures the absolute speed of the obstacle with high accuracy based on the absolute speed of the vehicle and the relative speed of the obstacle. This will be described in more detail in Section 3.

ただし、Ｋはカメラ（第１のカメラ２１及び第２のカメラ２２）の内部行列、Πはカメラ座標系Ｏ＿Ｃからカメラ画像平面π＿Ｃへの射影行列、Ｒ∈ＳＯ（３）及びＴ∈Ｒ＾３は、世界座標系Ｏ＿Ｗからカメラ座標系Ｏ＿Ｃへの回転行列、及び並進ベクトルをそれぞれ表す。 2. 2. Back-perspective projection conversion Section 2 describes the back-perspective projection conversion. It should be noted that FIG. 3 is a schematic diagram of the back-perspective projection conversion, and here, a pinhole camera is assumed as a model, and the expression is made considering only the pitch angle of the camera. Of course, a fisheye camera or an omnidirectional camera may be assumed, or a formula may be expressed in consideration of the roll angle. As shown in FIG. 2, the points (x, y) when the points (X_W, Y_W, Z_W) represented by the world coordinate system O_W are projected onto the camera image plane π_C are represented as [Equation 1]. To.

However, K is the internal matrix of the cameras (first camera 21 and second camera 22), Π is the projection matrix from the camera coordinate system O_C to the camera image plane π_C, R ∈ SO (3) and T ∈ R ^ 3. Represents a rotation matrix from the world coordinate system O_W to the camera coordinate system O_C and a translation vector, respectively.

と表されるとき、画像上の点（ｘ，ｙ）の逆透視投影像であるπ上の点（Ｘ＿Ｗ，Ｙ＿Ｗ，Ｚ＿Ｗ）は、（ｘ，ｙ）を用いて、［数３］のように計算される。

Now, consider the case where the obstacles shown in the first camera 21 and the second camera 22 exist only on the plane π. At this time, since there is a one-to-one correspondence between the points on the image plane and the points on π, a one-to-one mapping from the image plane to π can be considered. This mapping is called reverse perspective mapping. R and T respectively

When the point (X_W, Y_W, Z_W) on π, which is a back-perspective projection image of the point (x, y) on the image, is expressed as [Equation 3] using (x, y). Is calculated to.

Here, f_x and f_y are focal lengths in the x and y directions, respectively, and (o_x, o_y) is the optical center. In the present embodiment, the image obtained by projecting the image IM captured by the image pickup apparatus 2 by this mapping is referred to as an IPM image IM'. When two cameras (first camera 21 and second camera 22) are imaging the same plane, a pair of IPM images IM'(first IPM image IM1' and second IPM image" are calculated. In IM2'), the brightness values of the pixels corresponding to one point on the plane are the same. However, when an obstacle that is not on a plane is present in the field of view, there is a difference in brightness between the IPM image IM'pairs. By detecting this difference (difference D), it is possible to detect an obstacle such as a vehicle in front existing in the field of view. Since this method is robust to the texture of a flat surface, it can accurately detect obstacles even in situations where a monocular camera is not good at shadows.

These flows are shown in FIG. First, the first image IM1 by the first camera 21 (left) and the second image IM2 by the second camera 22 (right) are imaged. Subsequently, a first IPM image IM1 ′ obtained by converting the first image IM1 and a second IPM image IM2 ′ obtained by converting the second image IM2 are generated. Further, a difference D (binarized with a predetermined threshold value) between the first IPM image IM1'and the second IPM image IM2' is obtained. Note that in the difference D, the edge portion of the obstacle is detected as a white pixel. It should also be noted that the area in front of the white pixels (generally surrounded by the alternate long and short dash line in FIG. 4) is presumed to be an unobstructed road surface.

3. 3. Absolute speed measurement method of this vehicle Section 3 describes in detail the absolute speed measurement method of this vehicle.

3.1 Basic Principle Since the nearby road surface on which the vehicle travels is considered to be flat and stationary, the amount of movement of the vehicle, that is, the vehicle, is estimated by estimating the amount of movement of the road surface as seen from the image pickup device 2. The absolute speed of can be measured. At this time, the amount of movement of the road surface as seen from the image pickup device 2 is originally a three-dimensional amount, but if it can be assumed that the amount of movement in the height direction is constant between the frames of the image pickup device 2, it is assumed that the amount of movement in the height direction is constant with respect to the road surface. It can be formulated as a problem of estimating the amount of parallel two-dimensional movement. Since the measurement system 1 is fixed to the vehicle and used, the posture of the image pickup device 2 with respect to the road surface is known. That is, it should be noted that such estimation of the amount of movement is performed by continuously aligning the IPM image IM'obtained in time series. The alignment method is not particularly limited, but it is particularly preferable to use a phase-limited correlation method known as a highly accurate alignment method.

The basic principle of the phase-limited correlation method utilizes the feature that an extremely sharp peak corresponding to the translational amount is generated when the inverse Fourier transform is applied to the spectrum obtained by extracting only the phase information by normalizing the cross-power spectrum. It is a thing. By detecting this peak position with high accuracy, it is possible to estimate the translational parameters with high accuracy.

3.2 Setting of empty area R In this embodiment, when the above-mentioned alignment with respect to the road surface is performed, the alignment is performed with respect to the region estimated to be the road surface in the IPM image IM'(the portion of the alternate long and short dash line in FIG. 4). I do. FIG. 5A is a diagram (example) showing an embodiment in which an empty region R (empty region Ra to Rc) is variably set, and FIG. 5B sets the fixed regions Rd to Rf as an empty region R. It is a figure (comparative example) which shows the aspect.

In the embodiment (IPM images IM'a to IM'c) shown in FIG. 5A, the empty area R is set variably. That is, empty areas Ra to Rc of an appropriate size are set for each frame. Here, the empty region R is a rectangular shape (particularly a square) that does not include white pixels meaning obstacles and has as large a number of pixels as possible. However, for the convenience of calculation processing, the upper limit of the number of pixels in the empty area R is set by the number of pixels smaller than the total number of pixels of the IPM image IM'(or the difference D) (IPM image IM'c). Of course, the shape is not limited to this. For example, a circular shape, an elliptical shape, or a figure obtained by removing a part from the rectangle may be adopted.

On the other hand, in the comparative example (IPM images IM'd to IM'f) shown in FIG. 5B, the predetermined regions Rd to Rf are regarded as the empty region R. That is, in this comparative example, it is assumed that there are no obstacles in the range below a certain distance. The above-mentioned alignment is performed in such fixed regions Rd to Rf, but as in the regions Rd and Re, especially when an obstacle is located at a short distance to the vehicle. This may be included in the empty area R. In such a case, there is a problem that the estimation accuracy is lowered because a portion not represented by the similarity transformation is generated between the IPM images IM'in the time series.

In the measurement system 1 according to the present embodiment, highly accurate alignment can be realized by setting an appropriate empty area R so as to exclude obstacles with respect to the difference D. As a result, the measurement unit 335 of the information processing device 3 can measure the absolute speed of the vehicle.

3.3 Measurement of Relative Velocity of Obstacles As described above, the obstacle determination unit 333 is based on the difference D calculated by the histogram generation unit 332 and the first histogram HG1 and the second histogram HG2. Determine the presence or absence of an object or its position. Since the position of the obstacle here is the relative position with respect to the vehicle, it is possible to measure the relative speed of the obstacle by acquiring this in chronological order. In other words, when there is an obstacle, the measuring unit 335 can measure the relative speed of the obstacle with respect to the vehicle (moving object) with high accuracy based on the time-series change of the position.

3.4 Measurement of absolute speed of obstacles Since the measuring unit 335 can measure the absolute speed of the vehicle and the relative speed of obstacles with high time resolution and high accuracy, the measuring unit 335 is based on both. , It is also possible to measure the absolute velocity of obstacles. This makes it possible to grasp the absolute speed of the vehicle and the relative speed and absolute speed of obstacles located in the vicinity of the vehicle with high time resolution and high accuracy. Therefore, future automatic driving technology It is expected to greatly contribute to the development of such.

As described above, using one system 1, the absolute speed of the vehicle, the relative speed of the obstacle, and the absolute speed of the obstacle can all be measured with high accuracy. It is expected that such a system 1 will contribute to the control of moving objects such as automatic driving.

4. Measurement method Section 4 describes a measurement method using the measurement system 1 according to the present embodiment. FIG. 7 is an activity diagram showing the flow of the measurement method. Hereinafter, each activity in FIG. 7 will be described. As shown in the figure, activities A1 to A8 are repeated in chronological order.

[from here]
(Activity A1)
At a certain time t, the image pickup apparatus 2 (first camera 21 and second camera 22) images an object as an image IM (first image IM1 and second image IM2) at an image pickup rate of 100 fps or more.

(Activity A2)
Following the activity A1, the IPM conversion unit 331 performs a back-perspective projection conversion on the image IM to generate an IPM image IM'(first IPM image IM1'and a second IPM image IM2').

(Activity A3)
Following the activity A2, the histogram generation unit 332 calculates the difference D between the first IPM image IM1'and the second IPM image IM2'.

(Activity A4)
Following the activity A3, the histogram generator 332 generates a histogram HG (first histogram HG1 and second histogram HG2) generated based on different parameters (angle and distance), and based on this, the histogram HG is generated. The obstacle determination unit 333 determines the presence or absence of an obstacle or its position.

(Activity A5)
When the obstacle determination unit 333 determines that there is an obstacle following the activity A4, the measurement unit 335 determines the relative speed of the obstacle to the vehicle based on the time series information of the position including the past frame. To estimate.

(Activity A6)
Similarly, following the activity A4, the empty area setting unit 334 sets the empty area R so that the difference D does not include an obstacle (white pixel). The empty area R is variably set for each frame.

(Activity A7)
Following the activity A6, the measuring unit 335 measures the absolute speed of the vehicle by performing the time-series alignment calculation with respect to the empty area R.

(Activity A8)
Following the activities A5 and A7, the measuring unit 335 measures the absolute speed of the obstacle based on the absolute speed of the vehicle and the relative speed of the obstacle.
[So far]

According to such a measurement method, unlike Patent Document 1 described in [Prior Art Document], the absolute speed of the present vehicle can be measured by using the image pickup device 2 (camera) without using the acceleration sensor. It has the advantageous effect of being able to do it.

5. Modification Example Section 5 describes a modification of the measurement system 1 according to the present embodiment. That is, the measurement system 1 according to the present embodiment may be further creatively devised according to the following aspects.

First, when the measurement system 1 itself is configured to be movable, the empty region R is determined in consideration of at least one parameter of the speed, acceleration, movement direction, and surrounding environment of the measurement system 1. You may. In particular, it is preferable that the correlation between these parameters and the empty region R is learned in advance by machine learning. Further, it is preferable that further machine learning is performed while the measurement system 1 is continuously used to determine a more preferable empty region R.

Secondly, when there are a plurality of obstacles that can be obstacles, it is preferable that the obstacle determination unit 333 in the information processing apparatus 3 is configured so that each of the plurality of obstacles can be recognized separately. In particular, it is preferable that the obstacle determination unit 333 is configured to be able to separately recognize each of the plurality of obstacles because the predetermined area surrounded by each of the plurality of obstacles is learned in advance by machine learning.

Thirdly, for example, in the case of the present vehicle equipped with the measurement system 1, automatic driving may be performed for a part or all of the measured obstacles. For example, a braking operation for avoiding a collision or a steering operation may be considered. Further, the measured obstacle recognition status may be displayed on a monitor installed in the vehicle so that the driver of the vehicle can recognize it.

Fourth, in the above-described embodiment, the twin-lens image pickup device 2 including the first camera 21 and the second camera 22 is used, but the three-eye image pickup device 2 using three or more cameras is used. May be carried out. By increasing the number of cameras, the robustness related to the measurement by the measurement system 1 is improved, which is an advantageous effect.

Fifth, the image pickup device 2 and the information processing device 3 may be realized not as the measurement system 1 but as one device having these functions. Specific examples thereof include a three-dimensional measuring device, an image processing device, a projection display device, a three-dimensional simulator device, and the like.

Sixth, a different alignment method may be adopted instead of the phase-limited correlation method. For example, the image IM captured by the image pickup device 2 may be directly image-processed for positioning, or a sensor other than the image pickup device 2, for example, a laser displacement sensor, an infrared sensor, or the like may be used.

6. Conclusion As described above, according to the present embodiment, it is possible to implement the measurement system 1 which is used in a moving body and can measure the absolute speed of the moving body with high accuracy.

That is, "a measurement system used in a moving body traveling on a road surface, which includes an image pickup device and an information processing device, and the image pickup device can image a road surface or an obstacle included in an angle of view in time series. The information processing apparatus includes a communication unit, an IPM conversion unit, an obstacle determination unit, an empty area setting unit, and a measurement unit, and the communication unit is connected to the image pickup device. The image captured by the image pickup apparatus is configured to be receivable, and the IPM conversion unit is configured to perform back-perspective projection conversion of the image to generate an IPM image, wherein the IPM image is the road surface or the road surface. An image drawn so as to overlook a predetermined plane including an obstacle, the obstacle determination unit is configured to be able to determine the presence or absence or position of the obstacle in the IPM image, and the empty area setting unit determines. The empty area can be variably set based on the presence or absence or the position of the obstacle, and the empty area is a region in the IPM image that does not include the obstacle, and the measuring unit is a region. A measurement system configured to be able to measure the absolute speed of the moving object based on the empty area is provided.

Further, by using the measurement system 1, it is possible to implement a measurement method that is used in a moving body and can measure the absolute speed of the moving body with high accuracy.

That is, "a measurement method used in a moving body traveling on a road surface, which includes an image pickup step, an IPM conversion step, an obstacle determination step, an empty area setting step, and a measurement step. , The road surface or obstacle included in the angle of view is imaged in time series, and in the IPM conversion step, the image is subjected to back-perspective projection conversion to generate an IPM image, wherein the IPM image is the road surface or obstacle. An image drawn so as to give a bird's-eye view of a predetermined plane including an object. In the obstacle determination step, the presence or absence or position of the obstacle in the IPM image is determined, and in the empty area setting step, the determined obstacle is determined. An empty area is variably set based on the presence or absence or position of an object, where the empty area is a region in the IPM image that does not include the obstacle, and in the measurement step, the empty area is used as the basis for the empty area. A measurement method for measuring the absolute speed of a moving object "is provided.

Further, software for implementing the measurement system 1 used in the moving body and capable of measuring the absolute speed of the moving body with high accuracy as hardware can also be implemented as a program. Then, such a program may be provided as a non-temporary recording medium that can be read by a computer, may be provided for download from an external server, or the program may be started by an external computer. Then, so-called cloud computing, in which each function can be executed on the client terminal, may be implemented.

That is, "a measurement program in which a computer functions as an information processing device of a measurement system used in a moving body traveling on a road surface, and the information processing device includes a communication unit, an IPM conversion unit, and an obstacle determination. The communication unit includes a unit, an empty area setting unit, and a measurement unit, the communication unit is configured to be able to receive an image captured by an image pickup device, and the IPM conversion unit converts the image into a back-perspective projection conversion and IPM. The IPM image is configured to generate an image, and the IPM image is an image drawn so as to overlook a predetermined plane including the road surface or an obstacle, and the obstacle determination unit is the obstacle in the IPM image. The empty area setting unit is configured to be able to determine the presence / absence or position of an object, and the empty area setting unit is configured to be able to variably set an empty area based on the determined presence / absence or position of the obstacle. In the region of the IPM image that does not include the obstacle, the measuring unit is provided with a measurement program configured to be capable of measuring the absolute speed of the moving body based on the empty region.

Finally, various embodiments of the present invention have been described, but these are presented as examples and are not intended to limit the scope of the invention. The novel embodiment can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and the equivalent scope thereof.

1: Measurement system 2: Image pickup device 21: First camera 22: Second camera 3: Information processing device 30: Communication bus 31: Communication unit 32: Storage unit 33: Control unit 331: IPM conversion unit 332: histogram generation Unit 333: Obstacle determination unit 334: Empty area setting unit 335: Measurement unit D: Difference HG: histogram HG1: First histogram HG2: Second histogram IM: Image IM': IPM image IM1: First image IM1 ': First IPM image IM2: Second image IM2': Second IPM image O_C: Camera coordinate system O_W: World coordinate system R: Empty region π: Plane π_C: Camera image plane

Claims (10)

A measurement system used in moving objects traveling on the road surface.
Equipped with an image pickup device and an information processing device,
The image pickup device is configured to be able to image the road surface or obstacles included in the angle of view in chronological order.
The information processing device includes a communication unit, an IPM conversion unit, an obstacle determination unit, an empty area setting unit, and a measurement unit.
The communication unit
Connected to the image pickup device
It is configured to be able to receive the image captured by the image pickup device.
The IPM conversion unit is configured to generate an IPM image by performing a back-perspective projection conversion of the image, and the IPM image is drawn so as to overlook a predetermined plane including the road surface or an obstacle. In the image,
The obstacle determination unit is configured to be able to determine the presence or absence or position of the obstacle in the IPM image.
The empty area setting unit is configured to be able to variably set the empty area based on the presence or absence or position of the determined obstacle, and the empty area does not include the obstacle in the IPM image. In the area,
The measuring unit is a measuring system configured to be able to measure the absolute speed of the moving body based on the empty area. In the measurement system according to claim 1,
When there is an obstacle, the measuring unit is configured to be able to measure the relative speed of the obstacle with respect to the moving body based on the time-series change of the position.
Measurement system. In the measurement system according to claim 2,
The measuring unit is configured to be able to measure the absolute speed of the obstacle based on the absolute speed of the moving body and the relative speed of the obstacle.
Measurement system. In the measurement system according to any one of claims 1 to 3.
The empty area is set in a rectangular shape.
Measurement system. In the measurement system according to any one of claims 1 to 4.
The image pickup rate of the image pickup apparatus is 100 fps or more.
Measurement system. In the measurement system according to any one of claims 1 to 5.
The image pickup device is
It is a two-eye imaging device consisting of a first and a second camera.
The road surface or obstacles included in the angles of view of the first and second cameras are configured to be able to be imaged in chronological order as first and second images.
The IPM conversion unit performs back-perspective projection conversion of the first and second images to generate first and second IPM images, respectively.
The obstacle determination unit is configured to be able to determine the presence or absence or position of the obstacle based on the difference between the first and second IPM images.
Measurement system. In the measurement system according to any one of claims 1 to 6.
The empty area setting unit is configured to be able to set the empty area so that the number of pixels is large in a range not including the obstacle.
Measurement system. In the measurement system according to any one of claims 1 to 7.
The empty area setting unit is configured so that the empty area can be variably set for each frame of the IPM image.
Measurement system. It is a measurement method used for moving objects traveling on the road surface.
It includes an imaging step, an IPM conversion step, an obstacle determination step, an empty area setting step, and a measurement step.
In the imaging step, the road surface or obstacles included in the angle of view are imaged in chronological order.
In the IPM conversion step, the image is subjected to reverse perspective projection conversion to generate an IPM image, wherein the IPM image is an image drawn so as to overlook a predetermined plane including the road surface or an obstacle.
In the obstacle determination step, the presence or absence or position of the obstacle in the IPM image is determined.
In the empty area setting step, an empty area is variably set based on the presence or absence or position of the determined obstacle, and the empty area is a region in the IPM image that does not include the obstacle.
In the measurement step, the absolute velocity of the moving body is measured based on the empty area.
Measurement method. A measurement program that allows a computer to function as an information processing device for a measurement system used in a moving object traveling on a road surface.
The information processing device includes a communication unit, an IPM conversion unit, an obstacle determination unit, an empty area setting unit, and a measurement unit.
The communication unit is configured to be able to receive an image captured by an image pickup device.
The IPM conversion unit is configured to generate an IPM image by performing a back-perspective projection conversion of the image, and the IPM image is drawn so as to overlook a predetermined plane including the road surface or an obstacle. In the image,
The obstacle determination unit is configured to be able to determine the presence or absence or position of the obstacle in the IPM image.
The empty area setting unit is configured to be able to variably set the empty area based on the presence or absence or position of the determined obstacle, and the empty area does not include the obstacle in the IPM image. In the area,
The measuring unit is a measurement program configured to be capable of measuring the absolute speed of the moving body based on the empty area.

Images