JP2012118885A - Movement amount estimation device and movement amount estimation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device or a method for accurately estimating a movement amount of a moving object.SOLUTION: A movement amount estimation device comprises a projection unit, an extraction unit, and an estimation unit. The projection unit respectively generates road surface projection images for respective images obtained by plural cameras attached to a moving object. The extraction unit extracts image patterns matching each other between road surface projection images from a common visual field area in which visual fields of plural cameras overlap each other in the plural road surface projection images generated by the projection unit. The estimation unit estimates a movement amount of the moving object based on temporal variation in the image patterns extracted by the extraction unit.

Description

  The present invention relates to a movement amount estimation apparatus, a movement amount estimation method, and a movement amount estimation program for estimating a movement amount of a moving object.

  2. Description of the Related Art In recent years, techniques for detecting the amount of movement of a vehicle (movement distance and movement direction) have been studied for vehicle driving assistance or driving control. The amount of movement of the vehicle can be detected using, for example, GPS. However, the method using GPS or the like has insufficient detection accuracy depending on the application, and is not suitable for real-time control of vehicle travel. For this reason, a technique has been proposed in which a movement amount estimation device mounted on a vehicle estimates the movement amount of the host vehicle.

  For example, a method for estimating the amount of movement of a vehicle using an image of a vehicle-mounted camera has been proposed. In this method, the in-vehicle camera captures a moving image including a road surface while the vehicle is traveling. Further, the movement amount estimation device generates a road surface projection image at predetermined time intervals using the video from the in-vehicle camera. The road surface projection image is generated by projecting the video from the in-vehicle camera onto a plane corresponding to the road surface. Then, the movement amount estimation device estimates the movement amount of the vehicle by detecting the time change of the road surface projection image. In addition, the method of estimating the moving amount | distance of a vehicle using the image | video of a vehicle-mounted camera is described in the nonpatent literature 1, for example.

"Development of a monocular ranging verification system using an in-vehicle camera", Shuhei Takimoto, Takaaki Ito, SEI Technical Review No. 169, pp. 82-87, July 2006

  In the method of estimating the movement amount of the vehicle based on the time change of the road projection image, the movement amount cannot be estimated correctly when a three-dimensional object is captured in the camera image. However, in the prior art, it has not been considered whether or not the road surface projection image includes a three-dimensional object image. For this reason, in the prior art, the amount of movement of the vehicle may not be correctly estimated. This problem is not limited to estimating the amount of movement of the vehicle, but may occur when estimating the amount of movement of an arbitrary moving body.

  An object of the present invention is to provide an apparatus or a method for accurately estimating the amount of movement of a moving body.

  A movement amount estimation apparatus according to one aspect of the present invention includes a projection unit that generates a road surface projection image for each image obtained by a plurality of cameras attached to a moving body, and a plurality of road surface projections generated by the projection unit. Based on a common field region where the fields of view of the plurality of cameras overlap in an image, an extraction unit that extracts mutually matching image patterns between road surface projection images, and based on temporal changes of the image pattern extracted by the extraction unit, An estimation unit configured to estimate a moving amount of the moving body.

  According to the above-described aspect, it is possible to accurately estimate the moving amount of the moving body.

It is a figure shown about the camera attached to a moving body and its visual field. It is a functional block diagram of the movement amount estimation apparatus of the first embodiment. It is a figure explaining a coordinate system. It is a figure explaining the area | region which produces | generates a road surface projection image. It is a figure explaining the conversion of the coordinate by projection. It is a figure explaining the interpolation of a pixel value. It is a figure which shows an example of the environment around a moving body. It is a figure explaining the road surface projection image of the image | photographed solid object. It is a figure which shows the road surface projection image produced | generated from the video data obtained in the environment shown in FIG. It is a figure which shows the state which overlap | superposed two road surface projection images. It is a figure which shows the extracted road surface pattern. It is a figure explaining the process which extracts the image pattern which mutually matches between road surface projection images. It is a figure explaining a road surface pattern image. It is a figure explaining an integrated road surface projection image. It is a flowchart which shows the outline of a process of a movement amount extraction part. It is a flowchart which shows the process of step S1 of FIG. It is a flowchart which shows the process of step S14 of FIG. It is a figure explaining calculation of a collation score. It is a flowchart which shows the process of step S2 of FIG. It is a flowchart which shows the process of step S44 of FIG. It is a functional block diagram of the movement amount estimation apparatus of the second embodiment. It is a flowchart which shows the process of step S1 performed in 2nd Embodiment. It is a flowchart which shows the process of step S2 performed by 2nd Embodiment. It is a figure explaining a near integrated projection image. It is a flowchart which shows the process of the movement amount estimation part of 3rd Embodiment. It is a flowchart which shows the process of the movement amount estimation part of 4th Embodiment. It is a figure which shows an example of the hardware constitutions of a movement amount estimation apparatus.

  In the movement amount estimation method of the embodiment, the movement amount of a moving body to which a plurality of cameras are attached is estimated. At this time, the movement amount estimation apparatus estimates the movement amount of the moving body by using images respectively obtained by a plurality of cameras.

  The “movement amount” of the moving body is represented by a translation movement amount (movement distance and movement direction) and a rotation amount (yaw rotation). That is, the movement amount estimation device estimates the translational movement amount and the rotation amount of the moving body. However, the movement amount estimation device may estimate either the translational movement amount or the rotation amount of the moving body. In addition, although the moving body of the object which estimates movement amount is not specifically limited, For example, it is a conveyance trolley etc. which are used in vehicles, a factory, etc. including a passenger car. The moving body is assumed to move (for example, travel) on the road surface.

  FIG. 1 shows a camera attached to a moving body and its field of view. As shown in FIG. 1, cameras 2 a to 2 d are attached to the moving body 1. The camera 2a is attached to the front part of the moving body 1 and acquires an image in front of the moving body 1. The camera 2b is attached to the rear part of the moving body 1 and acquires an image behind the moving body 1. Furthermore, the cameras 2c and 2d are attached to the right side and the left side of the moving body 1, respectively, and acquire right and left images of the moving body 1. The traveling direction of the moving body 1 is referred to as “front”.

  The camera 2a, camera 2b, camera 2c, and camera 2d have visual field areas 3a, 3b, 3c, and 3d, respectively. The visual field area is an area where an image is acquired by a corresponding camera. Here, each of the cameras 2a to 2d is a wide-angle camera and has a large viewing angle. In the example shown in FIG. 1, the viewing angles of the cameras 2a to 2d are about 180 degrees.

  For this reason, the visual field regions 3a, 3b, 3c, and 3d partially overlap each other. For example, the common visual field area 4e is an area where the visual field area 3a of the camera 2a and the visual field area 3c of the camera 2c overlap each other. That is, the video of the common visual field 4e is acquired by both the camera 2a and the camera 2c. Similarly, the common visual field region 4f is a region where the visual field region 3b of the camera 2b and the visual field region 3c of the camera 2c overlap each other, and the common visual field region 4g is the same as the visual field region 3a of the camera 2a and the visual field region 3d of the camera 2d. The common visual field region 4h is a region where the visual field region 3b of the camera 2b and the visual field region 3d of the camera 2d overlap each other. In addition, each common visual field area | region 4e-4h is represented by the oblique line area | region in FIG.

  The movement amount estimation method of the embodiment estimates the movement amount of the moving body 1 using at least one video of the common visual field regions 4e to 4h. In addition, the movement amount estimation method of the embodiment is not applied only when four cameras are attached to the moving body 1, and when two or more cameras are attached to the moving body 1. Applicable.

<< First Embodiment >>
FIG. 2 is a functional block diagram of the movement amount estimation apparatus according to the first embodiment. The movement amount estimation apparatus 10 of the first embodiment includes an image input unit 11, a synchronization control unit 12, a road surface projection unit 13, a road surface pattern extraction unit 14, and a movement amount estimation unit 15. In addition, video data is input from the cameras 2a to 2d to the movement amount estimation apparatus 10, respectively. Further, the movement amount estimation apparatus 10 is connected to the video data DB 16. The road surface projection unit 13, the road surface pattern extraction unit 14, and the movement amount estimation unit 15 are examples of a projection unit, an extraction unit, and an estimation unit according to the invention, respectively.

  The cameras 2a to 2d are attached to the moving body 1 as shown in FIG. 1, and each acquire video data. The video data is not particularly limited, but is, for example, moving image data of 30 frames / second.

  The image input unit 11 captures video data of the cameras 2 a to 2 d according to the synchronization signal generated by the synchronization control unit 12. Here, the synchronization control unit 12 includes a timer or a counter, for example, in accordance with the video synchronization signal of the camera, and outputs the synchronization signal at a predetermined time interval. In the first embodiment, as an example, the mode in which the video signal is input from the camera 2a to the synchronization control unit 12 on the assumption that it is synchronized with the video synchronization signal from the camera 2a has been described. The camera to do may be other cameras 2b to 2d. Further, when the synchronization signal is output by the timer or counter provided in the synchronization control unit 12 without using the video synchronization signal from the camera, it is not necessary to use the input of the video synchronization signal from the camera. The image input unit 11 generates digital information including the pixel value of each pixel in the frame from the video data of the cameras 2a to 2d every time the synchronization signal is received. That is, digital image information having pixel values for each element of the two-dimensional array representing the image frame is generated. The pixel value is represented by a luminance value and / or color information.

  Note that the synchronization control unit 12 does not necessarily output the synchronization signal at a constant time interval. For example, the synchronization control unit 12 may determine the time interval for generating the synchronization signal according to the moving speed of the moving body 1. In this case, the synchronization control unit 12 generates a synchronization signal at a short time interval when the moving speed of the moving body 1 is high, and generates a synchronization signal at a long time interval when the moving speed of the moving body 1 is low. Good.

  The road surface projection unit 13 generates a road surface projection image by projecting the digital image input from the image input unit 11 onto the traveling road surface of the moving body 1. That is, the road surface projection unit 13 converts the video data of the cameras 2a to 2d into road surface projection images. The road surface projection unit 13 records the generated road surface projection image in the video data DB 16. In the following description, the traveling road surface of the moving body 1 is simply referred to as “road surface”.

  The road surface pattern extraction unit 14 extracts image patterns that coincide with each other between road surface projection images having a common field of view in a common field region of a plurality of road surface projection images generated by the road surface projection unit 13. The extracted image pattern represents a pattern on the road surface (hereinafter, road surface pattern). Then, the road surface pattern extraction unit 14 records a road surface pattern image including the extracted road surface pattern in the video data DB 16.

  The video data DB 16 is, for example, a semiconductor memory, and stores road surface projection images of the cameras 2a to 2d and road surface pattern images of the common visual field regions. Here, the video data DB 16 holds at least the immediately preceding road surface projection image and road surface pattern image in addition to the latest road surface projection image and road surface pattern image. In addition, the video data DB 16 may store an integrated road surface projection image to be described later.

  The movement amount estimation unit 15 estimates the movement amount of the moving body 1 based on the latest road surface pattern image recorded in the video data DB 16 and the previous road surface pattern image. That is, the movement amount estimation unit 15 estimates the movement amount of the moving body 1 based on the time change of the road surface pattern image. The movement amount estimation unit 15 may estimate the movement amount of the moving body 1 using not only a road surface pattern image but also a road surface projection image.

  Thus, in the movement amount estimation apparatus 10, the road surface projection unit 13 generates a road surface projection image from the video data of the cameras 2a to 2d. Further, the road surface pattern extraction unit 14 extracts image patterns that coincide with each other between the road surface projection images having a common field of view in the common field region of the plurality of road surface projection images. At this time, as will be described in detail later, the three-dimensional object is not extracted from the common visual field region, but the road surface pattern formed on the road surface is extracted. And the movement amount estimation part 15 estimates the movement amount of the mobile body 1 based on the time change of a road surface pattern image. That is, in the process of estimating the movement amount of the moving body 1, the influence of the three-dimensional object in the video data is removed, so that the movement amount estimation apparatus 10 can estimate the movement amount of the moving body 1 with high accuracy.

  Next, operations of the road surface projection unit 13, the road surface pattern extraction unit 14, and the movement amount estimation unit 15 will be specifically described. In the following description, the orthogonal coordinate system shown in FIG. 3 is used. In this coordinate system, the traveling direction of the moving body 1 is defined as the z direction. In addition, the x direction is defined in a direction orthogonal to the z direction and in the lateral direction of the moving body 1 (rightward with respect to the traveling direction in FIG. 3). Furthermore, the y direction is defined in the direction perpendicular to the z direction and in the height direction of the moving body 1 (downward in FIG. 3).

<Operation of Road Projection Unit 13>
The road surface projection unit 13 generates a road surface projection image from the video data of the cameras 2 a to 2 d at the timing when the synchronization signal is received from the synchronization control unit 12. At this time, it is assumed that a region for generating a road surface projection image is determined in advance with reference to the position of the moving body 1. That is, the road surface projection unit 13 generates a road surface projection image from the input video data for an area surrounded by a broken line in FIG. However, even within the area surrounded by the broken line shown in FIG. 4, no video data is generated for the area that is not the visual field area of the cameras 2a to 2d (for example, the area directly below the moving body 1). Is not generated.

  In the example shown in FIG. 4, the road surface projection unit 13 generates road surface projection images of the regions R1, R2, and R3 from the video data of the visual field region 3a obtained by the camera 2a. Similarly, the road surface projection unit 13 generates road surface projection images of the regions R3, R4, and R5 from the video data of the visual field region 3c obtained by the camera 2c, and generates the region R5, from the video data of the visual field region 3b obtained by the camera 2b. Road surface projection images of R6 and R7 are generated, and road surface projection images of regions R7, R8, and R1 are generated from video data of the visual field region 3d obtained by the camera 2d.

  At this time, in each common visual field area | region 4e-4h, a road surface projection image is each produced | generated overlappingly. For example, in the common visual field area 4e of the cameras 2a and 2c, a road surface projection image of the area R3 is generated from the video data of the camera 2a, and a road surface projection image of the area R3 is also generated from the video data of the camera 2c.

As described above, the road surface projection unit 13 generates a road surface projection image from the video data of the cameras 2a to 2d. Here, the process of generating the road surface projection image from the video data of the cameras 2a to 2d corresponds to the process of specifying the corresponding pixel of the input image for each pixel of the generated road surface projection image. Therefore, in the following, it is assumed that the pixel (sx, sy) of the input video is projected onto the pixel (u, v) of the road surface projection image, as shown in FIG. In this case, the pixel value Q _i (u, v) of the pixel (u, v) of the road surface projection image corresponding to the i (i = 2a, 2b, 2c, 2d) -th camera image is expressed by the following equation (1): Given in.
Q _i (u, v) = INTERP (I _i , (sx, sy)) (1)
“INTERP ()” represents an interpolation operation. “I _i ” represents an image from the i-th camera.

Here, the size of the road projection image is PW × PH pixels. For example, in the example shown in FIG. 4, the area surrounded by the wavy line is represented by an image of PW × PH pixels. PW and PH are the width and height of the road surface projection image, respectively, and are expressed by the following equations.
PW = 2 × ww + 1
PH = 2 × hh + 1
ww and hh are expressed by the following equations.
ww = MX / rx
hh = MY / ry
MX and MY (millimeters) respectively correspond to a half of the width and height of the region for generating the road surface projection image. rx, ry (millimeter / pixel) represents the projection size per pixel.

Further, the arrangement of the i-th camera is represented by the following equation. T _i represents a position where the i-th camera is attached. R _i represents the optical axis direction of the i-th camera.

Under the above conditions, the coordinates of an arbitrary target photographed by the i-th camera are expressed by the following equation (2).
L _x = r11 {(u−ww) rx−T ⁱ _x } + r12 {− (v−hh) ry−T ⁱ _y } + r13 (−T ⁱ _z )
L _y = r21 {(u−ww) rx−T ⁱ _x } + r22 {− (v−hh) ry−T ⁱ _y } + r23 (−T ⁱ _z )
_{L z = r31 {(u-} ww) rx-T i x} + r32 {- (v-hh) ry-T i y} + r33 (-T i z) ··· (2)

  Next, the following definition is introduced from the above equation (2).

Then, the pixel (sx, sy) of the input image corresponding to the pixel (u, v) of the road surface projection image is obtained by the following equation (3).
sx = fx × p + cx
sy = fy × p + cy (3)
Note that fx and fy represent the focal lengths in the horizontal and vertical directions of the camera, respectively. Also, cx and cy represent the image center position of the input video data as shown in FIG.

In the above formulas (2) and (3), T _i and R _i are determined according to the arrangement of the cameras 2a to 2d. Further, fx and fy are specifications of the cameras 2a to 2d and are constants. Here, information relating to the specifications of the camera, including information (T _i , R _i ) indicating the arrangement of the cameras 2a to 2d and focal lengths (fx, fy) of the cameras 2a to 2d, is a movement amount estimation device. 10 is stored in an accessible memory. The road surface projection unit 13 can refer to the camera information as shown in FIG. ww, hh, rx, and ry are constants set in advance, and these pieces of information are also stored in a memory accessible by the movement amount estimation apparatus 10. Therefore, the road surface projection unit 13 can uniquely calculate the corresponding coordinates (sx, sy) in the input video for the pixel (u, v) of the road surface projection image.

However, the coordinates (sx, sy) in the input image specified for the pixel (u, v) of the road projection image do not necessarily indicate one pixel. For example, in the example shown in FIG. 6, the coordinates (51.7, 45.6) in the input video are calculated for the pixel (u, v) of the road surface projection image. In this case, the pixel value Q _i (u, v) of the pixel (u, v) of the road projection image is calculated by interpolation based on the pixel values of the four pixels P1 to P4 adjacent to the coordinates (sx, sy). The The interpolation calculation is realized by, for example, a weighted average of the pixel values of the pixels P1 to P4.

As described above, the road surface projection unit 13 calculates the pixel value of each pixel of the road surface projection image based on the video data of the cameras 2a to 2d. Thereby, a road surface projection image is generated.
The road surface projection unit 13 may perform differentiation processing such as Sobel calculation on the road surface projection image generated as described above. When the differentiation process is executed on the image, the contour of the pattern is emphasized, and the influence due to the difference in the characteristics of the cameras 2a to 2d or the temporal change in the gray value of the video is suppressed. Therefore, in the following description, an image after further performing a differentiation process on the image obtained by the above-described projection process will be referred to as a road surface projection image.

  The road surface projection unit 13 records the road surface projection image generated as described above in the video data DB 16. That is, every time a synchronization signal is output from the synchronization control unit 12, four road surface projection images generated from the video data of the cameras 2a to 2d are recorded in the video data DB 16.

<Operation of the road surface pattern extraction unit 14>
The road surface pattern extraction unit 14 extracts, as a road surface pattern, image patterns that match each other between road surface projection images having a common field of view in a common field region of a plurality of road surface projection images generated by the road surface projection unit 13. Hereinafter, as one example, the common visual field region 4e of the cameras 2a and 2c shown in FIG. 1 or 4 will be described. The method for extracting the road surface pattern is the same for the other common visual field regions 4f to 4h.

  FIG. 7 is a diagram illustrating an example of the environment around the moving body 1. Here, FIG. 7 shows a state when the moving body 1 and its periphery are viewed from above. In the example illustrated in FIG. 7, the road surface pattern 21 and the three-dimensional object 22 exist in the right diagonal front of the moving body 1. The road surface pattern 21 is a pattern formed on the road surface, for example, a pattern drawn on a road surface such as a pedestrian crossing or a road sign. It is assumed that the road surface pattern 21 does not have a “height” with respect to the road surface. The three-dimensional object 22 is an object having a “height” with respect to the road surface. In this example, the three-dimensional object 22 is a plate-like object (for example, a standing signboard) that is set up perpendicular to the road surface.

  As described above, the cameras 2a and 2c are attached to the moving body 1. At the time shown in FIG. 7, the road surface pattern 21 and the three-dimensional object 22 are located within the visual field range of the camera 2a and also within the visual field range of the camera 2c. That is, the road surface pattern 21 and the three-dimensional object 22 are located in the common visual field region 4e shown in FIG. 1 or FIG.

  FIG. 8 is a diagram for explaining a road surface projection image of a captured three-dimensional object. Here, it is assumed that the three-dimensional object 22 is photographed by the camera 2a. In FIG. 8, the road surface pattern 21 is omitted. Here, as shown in FIG. 8A, the three-dimensional object 22 is a plate-like object that stands upright with respect to the road surface. In this case, when the video data obtained by the camera 2a is projected onto the road surface, a road surface projection image shown in FIG. 8B is generated. That is, the “height” of the three-dimensional object in the video data is represented as “depth” in the road projection image. Therefore, in the road surface projection image shown in FIG. 8B, the three-dimensional object 22 is an image pattern that spreads radially from the straight line 23 where the three-dimensional object 22 and the road surface are in contact with each other, as indicated by the hatched area. Is represented as Furthermore, when a differentiation process is performed on the image shown in FIG. 8B, a road surface projection image with an emphasized contour is obtained as shown in FIG. 8C.

  FIG. 9A shows a road projection image generated from the video data obtained by the camera 2a in FIG. FIG. 9B shows a road projection image generated from the video data obtained by the camera 2c in FIG. In addition, the road surface projection image shown to Fig.9 (a) and FIG.9 (b) represents the state after a differentiation process was performed. Therefore, only the contour of the road surface pattern 21 shown in FIG. 7 appears in the road surface projection image.

  In the road projection image, the three-dimensional object 22 is represented as an image pattern that spreads radially from the position where the three-dimensional object 22 is placed as the origin, as described with reference to FIG. For this reason, in the road projection image of the camera 2a, the three-dimensional object 22 is represented by image patterns 24a, 25a, and 26a as shown in FIG. Similarly, in the road surface projection image of the camera 2c, the three-dimensional object 22 is represented by image patterns 24c, 25c, and 26c as shown in FIG. 9B. Thus, the road surface projection image of the three-dimensional object 22 is represented as a different image pattern depending on the position of the camera.

  The road surface pattern extraction unit 14 compares the road surface projection image of the camera 2a and the road surface projection image of the camera 2c, and extracts image patterns that coincide with each other between the road surface projection images as a road surface pattern. That is, the road surface pattern extraction unit 14 compares the road surface projection images shown in FIGS. 9 (a) and 9 (b).

  FIG. 10 shows a state in which the road surface projection image of the camera 2a shown in FIG. 9A and the road surface projection image of the camera 2c shown in FIG. In this case, the road surface pattern 21 matches between the two road surface projection images. However, the three-dimensional object 22 is represented by image patterns 24a, 25a, and 26a in the road surface projection image of the camera 2a, and is represented by image patterns 24c, 25c, and 26c in the road surface projection image of the camera 2c. Here, the image patterns 24a and 24c coincide with each other between the two road surface projection images. However, the image patterns 25a and 25c are different from each other, and the image patterns 26a and 26c are also different from each other.

  The road surface pattern extraction unit 14 extracts image patterns that match each other between the two road surface projection images as a road surface pattern. Then, as shown in FIG. 11, an image pattern corresponding to the road surface pattern 21 is extracted. At this time, image patterns 24a and 24c are also extracted. The image patterns 24a and 24c correspond to the boundary line between the three-dimensional object 22 and the road surface. As a result, an image in which the three-dimensional object is removed from the road surface projection image is generated. Hereinafter, the image generated in this way may be referred to as a road surface pattern image.

  FIG. 12 is a diagram illustrating a process of extracting image patterns that match each other between road surface projection images. The road surface pattern extraction unit 14 divides two road surface projection images to be compared (or regions where the two road surface projection images overlap each other) into a plurality of small regions (hereinafter referred to as image blocks). In the example shown in FIG. 12, the road surface projection image of the camera 2a is divided into image blocks A1 to A9, and the road surface projection image of the camera 2c is divided into image blocks C1 to C9. The image blocks A1 to A9 and the image blocks C1 to C9 represent areas corresponding to each other. Note that the size of the image block is not particularly limited, and can be, for example, 4 × 4 pixels, 8 × 8 pixels, 16 × 16 pixels, or the like.

  The road surface pattern extraction unit 14 compares the image patterns of the corresponding image blocks between the two road surface projection images. For example, the road surface pattern extraction unit 14 compares image patterns using SAD (Sum of Absolute Difference). That is, the road surface pattern extraction unit 14 calculates the absolute difference of the pixel values of the corresponding pixels between the two corresponding image blocks, and obtains the sum of the absolute differences. If the sum of the absolute differences is smaller than a predetermined threshold value, it is determined that the image patterns of the two image blocks match each other. On the other hand, if the sum of absolute differences is equal to or greater than a predetermined threshold, it is determined that the image patterns of the two image blocks do not match each other.

  In FIG. 12, first, the image blocks A1 and C1 are compared. Here, the image A1 of the image block is “Δ”, and the image C1 of the image block is also “Δ”. Therefore, it is determined that the image patterns of the image blocks A1 and C1 match each other. Then, the road surface pattern extraction unit 14 outputs image data of one of the image blocks A1 and C1 as the image data of the image block T1 corresponding to the road surface pattern image. Here, since the image patterns of the image blocks A1 and C1 match each other, the surface pattern extraction unit 14 may output any image data of the image blocks A1 and C1.

  Subsequently, the image blocks A2 and C2 are compared. Here, the image A2 of the image block is “◯”, and the image C2 of the image block is “☆”. Therefore, it is determined that the image patterns of the image blocks A2 and C2 do not match each other. Then, the road surface pattern extraction unit 14 outputs “zero” as the image data of the image block T2 corresponding to the road surface pattern image. “Zero” is image data in which the pixel values of all the pixels in the image block are zero.

  Similarly, image blocks T3 to T9 of the road surface pattern image are generated. When the corresponding image blocks of the two road surface projection images are both “no pattern”, the image patterns can match each other. For example, it is determined that the image patterns of the image blocks A7 and C7 match each other. In this case, the road surface pattern extraction unit 14 outputs image data of one of the image blocks A7 and C7 as image data of the image block T7 corresponding to the road surface pattern image.

  As described above, the road surface pattern extraction unit 14 outputs “zero” when the image patterns of the corresponding image blocks are different between the road surface projection images. Thereby, the road surface pattern image from which the three-dimensional object is removed is generated.

  The road surface pattern extraction unit 14 generates the above road surface pattern image for each common visual field region. As a result, the road surface pattern extracting unit 14 obtains the entire road surface pattern image shown in FIG. The entire road surface pattern image has road surface pattern images generated by the method shown in FIG. 12 in the common visual field regions R1, R3, R5, and R7. Further, “zero” is given to each pixel of the other regions R2, R4, R6, R8, and R9 of the entire road surface pattern image. In the following description, the entire road pattern image may be simply referred to as a “road pattern image”.

  Further, the road surface pattern extraction unit 14 generates an integrated road surface projection image shown in FIG. The integrated road surface projection image has road surface pattern images generated by the method shown in FIG. 12 in the common visual field regions R1, R3, R5, and R7. In addition, road surface projection images of corresponding cameras are assigned to the single visual field regions R2, R4, R6, and R8. For example, in the single visual field region R2, video data is generated only by the camera 2a. In this case, a road surface projection image generated from the video data of the camera 2a is assigned to the single visual field region R2. Note that “zero” is given to each pixel in the region R9 that is not in the visual field range of any of the cameras 2a to 2d.

  In this way, the road surface pattern extraction unit 14 generates a road surface pattern image and an integrated road surface projection image based on the road surface projection image obtained by the road surface projection unit 13. In addition, the road surface pattern extraction unit 14 records the generated road surface pattern image and the integrated road surface projection image in the video data DB 16. The road surface pattern extraction unit 14 generates a road surface pattern image and an integrated road surface projection image at predetermined time intervals, for example, and records them in the video data DB 16. The video data DB 16 holds the past road surface pattern image and the integrated road surface projection image (at least the previous road surface pattern image and the integrated road surface projection image) in addition to the latest road surface pattern image and the integrated road surface projection image.

<Operation of Movement Amount Extracting Unit 15>
The movement amount extraction unit 15 estimates the movement amount of the moving body 1 based on the time change of the road surface pattern image generated by the road surface pattern extraction unit 14. FIG. 15 is a flowchart showing an outline of processing of the movement amount extraction unit 15.

  In step S <b> 1, the movement amount extraction unit 15 estimates the movement amount of the moving body 1 based on the time change of the road surface pattern extracted from the common visual field region. Subsequently, in step S2, the movement amount estimation unit 15 estimates the movement amount of the moving body 1 with higher accuracy using the integrated road surface projection image and the estimation result of step S1. That is, the movement amount estimation unit 15 corrects the movement amount obtained in step S1 using the integrated road surface projection image.

  FIG. 16 is a flowchart showing the process of step S1 shown in FIG. The processing of this flowchart is executed by the movement amount estimation unit 15. Further, the movement amount estimation unit 15 estimates the movement amount of the moving body 1 using the road surface pattern image recorded in the video data DB 16. Here, the movement amount estimation unit 15 displays the road surface pattern image C (t1) generated based on the video data at time t1 and the road surface pattern image C (t2) generated based on the video data at time t2. It shall be acquired from data DB16. Note that the time t2 is the current time in this example. The time t1 is a time in the past by a predetermined time with respect to the time t2. The time interval between the time t1 and the time t2 is not particularly limited, and is, for example, the frame interval (1/30 seconds) of the cameras 2a to 2d, or an integer multiple thereof. And the movement amount estimation part 15 estimates the movement amount of the mobile body 1 between the time t1 and the time t2.

  In step S11, the movement amount estimation unit 15 initializes the variable Mmax to zero. The variable Mmax represents the maximum value of the matching degree (or correlation) between the image at time t1 and the image at time t2.

  In step S12, the movement amount estimating unit 15 initializes the rotation angle a to “−A”. The rotation angle a is a variable representing the amount of rotation of plane rotation (or yaw rotation) that is one element of the amount of movement of the moving body 1.

  In step S13, the movement amount estimation unit 15 generates a rotating road surface pattern image D (t1) by rotationally converting the road surface pattern image C (t1) at the previous time by an angle a. Note that the rotation conversion of the image is realized, for example, by calculating the coordinates of each pixel using a rotation matrix.

  In step S14, the movement amount estimation unit 15 translates between the rotating road surface pattern image D (t1) obtained in step S13 and the road surface pattern image C (t2) at the current time to provide the maximum matching degree Mmax. (DXmax, DYmax) is specified. The process of step S14 will be described in detail later.

  In step S15, the movement amount estimation unit 15 adds a predetermined angle Δa determined in advance to the rotation angle a. The angle Δa represents an angle step for scanning the rotation angle a from −A to + A. In step S <b> 16, the movement amount estimation unit 15 determines whether the rotation angle a is equal to or less than A. If the rotation angle a is A or less, the process of the movement amount estimation unit 15 returns to step S13.

  In this way, the movement amount estimation unit 15 scans the rotation angle a of the rotating road surface pattern image D (t1) from −A to + A, while rotating the road surface pattern image D (t1) and the road surface pattern image C (t2). The translational displacement (DXmax, DYmax) that gives the maximum matching degree Mmax is specified. When the rotation angle a becomes larger than A (step S16: No), the movement amount estimation unit 15 gives a translational displacement (DXmax) that gives the maximum matching degree Mmax as the movement amount estimation result of the moving body 1 in step S17. , DYmax) and rotation angle (Amax). Note that the output of step S17 is an estimated value of the moving amount of the moving body 1 from time t1 to time t2.

Next, the process shown in step S14 will be described in detail. Here, the following variables are used:
dx: a displacement amount in the X direction of the rotating road surface pattern image D (t1) dy: a displacement amount in the Y direction of the rotating road surface pattern image D (t1) mmax: a matching score in the processing loop of steps 22 to S30 Dxmax representing the maximum value of Sc: dymax representing the amount of displacement in the X direction in which the matching score Sc is maximum in the processing loop of steps 22 to S30: the matching score Sc is maximum in the processing loop of steps 22 to S30 Mmax representing the amount of displacement in the Y direction: DXmax representing the maximum value of the collation score Sc in the process of scanning the rotation angle a from -A to the current angle: DYmax representing the amount of displacement in the X direction giving Mmax: Amax representing the amount of displacement in the Y direction giving Mmax: Representing the rotation angle giving Mmax

  FIG. 17 is a flowchart showing the process of step S14 of FIG. Note that the processing of the flowchart shown in FIG. 17 is executed for a given rotation angle a in the procedure for scanning the rotation angle a of the road surface pattern image D (t1) from −A to + A.

  In step S21, the movement amount estimation unit 15 initializes the variable mmax to zero. In step S <b> 22, the movement amount estimation unit 15 initializes the displacement amount dy to “−DY”. In step S <b> 23, the movement amount estimation unit 15 initializes the displacement amount dx to “−DX”.

  In step S24, the movement amount estimation unit 15 shifts the coordinates of the rotating road surface pattern image D (t1) according to the displacement amount dx and the displacement amount dy. That is, the movement amount estimation unit 15 shifts the rotating road surface pattern image D (t1) by (dx, dy). Then, the movement amount estimation unit 15 performs image matching between the road pattern image C (t2) and the rotating road pattern image D (t1) shifted by (dx, dy) to calculate a matching score Sc. .

The matching score Sc is obtained by executing the following processing on a region where the road surface pattern image C (t2) and the rotating road surface pattern image D (t1) shifted by (dx, dy) overlap each other (the hatched region shown in FIG. 18). Is calculated by
S = 0, N = 0
IF (C (x, y)! = 0 && D (x + dx, y + dy)! = 0) THEN
S = S + {C (x, y) −D (x + dx, y + dy)} absolute value
N = N + 1
ENDIF
Sc = LV− (S / N)

  That is, a variable S that represents the cumulative addition value of the absolute difference values of pixel values and a variable N that counts the number of pixels in the overlapping region are initialized. Subsequently, for each pixel in the overlapping region, the pixel value C (x, y) of the road pattern image C (t2) is not zero, and the pixel value D (x + dx) of the rotating road pattern image D (t1). , y + dy) is not zero, the absolute difference between the pixel value C (x, y) and the pixel value D (x + dx, y + dy) is cumulatively added. Further, the collation score Sc is obtained by subtracting the absolute difference value of the pixel value per pixel from the constant LV. The constant LV is determined such that the higher the correlation between the two images, the greater the collation score Sc. For example, when the pixel value is represented by 0 to 255, “LV> 255”, and as an example, “LV = 300”.

Returning to FIG. In step S25, the movement amount estimation unit 15 determines whether or not the matching score Sc calculated in step S24 is larger than the variable mmax. If the collation score Sc is larger than the variable mmax, the movement amount estimating unit 15 updates the variable mmax, the variable dxmax, and the variable dymax in step S26 as follows. If the matching score Sc is equal to or less than the variable mmax, step S26 is skipped.
mmax ← Sc
dxmax ← dx
dymax ← dy
That is, the newly calculated matching score Sc is held as a variable mmax representing the maximum value of the matching score Sc up to the present time. Further, the displacement amount that gives the maximum value of the matching score Sc is held as variables dxmax and dymax.

  In step S27, the movement amount estimation unit 15 adds Δdx to the displacement amount dx. Here, Δdx represents a displacement amount step for scanning the displacement amount dx in the X direction from −DX to + DX. Then, in step S28, the movement amount estimation unit 15 determines whether or not the displacement amount dx updated in step S27 is equal to or less than DX. If the variable amount dx is less than or equal to DX, the process of the movement amount estimation unit 15 returns to step S24.

  If the variable amount dx exceeds DX, the movement amount estimation unit 15 adds Δdy to the displacement amount dy in step S29. Here, Δdy represents a displacement amount step for scanning the displacement amount dy in the Y direction from −DY to + DY. Then, in step S30, the movement amount estimation unit 15 determines whether or not the displacement amount dy updated in step S29 is equal to or less than DY. If the variable amount dy is less than or equal to DY, the process of the movement amount estimation unit 15 returns to step S23.

If the variable amount dy exceeds DY, the movement amount estimation unit 15 compares the variable mmax with the variable Mmax in step S31. If “mmax> Mmax”, the movement amount estimation unit 15 updates Mmax, DXmax, DYmax, and Amax as follows in step S32. If “mmax> Mmax” is not satisfied, step S32 is skipped.
Mmax ← mmax
DXmax ← dxmax
DYmax ← dymax
Amax ← a
That is, the maximum value of the collation score Sc calculated during the process of scanning the rotation angle a from −A to the current angle is stored as Mmax. Further, the X-direction displacement amount, the Y-direction displacement amount, and the rotation angle that give Mmax are stored as DXmax, DYmax, and Amax, respectively.

  As described above, the movement amount estimation unit 15 performs the processing shown in FIGS. 16 to 17 to obtain the maximum matching degree Mmax between the rotating road surface pattern image D (t1) and the road surface pattern image C (t2). The amount of movement (DXmax, DYmax, Amax) to be given is calculated. Here, the movement amount calculated in this way corresponds to a time change in the layout (position and / or angle) of the road surface pattern. That is, the movement amount estimation unit 15 estimates the movement amount of the moving body 1 based on the time change of the road surface pattern.

  Note that “DX”, “DY”, and “A” used in the flowcharts shown in FIGS. 16 to 17 are determined according to the specifications or performance of the moving body 1. For example, “DX” and “DY” may be determined according to the maximum allowable speed of the moving body 1.

  The movement amount estimation unit 15 executes the process of step S2 following step S1 shown in FIG. That is, the movement amount estimation unit 15 corrects the movement amount of the moving body 1 obtained in step S1 using the integrated road surface projection image shown in FIG.

  FIG. 19 is a flowchart showing the process of step S2 shown in FIG. The movement amount estimation unit 15 executes steps S41 to S47 shown in FIG. The processing flow of steps S41 to S47 is substantially the same as steps S11 to S17 shown in FIG. However, in steps S11 to S17 and steps S41 to S47, an image to be processed, an initial value, some variables, and the like are different. Hereinafter, the difference between steps S11 to S17 and steps S41 to S47 will be described.

  In step S42, the movement amount estimating unit 15 sets the initial value of the rotation angle a to “Amax−A2”. Amax is obtained by the processing of steps S11 to S17. A2 is a constant smaller than “A” used in steps S11 to S17.

  In step S43, the movement amount estimating unit 15 generates a rotation integrated road surface projection image F (t1) by rotationally converting the integrated road surface projection image E (t1) at the previous time by an angle a. The integrated road projection image is as described with reference to FIG. 14 and is recorded in the video data DB 16. In step S44, the movement amount estimation unit 15 determines the maximum matching degree Mmax between the rotation integrated road surface projection image F (t1) obtained in step S43 and the integrated road surface projection image E (t2) at the current time. The translational displacement (DXmax2, DYmax2) that gives

  In step S45, the movement amount estimation unit 15 adds the angle Δa2 to the rotation angle a. The angle Δa2 represents an angle step for scanning the rotation angle a from Amax−A2 to Amax + A2. Δa2 is an angle step smaller than Δa used in steps S11 to S17. If the rotation angle a is equal to or less than Amax + A2 in step S46, the process of the movement amount estimation unit 15 returns to step S43.

  As described above, the movement amount estimation unit 15 scans the rotation angle a of the rotation integrated road surface projection image F (t1) from Amax−A2 to Amax + A2 in steps S43 to S46, and the rotation integrated road surface projection image F (t1). A translation displacement (DXmax2, DYmax2) that gives the maximum matching degree Mmax with the integrated road surface projection image E (t2) is specified. When the rotation angle a becomes larger than Amax + A2, the movement amount estimation unit 15 translates (DXmax2, DYmax2) which gives the maximum matching degree Mmax as a more detailed estimation result of the movement amount of the moving body 1 in step S47. And the rotation angle (Amax2) is output.

  FIG. 20 is a flowchart showing the process of step S44 of FIG. Note that the processing of the flowchart shown in FIG. 20 is executed for a given rotation angle a in the procedure of scanning the rotation angle a of the integrated rotating road surface projection image F (t1) from Amax−A2 to Amax + A2.

  The process flow of steps S51 to S62 shown in FIG. 20 is almost the same as steps S21 to S32 shown in FIG. However, the image to be processed, the initial value, some variables, and the like are different between steps S21 to S32 and steps S51 to S62. Hereinafter, differences between steps S21 to S32 and steps S51 to S62 will be described.

  In steps S51 to S62, the displacement dx is scanned in the range of dxmax ± DX2. Step Δdx2 for changing the displacement dx is smaller than Δdx used in steps S21 to S32. The displacement dy is scanned in the range of dymax ± DY2. Step Δdy2 for changing the displacement dy is smaller than Δdy used in steps S21 to S32.

Furthermore, if “mmax> Mmax” in step S61, the movement amount estimation unit 15 updates Mmax, DXmax2, DYmax2, and Amax2 as follows in step S62.
Mmax ← mmax
DXmax2 ← dxmax
DYmax2 ← dymax
Amax2 ← a
The movement amount estimation unit 15 outputs DXmax2 and DYmax2 as estimated values of the translational movement amount of the moving body 1. Further, the movement amount estimation unit 15 outputs Amax2 as an estimated value of the rotation amount of the moving body 1.

  Thus, the movement amount estimation apparatus 10 (or movement amount estimation method) of the first embodiment first estimates the movement amount of the moving body 1 using the road surface pattern image. Thereafter, the movement amount estimation apparatus 10 estimates the movement amount of the moving body 1 in more detail using the estimation result based on the road surface pattern image and the integrated road surface projection image.

  Here, the road surface pattern image is obtained by removing the three-dimensional object image during the process of projecting the camera video onto the road surface. Therefore, according to the first embodiment, the moving amount of the moving body 1 can be accurately estimated without being affected by the three-dimensional object.

  However, when the image area of the three-dimensional object to be removed is large, the number of features included in the road surface pattern image may be reduced. For this reason, the movement amount estimation apparatus 10 further performs estimation using a road surface projection image of an area that is not a common visual field area. This can prevent a decrease in the estimation accuracy of the movement amount.

  Note that the movement amount estimation apparatus 10 does not necessarily have to perform estimation using the integrated road surface projection image after the estimation processing based on the road surface pattern image. For example, when the image area of the three-dimensional object to be removed is small, the process of step S2 using the integrated road surface projection image may be omitted.

<< Second Embodiment >>
FIG. 21 is a functional block diagram of the movement amount estimation apparatus according to the second embodiment. The movement amount estimation apparatus 20 of the second embodiment includes an image input unit 11, a synchronization control unit 12, a road surface projection unit 13, a road surface pattern extraction unit 14, a rotation angle input unit 21, and a movement amount estimation unit 22. Here, the operations of the image input unit 11, the synchronization control unit 12, the road surface projection unit 13, and the road surface pattern extraction unit 14 are substantially the same as those in the first embodiment. In addition, the movement amount estimation apparatus 20 receives video data from the cameras 2 a to 2 d and receives rotation data from the rotation angle sensor 5. Furthermore, the movement amount estimation apparatus 20 is connected to the video data DB 16.

  The rotation angle sensor 5 is attached to the moving body 1 and outputs rotation data representing a rotation amount (yaw rotation amount) of the moving body 1 on the road surface. The rotation angle sensor 5 is, for example, a gyro sensor.

  The rotation angle input unit 21 calculates the rotation amount of the moving body 1 in a predetermined time based on the rotation data output from the rotation angle sensor 5. Specifically, the rotation angle input unit 21 calculates the amount of rotation of the moving body 1 between the synchronization signal generated by the synchronization control unit 12 and the synchronization signal immediately before. Then, the rotation angle input unit 21 notifies the movement amount estimation unit 22 of the calculated rotation angle.

  The movement amount estimation unit 22 estimates the movement amount of the moving body 1 based on the road surface pattern image, the integrated road surface projection image, and the rotation angle calculated by the rotation angle input unit 21. At this time, the movement amount estimation unit 22 estimates the movement amount of the moving body 1 by executing steps S1 to S2 shown in FIG. 15 as in the first embodiment. However, in the second embodiment, of the movement amount of the moving body 1, the rotation amount of the moving body 1 is detected by the rotation angle sensor 5. Therefore, the movement amount estimation unit 22 estimates the translational movement amount of the moving body 1.

  FIG. 22 is a flowchart showing the process of step S1 executed in the second embodiment. In the second embodiment, the movement amount estimation unit 22 executes steps S11, S12-2, S13-2, S14, and S17-2 as the process of step S1 in FIG. In addition, since the process of step S11, S14 is the same as 1st Embodiment, description is abbreviate | omitted.

  In step S <b> 12-2, the movement amount estimation unit 22 sets “Asen” notified from the rotation angle input unit 21 as the rotation angle a of the moving body 1. Subsequently, in step S13-2, the movement amount estimation unit 22 generates a rotating road surface pattern image D (t1) by rotationally converting the road surface pattern image C (t1) at the previous time by an angle Asen. Then, after executing step S14, the movement amount estimation unit 22 outputs a translational displacement (DXmax, DYmax) that gives the maximum matching degree Mmax as the estimation result of the movement amount of the moving body 1 in step S17-2. At this time, the movement amount estimation unit 22 also outputs the rotation amount Asen notified from the rotation angle input unit 21.

  FIG. 23 is a flowchart showing the process of step S <b> 2 executed in the second embodiment. In the second embodiment, the movement amount estimation unit 22 executes steps S41, S42-2, S43-2, S44, and S47-2 as the process of step S2 of FIG. In addition, since the process of step S41, S44 is the same as 1st Embodiment, description is abbreviate | omitted.

  In step S42-2, the movement amount estimation unit 22 sets “Asen” notified from the rotation angle input unit 21 as the rotation angle a of the moving body 1. Subsequently, in step S43-2, the movement amount estimation unit 22 generates the rotation integrated road surface projection image F (t1) by rotationally converting the integrated road surface projection image E (t1) at the previous time by the angle Asen. Then, after executing step S44, the movement amount estimation unit 22 outputs a translational displacement (DXmax2, DYmax2) that gives the maximum matching degree Mmax as the estimation result of the movement amount of the moving body 1 in step S47-2. At this time, the movement amount estimation unit 22 also outputs the rotation amount Asen notified from the rotation angle input unit 21.

  As described above, the movement amount estimation device 20 of the second embodiment acquires the rotation amount of the moving body 1 using the rotation angle sensor 5. That is, the movement amount estimation apparatus 20 can estimate the translational movement amount of the moving body 1 without executing the process of scanning the rotation angle of the road surface pattern image and the integrated road surface projection image. Therefore, in the second embodiment, the amount of calculation for estimating the amount of movement of the mobile body 1 is smaller than in the first embodiment.

<< Third Embodiment >>
As described above, in the movement amount estimation method of the embodiment (for example, the first embodiment), the movement of the movement amount 1 is performed using the road surface projection image (that is, the road surface pattern image) from which the three-dimensional object image is removed. The quantity is estimated. Thereby, the movement amount of the moving body 1 is accurately estimated without being affected by the three-dimensional object.

  In the third embodiment, as shown in FIG. 24, a road surface pattern image is used in a common visual field region of a plurality of cameras, and a road surface projection image of a region near the moving body 1 is used in other visual field regions. . In the example shown in FIG. 24, regions R1, R3, R5, and R7 are common visual field regions of two corresponding cameras, respectively, and have road surface pattern images generated by the method shown in FIG. Here, in the regions R1, R3, R5, and R7, the image corresponding to the three-dimensional object is removed. On the other hand, the regions R2, R4, R6, and R8 are visual field regions of the corresponding cameras 2a, 2c, 2b, and 2d, respectively, and have road surface projection images. However, in the third embodiment, a process of replacing the pixel value of each pixel in a region other than the region R2x close to the moving body 1 with zero is performed on the road surface projection image of the region R2. In the regions R4, R6, and R8, a process of replacing the pixel values of the pixels in the regions other than the regions R4x, R6x, and R8x adjacent to the moving body 1 with zero is executed. Hereinafter, the image shown in FIG. 24 generated in this way may be referred to as a neighborhood integrated projection image.

  Since the regions R2x, R4x, R6x, and R8x are close to the moving body 1, there is a low possibility that a three-dimensional object exists therein. Therefore, even if the movement amount of the moving body 1 is estimated using the images of the regions R2x, R4x, R6x, and R8x, the possibility of being affected by the three-dimensional object is low. The regions R2x, R4x, R6x, and R8x are generated by designating a distance range from the moving body 1 (for example, a range of 500 mm or less from the moving body 1).

  The configuration of the movement amount estimation apparatus of the third embodiment is substantially the same as that of the movement amount estimation apparatus 10 of the first embodiment. However, the process of the movement amount estimation unit of the third embodiment is different from the movement amount estimation unit 15 of the first embodiment.

  FIG. 25 is a flowchart illustrating the processing of the movement amount estimation unit according to the third embodiment. The movement amount estimation unit of the third embodiment performs steps S71 to S77 shown in FIG. 25 to estimate the movement amount of the moving body 1. Here, the process flow of steps S71 to S77 is substantially the same as steps S11 to S17 of the first embodiment. However, while the road pattern image is used in the first embodiment, the movement amount of the moving body 1 is estimated using the neighborhood integrated projection image shown in FIG. 24 in the third embodiment.

  As described above, the estimation method of the third embodiment estimates the amount of movement of the moving body 1 according to the flowchart shown in FIG. 25 using the neighborhood integrated projection image shown in FIG. Here, in the estimation method of the first embodiment, after the movement amount is estimated in step S1 shown in FIG. 15, a process of correcting the movement amount is further executed in step S2. Therefore, the estimation method according to the third embodiment requires a smaller amount of computation than the first embodiment.

  Moreover, step S1 performed in 1st Embodiment estimates the movement amount using the road surface pattern of a common visual field area | region. On the other hand, the estimation method of the third embodiment uses the road surface projection image in which there is a low possibility that a three-dimensional object in the non-common visual field area exists in addition to the road surface pattern of the common visual field area. presume. Therefore, the estimation method of the third embodiment has higher estimation accuracy than the case where only step S1 is executed in the first embodiment.

<< Fourth Embodiment >>
The configuration of the movement amount estimation apparatus of the fourth embodiment is substantially the same as that of the movement amount estimation apparatus 20 of the second embodiment. That is, also in the fourth embodiment, the rotation amount of the moving body 1 is detected using the rotation angle sensor 5. However, the process of the movement amount estimation unit of the fourth embodiment is different from the movement amount estimation unit 22 of the second embodiment. Specifically, as in the third embodiment, the estimation method of the fourth embodiment uses the neighborhood integrated projection image shown in FIG. 24 to calculate the movement amount (however, the translation movement amount) of the moving body 1. presume.

  FIG. 26 is a flowchart illustrating processing of the movement amount estimation unit according to the fourth embodiment. The movement amount estimation unit of the fourth embodiment performs steps S71, S72-2, S73-2, S74, and S77-2 to estimate the movement amount of the moving body 1. Here, the processing flow of steps S71, S72-2, S73-2, S74, and S77-2 is the same as that of steps S11, S12-2, S13-2, S14, and S17- of the second embodiment shown in FIG. 2 is almost the same. However, while the road pattern image is used in the second embodiment, the movement amount of the moving body 1 is estimated using the neighborhood integrated projection image shown in FIG. 24 in the fourth embodiment.

  Thus, the movement amount estimation method of the fourth embodiment acquires the rotation amount of the moving body 1 using the rotation angle sensor 5. That is, the movement amount estimation method of the fourth embodiment can estimate the translational movement amount of the moving body 1 without executing the process of scanning the rotation angle of the neighborhood integrated projection image. Therefore, in the fourth embodiment, compared with the third embodiment, the amount of calculation for estimating the moving amount of the moving body 1 is reduced.

<Hardware configuration of movement estimation device>
FIG. 27 is a diagram illustrating a hardware configuration of a computer system 100 for realizing the movement amount estimation apparatuses 10 and 20 when the movement amount estimation methods of the first to fourth embodiments are realized by software. . As shown in FIG. 27, the computer system 100 includes a CPU 101, a memory 102, a storage device 103, a reading device 104, a communication interface 106, and an input / output device 107. The CPU 101, the memory 102, the storage device 103, the reading device 104, the communication interface 106, and the input / output device 107 are connected to each other via a bus 108, for example.

  The CPU 101 uses the memory 102 to execute a program that describes the procedure of the above-described movement amount estimation method, whereby the image input unit 11, the road surface projection unit 13, the road surface pattern extraction unit 14, and the movement amount estimation units 15, 22 are executed. The function of part or all of the rotation angle input unit 21 is provided.

  The memory 102 is a semiconductor memory, for example, and includes a RAM area and a ROM area. The storage device 103 is, for example, a hard disk, and stores a program related to movement amount estimation according to the embodiment. Note that the storage device 103 may be a semiconductor memory such as a flash memory. The storage device 103 may be an external recording device. The video data DB 16 shown in FIG. 2 or 21 is formed in the storage device 103, for example. The camera information shown in FIG. 2 or FIG. 21 is stored in the storage device 103, for example.

  The reading device 104 accesses the removable recording medium 105 in accordance with an instruction from the CPU 101. The detachable recording medium 105 includes, for example, a semiconductor device (USB memory or the like), a medium to / from which information is input / output by a magnetic action (magnetic disk or the like), a medium to / from which information is input / output by an optical action (CD-ROM, For example, a DVD). The communication interface 106 transmits / receives data via a network according to instructions from the CPU 101. The input / output device 107 corresponds to, for example, a device that accepts an instruction from the user, an interface for connecting to the cameras 2 a to 2 d, and an interface for connecting to the rotation angle sensor 5.

The program relating to the movement amount estimation of the embodiment is provided to the computer system 100 in the following form, for example.
(1) Installed in advance in the storage device 103.
(2) Provided by the removable recording medium 105.
(3) Download from the program server 110.

  Note that the movement amount estimation apparatus according to the embodiment is mounted on, for example, a target moving body whose movement amount is estimated. Here, when the movement amount estimation apparatus is realized by executing a movement amount estimation program by a processor, the movement amount estimation program may be executed by a dedicated processor for movement amount estimation, or other applications. The movement amount estimation program may be executed by a shared processor used for the above.

  In addition, the movement amount estimation apparatus of the embodiment may be provided outside the target moving body whose movement amount is to be estimated. In this case, the video data of each camera attached to the moving body is transmitted to the movement amount estimating device via the wireless link. Then, the movement amount estimation apparatus estimates the movement amount of the moving body using video data received via the wireless link. In this case, the movement amount estimation apparatus may transmit the movement amount estimation result to the moving body via the wireless link.

  Furthermore, the movement amount estimation apparatus may be realized by hardware. Alternatively, the movement amount estimation apparatus may be realized by a combination of software and hardware.

DESCRIPTION OF SYMBOLS 1 Moving body 2a-2d Camera 5 Rotation angle sensor 10, 20 Movement amount estimation apparatus 11 Image input part 12 Synchronization control part 13 Road surface projection part 14 Road surface pattern extraction part 15, 22 Movement amount estimation part 16 Image | video data DB
21 Rotation angle input unit

Claims (6)

A projection unit that generates a road surface projection image for each image obtained by a plurality of cameras attached to a moving body;
An extraction unit that extracts image patterns that match each other between road surface projection images from a common field region in which the fields of view of the plurality of cameras overlap in a plurality of road surface projection images generated by the projection unit;
An estimation unit configured to estimate a movement amount of the moving body based on a temporal change of the image pattern extracted by the extraction unit;
A movement amount estimation apparatus comprising: The estimation unit estimates a translational movement amount of the moving body based on a temporal change in the position of the image pattern and rotation data output from a rotation sensor that detects a rotation amount of the moving body. The movement amount estimation apparatus according to claim 1. The movement according to claim 1, wherein the estimation unit corrects an estimation result obtained using the image pattern using a road surface projection image other than a region where road surface projection images overlap each other. Quantity estimation device. The estimation unit is configured to detect a movement amount of the moving body based on a temporal change of the image pattern and a temporal change of a road surface projection image in a region near the moving body in a road surface projection image other than a region where road surface projection images overlap each other. The movement amount estimation apparatus according to claim 1, wherein the movement amount estimation apparatus according to claim 1. Computer
A road surface projection image is generated for each image obtained by a plurality of cameras attached to a moving body,
Extracting image patterns that coincide with each other between the road surface projection images from a common visual field region where the fields of view of the plurality of cameras overlap in a plurality of road surface projection images,
Estimating the amount of movement of the moving body based on the temporal change of the extracted image pattern,
A moving amount estimation method characterized by the above. A road surface projection image is generated for each image obtained by a plurality of cameras attached to a moving body,
Extracting image patterns that coincide with each other between the road surface projection images from a common visual field region where the fields of view of the plurality of cameras overlap in a plurality of road surface projection images,
Estimating the amount of movement of the moving body based on the temporal change of the extracted image pattern,
A moving amount estimation program for causing a computer to execute processing.