JP4847779B2 - In-vehicle information processing method, in-vehicle information processing program, in-vehicle information processing apparatus, and in-vehicle information processing system - Google Patents

Description

  The present invention relates to an in-vehicle information processing method, an in-vehicle information processing program, an in-vehicle information processing apparatus, and an in-vehicle information processing system for monitoring a vehicle approaching from both left and right sides.

  Conventionally, when a vehicle is approaching an intersection with poor visibility or no signal, a camera is installed on the side of the vehicle in order to check other vehicles approaching from the side of the vehicle. An apparatus in which a driver grasps an approaching vehicle or the like by using the displayed image by displaying an image captured by the camera on a monitor installed inside the vehicle has been commercialized.

Therefore, a method and an apparatus are disclosed that process an image obtained by imaging the side of the host vehicle to detect an approaching vehicle and warn the driver when there is a high risk of collision with the host vehicle. Examples of such a method and apparatus include the following.
First, a vehicle collision warning method and apparatus for changing a frame interval according to the length of an edge of an approaching vehicle is disclosed (for example, Patent Document 1). According to this, the input image is differentiated to create an edge image, a horizontal edge is extracted from the created edge image, and the length and processing size of the edge are obtained. Then, the frame interval is obtained according to the length of the edge, and the optical flow of each edge is calculated using the edge image before the obtained frame interval and the current edge image. Based on the calculated optical flow, the estimated arrival time until the approaching vehicle collides with the own vehicle is obtained.

Further, a vehicle side image generation method and a vehicle side image generation apparatus that detect an approaching vehicle by obtaining a luminance time difference in an image separated by a predetermined time are disclosed (for example, Patent Document 2). According to this, when calculating the arrival time until the approaching vehicle collides with the own vehicle after the approaching vehicle is detected by the method described above, the rotation angle of the own vehicle, that is, the relative relationship between the own vehicle and the road. The arrival prediction time is predicted using the corner.
Japanese Patent Laid-Open No. 11-353565 ([0048], FIG. 8) Japanese Patent No. 3606223 (Claim 1, [0086], [0100], FIG. 2)

However, in the apparatus described in Patent Document 1, the edge image includes not only an approaching vehicle but also many edges having different edge lengths such as road markings and white lines. Furthermore, since the frame interval is different for each edge, the process of calculating the optical flow becomes complicated. Further, when the edge length is long, there is a problem that the processing size increases and the processing time increases.
Further, according to the apparatus described in Patent Document 2, since the apparent movement of the approaching vehicle differs between the distant and the vicinity, it is difficult to detect the approaching vehicle from the luminance difference between the images at the same frame interval. Furthermore, when obtaining the predicted arrival time, there is a problem that the intersection angle of the road that intersects with the road on which the vehicle travels is not considered.

  The present invention has been made in view of the above problems, and an object of the present invention is to reduce the processing size and the processing time.

In order to solve the above problems, the present invention has been completed. That is, the in-vehicle information processing method, the in-vehicle information processing program, the in-vehicle information processing apparatus, and the in-vehicle information processing system according to the present invention have an image input from the imaging device , rather than a nearby region that is an image region near the vehicle Set the measurement points in the far field, which is far from the vehicle, to be dense, and calculate the optical flow based on these measurement points in each of the near and far areas, and the calculated optical flow in the far field. based calculates the predicted arrival time is a predicted time to reach the mobile and the vehicle, based on the calculated predicted arrival time to determine the necessity of an alarm, in the vicinity area, based on the calculated optical flow It is characterized by determining the necessity of an alarm .

  According to the present invention, the processing size can be reduced and the processing time can be shortened.

Next, the best mode for carrying out the present invention (referred to as “embodiment”) will be described in detail with reference to the drawings as appropriate.
In the present embodiment, all objects to be information processing, that is, those captured in an image are described as objects, and among the objects, those that are moving are described as moving objects, A moving object that is approaching the vehicle is described as an approaching object.

FIG. 1 is a functional block diagram illustrating a configuration example of the in-vehicle information processing system according to the present embodiment.
The in-vehicle information processing system 9 includes a right camera (imaging device in claim) 3 and a left camera (imaging device in claim) that are attached to the front portion of the host vehicle (vehicle in the claim) so that left and right side images can be taken. ) The vehicle-mounted information processing apparatus 1 mounted on the vehicle is provided for performing information processing on the image input from 5 according to the vehicle-mounted information processing method described later. The in-vehicle information processing system 9 stores a road map, monitors the position of the own vehicle based on the road map, and sends the information related to the position of the own vehicle and the road map to the in-vehicle information processing apparatus 1. 8 is provided. Furthermore, a monitor (output device in claims) 7 for displaying the result of the information processing performed by the in-vehicle information processing device 1 is provided.

  The in-vehicle information processing apparatus 1 includes a right camera processing unit (processing unit in claims) 2 and a left camera processing unit (processing unit in claims) that perform information processing of images input from the right camera 3 and the left camera 5. Prepare. Further, when the information processing result is received from the right camera processing unit 2 or the left camera processing unit 4, an output processing unit 6 that displays the result of the information processing on the display device is provided.

FIG. 2 is a functional block diagram illustrating a configuration example of the right camera processing unit or the left camera processing unit.
The right camera processing unit 2 includes the following units and memories.
The measurement point setting processing unit 201 divides the input image into a near area that is an image area near the own vehicle and a far area that is an image area far from the own vehicle. Set measurement points with different densities.
The moving body processing unit 202 creates an edge image and an edge profile for the image acquired from the time-series image memory 109, creates an edge profile based on the created edge profile, detects the moving body, and determines the amount of movement. Perform the calculation.
The optical flow calculation unit 203 calculates the optical flow (velocity vector) of the moving object in the image.

The moving body determination unit 204 determines the presence / absence of an approaching object based on the results of processing performed by the moving body processing unit 202, the optical flow calculation unit 203, and the like.
The estimated arrival time calculation unit 205 approaches the vehicle based on the optical flow calculated by the optical flow calculation unit 203 and the optical axis direction of the right camera 3 installed in the vehicle calculated by the optical axis direction calculation unit 206. A predicted arrival time, which is a predicted arrival time of the object, is calculated.
The alarm determination unit 207 determines whether or not to output an alarm by determining whether or not the predicted arrival time calculated by the predicted arrival time calculation unit 205 is equal to or less than a predetermined threshold in the processing in the far region.
The warning processing unit 208 is output when the warning determination unit 207 determines that the predicted arrival time is equal to or less than a predetermined threshold, or when the moving body determination unit determines that an approaching object exists in the vicinity area. The processing unit 6 outputs an alarm.

A time-series image memory (storage unit in claims) 209 is a memory configured to store images captured by the right camera 3 in a time-series, and discard images that have passed a predetermined time from the current time. For example, the memory address is configured from a ring memory that counts up from an initial value and returns to the initial value when the memory address is counted up to a predetermined value.
The time-series edge profile memory 210 is a memory that stores an edge profile created from an edge image by a method described later in time series. Similar to the time-series image memory 209, the time-series edge profile memory 210 is a memory configured to discard an edge profile that has passed a predetermined time from the current time.

The configuration of the left camera processing unit 4 input from the left camera 5 is also the same as that of the right camera processing unit 2 (components of the left camera processing unit 4 in FIG. 2 are denoted by reference numerals 401 to 410).
1 and FIG. 2 are realized by executing a program stored in a storage device such as a ROM (Read Only Memory) (not shown) by a CPU (Central Processing Unit) (not shown). .

(System schematic)
FIG. 3 is a diagram showing an outline of the in-vehicle information processing system according to the present embodiment.
FIG. 3 shows an example in which one camera is attached to the front portion of the own vehicle 31 in order to monitor the left-right direction of the own vehicle (vehicle in claims) 31. The right camera 3 monitors an approaching object from the right, and the left camera 5 monitors an approaching object from the left.

(Outline of processing)
Next, an in-vehicle information processing method in the in-vehicle information processing apparatus 1 of the present embodiment will be described along FIG. 4 with reference to FIGS. 1 to 3.
FIG. 4 is a flow showing the flow of the in-vehicle information processing method according to the present embodiment.
In the present embodiment, the processing in the right camera processing unit 2 will be described, but the processing in the left camera processing unit 4 is the same as described later.
First, the right camera processing unit 2 receives an image via the right camera 3 (S1), and stores the input image in the time-series image memory 209 (S2). The description of the input image will be described later with reference to FIG.
The moving object processing unit 202 uses the images stored in the time-series image memory 209 to detect a moving object within the processing range, and the amount of movement that the detected moving object moves within the processing range within a predetermined time. A moving body process for calculating the amount is performed (S3). The flow of the mobile processing will be described later with reference to FIG.
Next, the measurement point setting processing unit 201 has a distant area that is an image area in which a distant image is captured and a near area that is an image area in which a nearby image is captured with respect to the image input in step S1. After setting the far area and the near area to be divided into areas (S4), the measurement points are set for the far area at preset intervals, and the near area has a density different from that of the far area. A measurement point is set (S5). Details of the setting of the far and near areas and the measurement point setting process will be described later with reference to FIG.

(Processing in remote area)
Steps S6 to S11 are processing in a far field.
Subsequently, the optical flow calculation unit 203 calculates the frame interval between the two images used in the calculation of the optical flow from the movement amount calculated in the moving object process (S6), and uses the calculated frame interval to calculate the optical flow. Calculate (S7). Details of the optical flow calculation process will be described later with reference to FIGS. 13 and 14.
And the mobile body determination part 204 determines whether an approaching object exists in a distant area | region based on the calculated optical flow for every measurement point (S8). Details of the approaching object determination process will be described later with reference to FIG.
When the moving body determination unit 204 determines that there is no approaching object in the far region (S8 → No), the right camera processing unit 2 advances the process to step S10.
When the moving body determination unit 204 determines that an approaching object is present in the far region (S8 → Yes), the right camera processing unit 2 advances the process to step S9.

Next, the predicted arrival time calculation unit 205 and the optical axis direction calculation unit 206 calculate the predicted arrival time, which is the predicted time for the mobile object that approaches the host vehicle using the optical flow calculated in step S7 to reach the host vehicle. Calculate (S9). Details of the process for calculating the predicted arrival time will be described later with reference to FIGS.
Then, the alarm determination unit 207 determines whether or not to generate an alarm by determining whether or not the calculated predicted arrival time is equal to or less than a preset threshold value (S10).
When the warning determination unit 207 determines that the predicted arrival time is greater than the threshold (not less than the threshold) (S10 → No), the right camera processing unit 2 advances the process to step S12.
When the alarm determination unit 207 determines that the predicted arrival time is equal to or less than the threshold (S10 → Yes), the alarm processing unit 208 instructs the output processing unit 6 to display an alarm.
Then, the instructed output processing unit 6 displays an alarm on the monitor 7 (S11).

(Processing in the neighborhood)
Next, steps S10 to S15 are processing in the vicinity region.
The difference between the processing in the vicinity region and the processing in the far region is that the arrival prediction time is not calculated.
In step S12 (frame interval calculation process), step S13 (optical flow calculation process) in the vicinity area, and in the vicinity area in step S14, whether or not there is an approaching object is determined by each process (step Since it is the same processing as S6, S7, S8), the description is omitted.
When the moving body determination unit 204 determines that there is no approaching object in the vicinity region (S14 → No), the right camera processing unit 2 advances the process to step S16.
When the moving body determination unit 204 determines that an approaching object is present in the vicinity region (S14 → Yes), the alarm processing unit 208 instructs the output processing unit 6 to display an alarm.
Then, the instructed output processing unit 6 displays an alarm on the monitor 7 (S15).
Next, the right camera processing unit 2 displays the image input in step S1 on the monitor 7 (S16). At this time, if the arrival prediction time is calculated in S9, the right camera processing unit 2 may display the arrival prediction time on the monitor 7.

In the present embodiment, the alarm is displayed on the monitor 7, but the present invention is not limited to this, and the alarm may be output by a voice output device (not shown) or the like.
Then, the right camera processing unit 2 returns the process to step S1. The right camera processing unit 2 may execute the processing from step S1 to step S16 while the host vehicle is traveling.
Further, the processing described below may be performed. In the right camera processing unit 2, the navigation device monitors the position of the own vehicle, and based on the information sent from the navigation device, whether the right camera processing unit 2 is located within a predetermined distance from the intersection. Determine whether. And while the own vehicle is located within a predetermined distance from the intersection, the processing from step S1 to step S16 may be continued.

It is not meaningful for the right camera processing unit 2 to calculate the predicted arrival time for a vehicle existing in the vicinity of the host vehicle, and it is desirable to immediately generate an alarm.
Therefore, when a moving body is detected in the vicinity region as in this embodiment, the processing time can be shortened by omitting the calculation of the predicted arrival time, and an alarm can be generated immediately.

(Moving object processing)
Next, the moving body process in step S3 will be described with reference to FIG.
FIG. 5 is a flow showing the flow of the moving object process in step S3.
First, the moving body processing unit 202 acquires an image from the time-series image memory 209, and creates an edge image from the acquired image (S41). The edge image creation process will be described later with reference to FIG.

Next, the moving body processing unit 202 creates an edge profile for the created edge image (S42). The edge profile creation process will be described later with reference to FIG.
Then, the mobile processing unit 202 stores the created edge profile in the time-series edge profile memory 210 (S43).
Further, the moving object processing unit 202 detects a moving object using two edge profiles that are temporally different (for example, at an imaging interval) in the time-series edge profile memory 210 (S44). Details of the moving object detection process will be described later with reference to FIGS. 9 and 10.

(Image input: Step S1 in FIG. 4)
Hereinafter, each process in FIGS. 4 and 5 will be described in detail along FIGS. 6 to 18 with reference to FIGS. 1 to 5 as appropriate.
First, an example of an image input to the right camera processing unit 2 and the left camera processing unit 4 along FIG. 6 will be shown with reference to FIG.
FIGS. 6A and 6B are diagrams illustrating examples of images captured by the right camera and the left camera illustrated in FIG. 3 and input to the camera processing unit. FIG. 6A illustrates an example of the right camera image and FIG. 6B illustrates an example of the left camera image. FIG.
Here, the optical flow will be described with reference to FIG.
When the moving bodies 62 and 66 move with respect to the stationary right camera 3 or left camera 5, the optical flow 63 is determined from the movement of the images of the moving bodies 62 and 66 in the image (the right camera image 61 or the left camera image 65). , 67 is obtained. The optical flows 63 and 67 are velocity vectors at respective points on the images of the moving bodies 62 and 67. The extension lines of the optical flows 63 and 67 intersect within a certain range in the right camera image 61 or the left camera image 65. This intersection is called FOE (Focus of Expansion) 64, 68 and is the center of the optical flows 63, 67.

(Creation of edge profile: Step S41 and Step S42 in FIG. 5)
FIG. 7 is a diagram illustrating an example of edge profile creation, where (a) is an example of an edge image, and (b) is a diagram illustrating an example of an edge profile.
Here, it is assumed that the image input to the right camera processing unit 2 is the image shown in FIG. The edge image shown in FIG. 7A is a result of the moving body processing unit 202 extracting the edge 82 in the vertical direction within the predetermined processing range 81 with respect to the image acquired from the time-series image memory 209. . Specifically, the moving body processing unit 202 calculates the difference between the density values of the horizontal coordinates i and i + 2 of the right camera image 61 (hereinafter referred to as the image 61 as appropriate), and calculates the difference between the horizontal coordinates i within the processing range 81. It can be obtained by performing for each vertical coordinate j. Here, the coordinates of the image represent pixel positions. That is, the moving body processing unit 202 executes a process for calculating a difference in density value between every other horizontal pixel in all the pixels. Here, the predetermined processing range is, for example, the entire image.
The edge profile shown in FIG. 7B is a value obtained by accumulating the density difference values in the edge image shown in FIG. 7A in the vertical direction by the moving body processing unit 202.
The edge profile is a one-dimensional waveform having a horizontal coordinate of the image on the horizontal axis and a cumulative density difference value on the vertical axis, and has a peak at a position corresponding to the edge as can be seen from FIG.

(Storage of edge profile: Step S43 in FIG. 5)
FIG. 8 is a diagram illustrating an example of edge profile storage in the time-series edge profile memory.
As shown in FIG. 8, the moving object processing unit 202 stores from the edge profile created in the past to the most recently created edge profile in the time-series edge profile memory 210. Here, Δt in FIG. 8 is an imaging interval, and as shown in FIG. 8, a past edge profile before p * Δt from the current t is stored for each Δt.

(Moving object detection process: Step S44 in FIG. 5)
FIG. 9 is a diagram illustrating an example of processing for detecting a moving object.
After acquiring the current edge profile and the edge profile before time Δt1 from the time-series edge profile memory 210, the moving object processing unit 202 detects the moving object by the method described below.
As a moving body detection method, it is conceivable that the moving body processing unit 202 performs a correlation calculation between the current edge profile and the edge profile before time Δt1 as shown in FIG. 9 (template matching). This is a method in which the mobile processing unit 202 searches for a partial waveform close to the partial waveform in the edge profile obtained from the current image from the waveform of the past edge profile before time Δt1. For example, as shown in FIG. 9, the moving body processing unit 202 uses a waveform corresponding to m pixels from the coordinate i0 as a template, and resembles the template using n pixels before and after the coordinate i0 of the edge profile before the predetermined time Δt1 as a search range. Find the waveform. As a template determination method, instead of using the predetermined pixel width m, a convex or concave portion of the edge profile waveform may be searched, and a portion surrounded by the concave-convex-concave may be used as a template.

As the similarity r by the correlation calculation, for example, r (i1) = Σ {f (i0 + x−i1) −g (i0 + x)} ² can be considered. Here, g (i0 + x) is a template, and f (i0 + x−i1) is a waveform within the search range of the edge profile before time Δt1. The similarity r (i1) represents the coordinates where the waveform having the smallest i1 closest to the template exists. Then, the moving body processing unit 202 calculates a movement amount d (i0) = i0−i1 based on the determined i1 and i0. Next, the moving body processing unit 202 determines the template by changing the coordinate i0, and searches for the waveform closest to the template from the edge profile before the predetermined time Δt1. The moving body processing unit 202 repeats this template matching to calculate the movement amount d (i) for each i0. When the moving body is far away, the apparent movement on the image is small. Therefore, if Δt1 is reduced, the amount of movement also becomes a small value. Therefore, when the moving body is far away, Δt1 may be set large.

FIG. 10 is a diagram showing the moving object region obtained as a result of the processing shown in FIG.
The moving body processing unit 202 calculates the moving body area shown in FIG. 10 by graphing the moving amount calculated for each i0 described with reference to FIG. 9 with the horizontal axis representing i0 and the vertical axis representing the moving amount. To do.
Since this process uses an image input from the right camera 3, when the host vehicle is stopped, it indicates a stationary object when the movement amount d (i) is zero, and when it is positive, A moving object that leaves the vehicle is shown, and when it is negative, it indicates that there is a moving object that approaches the vehicle. When a moving body (approaching object) approaching the host vehicle is detected as a moving body, a range where the moving amount d (i) is negative as shown in FIG. 10 may be set as the moving body region (i1, i2). . Here, i1 and i2 are an upper limit value and a lower limit value of i at which the movement amount d (i) is a negative value. Then, the moving object processing unit 202 outputs the moving object movement amount d in the detected moving object region (i1, i2), the moving object region (i1, i2), and the time Δt1 to the optical flow calculating unit 203. . Here, d is a value obtained by averaging d (i) in the moving body region with respect to i in the range from i1 to i2.

(Distant area and neighborhood area setting, measurement point setting: Steps S4 and S5 in FIG. 5)
FIG. 11 is a diagram illustrating an example of setting of a far region and a near region and measurement point setting according to the present embodiment.
Hereinafter, the right camera image 61 is referred to as an image 61 as appropriate.
As shown in FIG. 11, in the case of the right camera image 61 acquired from the right camera 3, the measurement point setting processing unit 201 sets the predetermined region on the right side of the right camera image 61 as the far region 72 and the predetermined region on the left side of the image as the neighborhood region. 71. The size of the far region is assumed to be set in the right camera processing unit 2 in advance via an input unit (not shown).
Then, the measurement point setting processing unit 201 sets the measurement points 73 for calculating the optical flow at different densities in the near region 71 and the far region 72. Specifically, the distant area 72 sets the measurement points 73 more densely than the neighboring area 71.
For example, the measurement points 73 are set at intervals of 4 pixels in the far region 72, whereas the measurement points 73 are set at intervals of 28 pixels in the neighborhood region 71, for example, at a density of about 1/4. The density of the measurement points 73 may be changed as appropriate according to the required detection accuracy.

When the moving body is far away from the own vehicle 31, the moving body is imaged small. Therefore, in order to accurately calculate the optical flow of the moving body in the far region 72, it is necessary to set the measurement points 73 densely. There is.
However, when the moving body is in the vicinity of the own vehicle, the moving body is imaged large, so the measurement points 73 do not have to be set so densely.
The size of the distant area 72 needs to be determined so as to include a distant moving body area (described later with reference to FIG. 9). For this reason, there is a method of determining the width of the far region 72 so that the moving body region enters the far region 72 even in the case of the shortest imaging interval. The height of the far region 72 may be determined according to the FOE of the optical flow obtained in the previous loop.

FIG. 12 is a diagram illustrating an example of measurement point setting according to the comparative example.
As shown in the comparative example, conventionally, the measurement points 73 are not limited to the far and near areas, but are set with equal density. That is, the measurement points in the vicinity region are set with a density according to the far region. Therefore, since the measurement points are set at the same density as that of the far field even in the vicinity area where it is not necessary to set the measurement points finely because the moving body is imaged large, the processing size when calculating the optical flow described later Will become bigger.
On the other hand, in this embodiment, as shown in FIG. 11, the processing size is reduced without reducing the detection accuracy by changing the density of the setting of the measurement points 73 in the near region 71 and the distant region 72. In addition, the processing time can be shortened.

(Overview of optical flow)
Hereinafter, with reference to FIG. 13 to FIG. 18, detection of an approaching object by optical flow and calculation of an estimated arrival time for the approaching object to reach the host vehicle will be described.
In the description from FIG. 13 to FIG. 15, the same processing is performed except that the distant area and the near area differ only in the density of measurement point arrangement.
FIG. 13 is a diagram illustrating an example of an optical flow.
In order to calculate the approaching object and the predicted arrival time, it is necessary to obtain a movement vector (that is, an optical flow) 1201 in which the moving body has moved during a predetermined time. Here, as shown in FIG. 13, the movement vector 1201 in the image 61 is a vector expression (illustrated by an arrow) of the position (indicated by a broken line) of the moving object before the time Δt2 and the current position (illustrated by a solid line). ).

(Frame interval calculation: Steps S6 and S12 in FIG. 4)
FIG. 14 is a diagram illustrating an example of a frame interval calculation process.
First, the time interval (hereinafter referred to as the frame interval) Δt between the two images 61 used for calculating the optical flow is set to the same value as the time Δt1 used when the moving body processing unit 202 calculates the movement amount d. In this case, if the movement amount d is large, it is necessary to increase the search range a for searching for an image similar to the template image 1301 shown in FIG. 14 (similar image 1302), which increases calculation time. Therefore, the optical flow calculation unit 203 changes the frame interval according to the movement amount d calculated by the moving body processing unit 202. That is, when the movement amount d is large, the frame interval Δt is made smaller than the time Δt1, and when the movement amount d is small, the frame interval Δt is made larger than the time Δt1. For example, if the horizontal search range shown in FIG. 14 is a, the optical flow calculation unit 203 determines the frame interval Δt by the following formula 1.

    Δt = a / d * Δt1 (Formula 1)

  Using this equation, the search range a can be made constant by adjusting the frame interval Δt. Therefore, the optical flow calculation time can be made constant regardless of the movement amount d.

  The movement amount calculated in step S44 in FIG. 5 is a change in the horizontal coordinate direction of the image. In order to calculate the optical flow 1303, it is also necessary to calculate the amount of movement in the vertical coordinate direction. As a method for calculating the optical flow, there are a block matching method and a gradient method. Although the block matching method will be described here, it is natural that the gradient method may be used without being limited to this.

In the block matching method, first, as described with reference to FIG. 18, the measurement point setting processing unit 201 arranges measurement points 1304 on the image 61 for obtaining the optical flow 1303 at predetermined positions. Then, the optical flow calculation unit 203 acquires the image 61 from the time-series image memory 209 using the image of the local region centering on the measurement point 1304 as the template image 1301. Next, the optical flow calculation unit 203 acquires from the time-series image memory 209 the image 61 that has passed the frame interval Δt from the template image 1301, and sets the template image previously set around the measurement point 1304 on the acquired image 61. An image most similar to 1301 (similar image 1302) is searched. If the coordinates of the measurement point 1304 in the template image 1301 are (i _p (t), j _p (t)), and the coordinates of the measurement point 1304 in the similar image 1302 are (i _q (t + Δt), j _q (t + Δt)), The optical flow 1303 (u, v) can be calculated by the following equation.
u = i _q (t + Δt) −i _p (t) (Formula 2)
v = j _q (t + Δt) −j _p (t) (Formula 3)
This is shown in FIG. If the template image 1301 is g (x, y) and the image (similar image 1302) cut out with the same size as the template image 1301 around the coordinates (k, l) in the search range a is f (k + x, l + y), The similarity r (k, l) can be calculated by Equation 4.

(Approaching object determination process: Step S8 in FIG. 4)
FIG. 15 is a diagram illustrating an example of approaching object determination processing, where (a), (b), and (c) are examples of approaching object detection, and (d) and (e) are no approaching object detection. It is a figure which shows an example.
In FIG. 15, “■” indicates a measurement point where an optical flow has occurred (optical flow generation point 1401), and “□” indicates a measurement point where no optical flow has occurred (optical flow non-occurrence point 1402). That is, the moving body determination unit 204 determines that there is an approaching object if there are two or more optical flow generation points 1401 among the measurement points in adjacent rows or columns (FIGS. 20A and 20B). , (C)), and other than that is set as an approaching object not detected (FIGS. 20D and 20E). That is, if there are less than two optical flow generation points 1401 in adjacent rows or columns, the moving body determination unit 204 determines that the optical flow generation point 1401 is noise and does not determine that it is an approaching object.

(Calculation of predicted arrival time: step S9 in FIG. 4)
Next, a method for calculating the predicted arrival time between the approaching object and the own vehicle from the calculated optical flow will be described.
FIG. 16 is a diagram illustrating an example of a reference coordinate system and a camera coordinate system.
As coordinate systems handled by the predicted arrival time calculation unit 205, there are a reference coordinate system, a camera coordinate system, an imaging plane coordinate system, and an image memory coordinate system. In the reference coordinate system, an axis along a road that intersects the road on which the vehicle is traveling is defined as a Z ′ axis, and axes perpendicular to the axis are defined as an X ′ axis and a Y ′ axis (X ′, Y ′, Z ′) coordinate system. On the other hand, the camera coordinate system is an (X, Y, Z) coordinate system in which the optical axis with respect to the lens of the right camera 3 is set as the Z axis, and the X axis and the Y axis perpendicular thereto are defined. The origin of the basic coordinate system and the camera coordinate system is the center of the lens of the right camera 3 installed in the own vehicle. When the angle between the Z ′ axis and the Z axis is θ, the following relational expression is established between the basic coordinate system and the camera coordinate system.

FIG. 17 is a diagram illustrating an example of the imaging plane coordinate system.
As shown in FIG. 17, the imaging surface coordinate system is a coordinate system with the center of the imaging surface (the surface on the image sensor provided inside the camera) as the origin, and the coordinates in the imaging surface coordinate system are (x, y). Represented by At this time, if the focal length is F, the following relational expression is established between the point P (Xp, Yp, Zp) on the camera coordinate system and the point p (x _p , y _p ) on the imaging plane coordinate system.

FIG. 18 is a diagram illustrating an example of an image memory coordinate system.
The image memory coordinate system is represented by (i, j) with the upper left corner of the image as the origin as shown in FIG. At this time, the following relational expression is established between the imaging plane coordinate system (x, y) and the image memory coordinate system (i, j).

Here, I and J are coordinates with the center of the image memory as the origin.
Here, the size of the imaging surface is W × H [m], the number of effective pixels is P _W × PH _, and the offset between the origin of the imaging surface coordinate system and the origin of the image memory coordinate system is offset1 (x-axis direction), Let it be offset2 (y-axis direction).
Now, the vehicle is stationary, an approaching object is a running straight along a road at a constant speed V _0. When a point with an approaching object is expressed in the reference coordinate system, it becomes P (X′p, Y′p, Z′p), and X′p, Y′p is constant regardless of time. The predicted arrival time calculation unit calculates Z′p, which is the distance between the host vehicle and the approaching object, using Equation 11.

Next, the predicted arrival time calculation unit 205 includes the coordinates (i _p (t), j _p (t)) of times t ₀ , t _{1 =} t ₀₊ Δ _t obtained by the optical flow calculation process shown in FIG. The coordinates of the image memory coordinate system calculated by substituting (i _q (t + Δt), j _q (t + Δt)) into Equations 9 and 10 are I (t ₀ ), I (t ₁ ), J (t ₀ ), J (t ₁ ) is obtained, and the ratios are r (x) and r (y). At this time, r (x) and r (y) are expressed by Equation 12 and Equation 13, respectively. Here, t ₁ is the time when the currently processed image is captured, and t ₀ is the time when the previously processed image is captured.

  Substituting Equation 5, Equation 6, and Equation 11 into Equation 13, r (y) can be expressed as Equation 14.

If the estimated arrival time is T, T can be expressed as Z ′ (t ₀ ) / V ₀ . Therefore, Expression 14 is as follows.

Here, b = X ₀ / V ₀ . Also, r (x) can be transformed as shown in Equation 16.

  Here, when r = r (x) / r (y), Expression 17 is obtained.

Here, X ₀ is the x coordinate value of the imaging plane coordinate system at t ₀ , and V ₀ is the speed of the host vehicle at time t ₀ under the assumption that the host vehicle travels at a constant speed.
By obtaining b from Expression 17 and substituting it into Expression 15, the arrival prediction time calculation unit 205 can calculate the arrival prediction time T using the following Expression 18. Here, Δt is a frame interval calculated by the optical flow calculation unit 203.

  As can be seen from Equation 18, in order to calculate the predicted arrival time T, the optical axis direction θ needs to be obtained. θ is composed of the following components. That is, the angle when the camera is installed in the vehicle, the rotation angle (steering angle) of the vehicle when the driver performs steering operation, and the intersection angle α between the traveling road and the intersection road at the intersection as shown in FIG. is there. The camera installation angle is defined by a design value, and the rotation angle of the vehicle can be measured by using a steering angle sensor, a yaw rate sensor, or the like. The road intersection angle α may be stored in the road map of the navigation system 83. The rotation angle of the host vehicle may be calculated based on the slope of the white line detected by the imaging device. The optical axis direction calculation unit 206 uses the value acquired from the steering angle sensor or the yaw rate sensor, the road intersection angle α acquired from the road map stored in the navigation system 83, and the camera installation angle. The optical axis direction θ is calculated by the following equation.

  θ = Camera installation angle + Steering angle + Road intersection angle (Equation 19)

  The arrival prediction time calculation unit 205 substitutes the coordinates of the measurement points in the moving body region (see FIG. 10) calculated by the moving body processing and θ of Expression 19 into Expression 18, so that the arrival prediction time calculation section 205 The arrival prediction time at each measurement point in the moving body region is calculated. And the arrival prediction time calculation part 205 should just make the arrival prediction time which becomes the minimum in the arrival prediction time of each calculated measurement point the arrival prediction time of the mobile body.

  In the left camera processing unit 4 that processes an image input from the left camera 5, the relationship between the moving amount d and the approach / separation of the moving object is reversed. However, by reversing the left and right mirror images of the image of the right camera 3 described above, the relationship between the positive / negative of the moving amount d (i) and the approach / separation of the moving body can be made the same. Therefore, the processing content in the left camera processing unit 4 can be the same as the processing in the right camera processing unit 2 described above.

  If there is an approaching object in the image captured by the right camera 3 by such a method, it can be detected and an alarm can be generated according to the predicted arrival time. In the case of the right camera 3, an approaching object that moves from the right to the left of the image is an approaching object to the own vehicle. In the case of the left camera 5, an object that moves from the left to the right of the image is directed to the own vehicle. It becomes an approaching object. Therefore, if the image of the left camera 5 is mirror-inverted, the processing used for the right camera 3 can be used as it is as described above, and the program capacity can be suppressed.

  In the present embodiment, the processing in the far region in FIG. 4 is performed prior to the processing in the near region. However, the present invention is not limited to this. You may perform before a process (namely, step S6 to step S11 of FIG. 4).

(effect)
According to the present embodiment, the image is divided into a near region and a far region, and the measurement points set in the near region are set to be sparse compared with the measurement points in the far region. By not performing the calculation, it is possible to shorten the time from when the moving body is detected in the vicinity region until the alarm is output. Therefore, the processing size can be reduced by reducing the number of measurement points in the vicinity region, and the processing time can be shortened by not calculating the arrival prediction time in the vicinity region.

It is a functional block diagram which shows the structural example of the vehicle-mounted information processing system concerning this embodiment. It is a functional block diagram which shows the structural example of a right camera process part or a left camera process part. It is the figure which showed the outline of the vehicle-mounted information processing system which concerns on this embodiment. It is a flow which shows the flow of the vehicle-mounted information processing method which concerns on this embodiment. It is a flow which shows the flow of the mobile body process in step S3. It is a figure which shows the example of the image which the right camera and left camera shown in FIG. 3 image, and is input into a camera process part, (a) A right camera image, (b) It is a figure which shows the example of a left camera image. It is a figure which shows the example of edge profile creation, (a) is an example of an edge image, (b) is a figure which shows the example of an edge profile. It is a figure which shows the example of edge profile storage in a time-sequential edge profile memory. It is a figure which shows the example of a process of a mobile body detection. It is a figure which shows the moving body area | region obtained as a result of the process shown in FIG. It is a figure which shows the example of the setting of a distant area | region and vicinity area | region which concerns on this embodiment, and measurement point setting. It is a figure which shows the example of the measurement point setting which concerns on a comparative example. It is a figure which shows the example of an optical flow. It is a figure which shows the example of a process of frame space | interval calculation. It is a figure which shows the example of a process of approaching object determination, (a), (b), (c) is an example of an approaching object detection, (d), (e) is a figure which shows the example of an approaching object non-detection. It is. It is a figure which shows the example of a reference | standard coordinate system and a camera coordinate system. It is a figure which shows the example of an imaging surface coordinate system. It is a figure which shows the example of an image memory coordinate system.

Explanation of symbols

1 On-vehicle information processing device 2 Right camera processing unit (processing unit)
3 Right camera (imaging device)
4 Left camera processing unit (processing unit)
5 Left camera (imaging device)
6 Output processing unit 7 Monitor 8 Navigation system 9 On-board information processing system 31 Own vehicle 61 Right camera image (image)
62 Moving object 63, 1303 Optical flow 65 Left camera image 71 Near area 72 Distant area 73, 1304 Measurement point 81 Processing range 82 Edge 83 Navigation system 201 Measurement point setting processing part 202 Moving object processing part 203 Optical flow calculation part 204 Moving object Determination unit 205 Estimated arrival time calculation unit 206 Optical axis direction calculation unit 207 Alarm determination unit 208 Alarm processing unit 209 Time-series image memory (storage unit)
210 Time-series edge profile memory 1201 Movement vector 1301 Template image 1302 Similar image 1401 Optical flow generation point 1402 Optical flow non-occurrence point

Claims (8)

An in-vehicle information processing method in an in-vehicle information processing apparatus that detects the presence of a moving body based on an image captured by an imaging device mounted on a vehicle,
The in-vehicle information processing apparatus includes a storage unit that stores information,
The image input from the imaging device is stored in the storage unit,
The in-vehicle information processing apparatus is
Two images different in time are acquired from the storage unit,
Based on the two images, detection of the moving body in the image and calculation of the moving amount of the moving body ,
Of the image, an image region near the vehicle as a neighboring region, the distant image areas from the vehicle than the neighboring region and far region,
Set the measurement points in the far region to be denser than the nearby region ,
In processing in the far field,
Calculate the frame interval based on the amount of movement,
From the storage unit, obtain two images of the frame interval,
Based on the two acquired images and the measurement points in the far region, an optical flow is calculated,
Based on the optical flow and the calculated, said the vehicle, calculates a is predicted arrival time prediction time said detected moving object arrives,
Based on the calculated predicted arrival time, it is determined whether an alarm is necessary,
In the processing in the vicinity region,
Calculate the frame interval based on the amount of movement,
From the storage unit, obtain two images of the frame interval,
Based on the acquired two images and the measurement points in the vicinity region, an optical flow is calculated,
An in-vehicle information processing method that determines whether or not an alarm is necessary based on the calculated optical flow. The in-vehicle information processing device can be connected to an output device that outputs an image or sound sent from the in-vehicle information processing device,
The in-vehicle information processing apparatus is
The in-vehicle information processing according to claim 1, further comprising: causing the output device to output the alarm when it is determined that the alarm is necessary as a result of determining whether the alarm is necessary. Method. The in-vehicle information processing device can be connected to an output device that outputs an image sent from the in-vehicle information processing device,
The in-vehicle information processing apparatus is
The in-vehicle information processing method according to claim 1 , further comprising causing the output device to output the input image. The in-vehicle information processing apparatus can be connected to a navigation system that stores road map information,
The in-vehicle information processing apparatus is
As the calculation of the predicted arrival time,
Using at least one of the preset installation angle of the imaging device, the output of the yaw rate sensor installed in the vehicle, the output of the steering angle sensor, or the inclination of the white line detected by the imaging device, Get the rotation angle,
Obtaining the intersection angle between the road on which the vehicle travels and the road intersecting at an intersection from the road map obtained from the navigation system;
Installation angle of the imaging device in the vehicle, the based on the rotation angle and the crossing angle of the vehicle, vehicle according to claim 1, characterized by further executing that calculates the predicted arrival time Information processing method. The in-vehicle information processing method according to claim 1, wherein at least two imaging devices are installed on both left and right sides of the vehicle. An in-vehicle information processing program that causes a computer to execute the in-vehicle information processing method according to any one of claims 1 to 5 . An in-vehicle information processing device that detects the presence of a moving object based on an image captured by an imaging device mounted on a vehicle,
A storage unit for storing information;
A processing unit that stores an image input from the imaging device in the storage unit;
A moving body processing unit that acquires two images that are temporally different from the storage unit, detects a moving body in the image based on the two images, and calculates a moving amount of the moving body,
Of the image, an image region near the vehicle as a neighboring region, wherein the remote image area from the own vehicle than the region near the far region, than the neighboring region, dense measurement points in the far region A measurement point setting processing unit for setting to be ,
In each of the near area and the far area , a frame interval is calculated based on the movement amount, two images of the frame interval are acquired from the storage unit, the acquired two images, and the measurement An optical flow calculation unit for calculating an optical flow based on the points;
In the far region, based on the optical flow and the calculated, and the on the vehicle, estimated arrival time calculation unit for the detected moving body calculates the predicted arrival time is a predicted time to arrive,
An alarm determination unit that determines whether or not an alarm is required based on the calculated predicted arrival time in the processing in the remote area;
A vehicle information processing apparatus comprising: a moving body determination unit that determines whether or not an alarm is required based on the calculated optical flow in the process in the vicinity region. An imaging device mounted on a vehicle, mounted on the vehicle, based on the image of the imaging device takes an image, a vehicle information processing system comprising and a vehicle information processing apparatus for detecting the presence of the moving body,
The in-vehicle information processing apparatus
A storage unit for storing information;
A processing unit that stores an image input from the imaging device in the storage unit;
A moving body processing unit that acquires two images that are temporally different from the storage unit, detects a moving body in the image based on the two images, and calculates a moving amount of the moving body,
Of the image, an image region near the vehicle as a neighboring region, wherein the remote image area from the own vehicle than the region near the far region, than the neighboring region, dense measurement points in the far region A measurement point setting processing unit for setting to be ,
In each of the near area and the far area , a frame interval is calculated based on the movement amount, two images of the frame interval are acquired from the storage unit, the acquired two images, and the measurement An optical flow calculation unit for calculating an optical flow based on the points;
In the far region, based on the optical flow and the calculated, and the on the vehicle, estimated arrival time calculation unit for the detected moving body calculates the predicted arrival time is a predicted time to arrive,
An alarm determination unit that determines whether or not an alarm is required based on the calculated predicted arrival time in the processing in the remote area;
An in-vehicle information processing system comprising: a moving body determination unit that determines whether or not an alarm is required based on the calculated optical flow in the process in the vicinity region.

Images