JP2007149077A - Video processing method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a driver of an automobile to easily grasp positional relation between his own vehicle and a road, etc. shown by a video image. <P>SOLUTION: This video processing method includes: a receiving step of receiving video data of a cross road crossing a running road of the own vehicle; a conversion step of executing shearing conversion to the video data at a shearing angle determined based on a cross angle between the cross road and the running road regarding the received video data and storing the video data in a storage device and a display step of generating display video data from the video data after the shearing conversion stored in the storage device and displaying the display video data on a display device. Thus, the driver is enabled to easily grasp a cross road in what direction the video image is about and it leads to safe driving by inclining the video image at the shearing angle determined based on the cross angle between the cross road and the running road. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for generating video data to be presented to an automobile driver.

  Conventionally, road reflectors have been installed at intersections with poor visibility so that vehicles, pedestrians, and the like can mutually confirm and call attention. However, there are problems such as confusing the positional relationship and direction because the left and right images are reflected on the road reflector, the reflection image is distorted due to the convex mirror, and there are blind spots that cannot be projected due to restrictions on the installation position and orientation. .

  In order to solve such a problem of the road reflector, for example, Japanese Patent Laid-Open No. 2001-155297 discloses an image of a blind spot position on a road to a moving body to reduce the blind spot position, so that the driver can Techniques have been disclosed for enabling confirmation more quickly and during the required time. Specifically, it is equipped with a solar cell and photographing means, installed in a monitoring area such as a road intersection, and transmitting means for wirelessly transmitting a video signal from the photographing means, and a video signal from the transmitting means that is equipped on a moving body. Receiving means. Then, when the moving body enters the reception range of the video signal, the receiving means of the moving body automatically receives the video signal, so that the driver can check the road blind spot position and operate. However, there is no particular idea about the content of the video presented to the driver.

Japanese Patent Application Laid-Open No. 2003-109199 discloses a vehicle-to-vehicle, a vehicle-to-bicycle, and a vehicle at an intersection with particularly poor visibility by providing an image of a road situation in the vicinity of the intersection to a vehicle occupant. A vehicle accident prevention system that can prevent a vehicle accident such as an interpersonal accident and improve the convenience of transportation is disclosed. Specifically, cameras C1 and C2 are installed to photograph priority roads at intersections including priority roads and non-priority roads, and light projectors / receivers are provided to provide video information to vehicles entering the intersections on the non-priority roads. Is installed. Images taken by the cameras C1 and C2 are transmitted as image information from the projector / receiver via the image providing device, and the image information received by the in-vehicle device mounted on the vehicle is simply displayed on the display unit as shown in FIG. They are displayed side by side. In FIG. 31, hatched squares schematically show the vehicle. Even in this publication, there is no particular idea about the display contents.
JP 2001-155297 A JP 2003-109199 A

  However, in the technique disclosed in the above publication, the road image is simply displayed on the monitor in the vehicle as it is, so at an intersection where the driver is not familiar with the road, it is impossible to instantaneously determine which direction the image is taken. There is a problem. That is, in the case of a road reflector, the driver can grasp the positional relationship between himself and the reflected image with reference to the road reflector, but the technique disclosed in the above publication has no reference, and the driver and video It is difficult to grasp the positional relationship with other roads.

  Accordingly, an object of the present invention is to provide a video processing technique for facilitating the driver of a car to grasp the positional relationship between the vehicle and a road or the like shown on the display screen.

  The video processing method according to the first aspect of the present invention includes a reception step of receiving video data of an intersection road that intersects with the traveling road of the host vehicle, and an intersection angle between the intersection road and the traveling road according to the received video data. The image data is subjected to shear transformation at a shear angle determined based on the transformation step for storing in the storage device, and display video data is generated from the video data after the shear transformation stored in the storage device, A display step for displaying. By tilting the video at the shear angle determined based on the crossing angle between the crossing road and the traveling road in this way, the driver can easily grasp the video about the crossing road in which direction, It leads to safe driving.

  Further, the conversion step described above includes an intersection angle calculating step for calculating an intersection angle between the intersection road and the traveling road related to the received video data, and an inclination angle of the intersection road related to the video data in the received video data. An inclination angle calculation step to calculate, a shear angle calculation step to calculate data related to the shear angle based on the intersection angle and the inclination angle, and a shear transformation is performed on the video data using the data related to the calculated shear angle, Storing in the storage device. By introducing the inclination angle in this way, a more appropriate shear angle can be derived.

  Further, when there are cross roads on both sides of the driving road, the display step described above arranges the display video data related to the left cross road on the left side and the display video data related to the right cross road to the right side. May be included in the step. In this way, for example, at the intersection of a crossroad, an image with the road tilted on the right side is displayed on the right side, and an image with the road tilted on the left side is displayed on the left side. It becomes possible to grasp the positional relationship with the vehicle.

  The video processing method according to the second aspect of the present invention includes a step of receiving video data in the traveling direction of the host vehicle including an opposite lane of the traveling road of the host vehicle, and a host vehicle position in the coordinate system of the received video data. And a display step of generating display image data by arranging the video data and a mark for representing the host vehicle while maintaining the positional relationship in the coordinate system, and displaying the display image data on the display device. Thus, for example, when making a right turn at an intersection, it is possible to check the oncoming vehicle on the image while grasping the position of the host vehicle with the mark, and to make a right turn safely.

  The second aspect of the present invention may further include a step of specifying the traveling direction of the host vehicle in the coordinate system of the received video data. At that time, the display step described above may include a step of generating a mark facing the traveling direction.

Further, in the second aspect of the present invention, the display step described above includes a right turn arrow from the own vehicle position in the display image data to a predetermined position on the right road of the own vehicle that intersects the traveling road of the own vehicle. A step of generating and arranging on the display image data may be included. This makes it easier to grasp the direction of travel when turning right.
A program for causing the computer to execute the video processing method can be created, and the program is stored in a storage medium or storage device such as a flexible disk, a CD-ROM, a magneto-optical disk, a semiconductor memory, or a hard disk. Is done. Moreover, it may be distributed as a digital signal via a network or the like. The intermediate processing result is temporarily stored in a storage device such as a main memory.

  ADVANTAGE OF THE INVENTION According to this invention, the driver | operator of a motor vehicle can make it easy to grasp | ascertain the positional relationship between the own vehicle and the road etc. which are shown with an image | video on a display screen.

[Embodiment 1]
FIG. 1 shows an outline of a road and a system according to the first embodiment of the present invention. In FIG. 1, the non-priority road R100 (including the lower portion R100a and the upper portion R100b of the non-priority road R100) and the priority road R101 (including the left portion R101a and the right portion R101b of the priority road R101) intersect. Indicates the state. On the priority road R101, the vehicle C3 is traveling from left to right on the left portion R101a, and the vehicle C4 is traveling from right to left on the right portion R101b. In addition, on the non-priority road R100, the vehicle C2 travels from the lower portion R100a from the bottom to the top, and the vehicle C1 travels from the top portion R100b to the bottom. The on-vehicle terminals 100 described below are mounted on the vehicles C1 to C4.

  A camera 4 for photographing the left portion R101a is installed on the right portion R101b of the priority road R101, and a camera 5 for photographing the right portion R101b is installed on the left portion R101a. The cameras 4 and 5 are connected to the video transmission device 3, and the video transmission device 3 is connected to antennas 1 and 2 for transmitting video data and the like wirelessly. Such a road side system composed of the cameras 4 and 5, the video transmission device 3, and the antennas 1 and 2 is the same as the conventional one except for the contents of the transmission data.

  FIG. 2 shows a functional block diagram of the in-vehicle terminal 100 mounted on the vehicles C1 to C4. The in-vehicle terminal 100 is mounted on the position detection unit 11 such as a GPS (Global Positioning System) that detects the vehicle position, the position data storage unit 12 that stores the position data detected by the position detection unit 11, and the in-vehicle terminal 100. A speed detection unit 13 that acquires speed data from a speedometer of the vehicle that is being used, a speed data storage unit 14 that stores speed data acquired by the speed detection unit 13, and position data stored in the position data storage unit 12 Based on the travel direction determination unit 15 for determining the travel direction of the vehicle, the travel direction data storage unit 16 for storing the travel direction data determined by the travel direction determination unit 15, the image data from the antenna 1 or 2, etc. The received video data receiving unit 17, the video data storing unit 18 for storing the video data received by the video data receiving unit 17, and the position data A shear angle calculation unit 19 for calculating an angle (shear angle) of shear transformation performed on the video based on data stored in the data storage unit 12, the traveling direction data storage unit 16, and the video data storage unit 18, Using the data stored in the shear angle data storage unit 20 for storing the data of the shear angle calculated by the shear angle calculation unit 19, and the data stored in the shear angle data storage unit 20 and the video data storage unit 18, the image is sheared. A video conversion unit 21 that performs shearing conversion according to a corner, a post-conversion video data storage unit 22 that stores video data after shear conversion, and video data stored in the post-conversion video data storage unit 22 for display And a display device 24 for displaying the output of the video display processing unit 23.

  In the present embodiment, for example, in the vehicle C2 that moves from the bottom to the top on the non-priority road R100, a screen as shown in FIG. In the example of FIG. 3, two images 201 and 202 are displayed on the display screen. The left image 201 shows the left portion R101a of the priority road R101 and the vehicle C3 traveling on the priority road R101. Compared with the conventional video (left side) shown in FIG. 25, in the present embodiment, it can be seen that the left portion R101a of the priority road R101 is tilted to the left. Similarly, the right image 202 shows the right portion R101b of the priority road R101 and the vehicle C4 traveling on the priority road R101. Compared with the conventional video (right side) shown in FIG. 25, in the present embodiment, it can be seen that the right portion R101b of the priority road R101 is tilted to the right.

  Further, for example, in the vehicle C1 moving from the top to the bottom on the non-priority road R100, a screen as shown in FIG. In the example of FIG. 4, two images 203 and 204 are displayed on the display screen. The left image 203 shows the right portion R101b of the priority road R101 and the vehicle C4 traveling on the priority road R101. It can be seen that the right portion R101b of the priority road R101 is tilted to the left. Similarly, the right image 204 shows the left portion R101a of the priority road R101 and the vehicle C3 traveling on the priority road R101. It can also be seen that the left portion R101a of the priority road R101 is tilted to the right. As described above, in FIGS. 3 and 4, the images are reversed left and right, and the direction of tilting is also reversed because the traveling direction is the opposite direction.

  In this way, by tilting the road in the video according to the intersection angle between the priority road R101 and the non-priority road R100, it is easy to grasp the direction of the road intersecting with the road on which the host vehicle travels and the positional relationship of the road. ing. In FIG. 1, the priority road R101 and the non-priority road R100 are orthogonal to each other. However, as shown in the center of FIG. 5, for example, the priority road R101 and the non-priority road R100 intersect at the intersection angle θ on the left hand side on the left side. On the right side, it is assumed that they intersect at an intersection angle Φ on the right hand side. Note that Φ> θ. Here, as shown on the left side of FIG. 5, the in-vehicle terminal 100 of the vehicle C2 receives the video data 211 photographed by the camera 4, and is left by an angle corresponding to the crossing angle θ on the left hand side with respect to the video. The image data 212 is generated by performing shear transformation on the image and displayed on the display device. Similarly, as shown on the right side of FIG. 5, the video data 213 photographed by the camera 5 is received, and the video data 214 is subjected to shear conversion on the right side by an angle corresponding to the right intersection angle Φ. Is generated and displayed on the display device. Comparing the video data 212 and the video data 214, since Φ> θ, the slope of the right portion R101b of the priority road R101 in the video is larger than that of the left portion R101a of the priority road R101 in the video. ing. Further, by displaying side by side as shown in FIGS. 3 and 4, it becomes possible to easily grasp the situation of the intersection including the intersection state of the road. This promotes safe driving. Even if the display as shown in FIG. 25 is performed, the direction of the road and the positional relationship of the road cannot be grasped instantaneously, and the effect of the display is weakened.

  In addition, in the case of the intersection as shown on the left side of FIG. That is, the non-priority road R100 on which the vehicle C2 is traveling intersects the right portion R101b of the priority road R101 on which the vehicle C4 is traveling at a relatively large intersection angle, and the road R102 on which the vehicle C5 is traveling. Intersects with a relatively small crossing angle. However, both are roads on the right hand side of the vehicle C2. Therefore, the image captured by the camera 4 and including the vehicle C5 and the road R102 is shear-transformed so that a relatively small angle corresponding to a relatively small intersection angle, that is, the road center line changes from the vertical to the display angle θ. The On the other hand, the image captured by the camera 5 and including the vehicle C4 and the road R101b is shear-transformed so that a relatively large angle corresponding to a relatively large intersection angle, that is, the road center line changes from the vertical to the display angle Φ. The Since Φ> θ, the driver can easily discriminate and judge even the same right crossing road if the video as shown on the right side of FIG. 6 is displayed.

The processing contents of the system shown in FIGS. 1 and 2 will be described below with reference to FIGS. First, the speed detection unit 13 detects the host vehicle speed from the speedometer of the host vehicle at predetermined time intervals and stores it in the speed data storage unit 14 (step S1). In the speed data storage unit 14, the vehicle speed is stored in time series. The position detection unit 11 detects the vehicle position at predetermined time intervals, stores the vehicle position data in the position data storage unit 12, and the traveling direction determination unit 15 is stored in the position data storage unit 12. The traveling direction vector is determined using the data of the position P _{Vt at} the current time t and the data of the position P _{Vt-1 at} the time (t−1), and is stored in the traveling direction data storage unit 16 (step S3). ). In the position data storage unit 12 and the traveling direction data storage unit 16, data on the vehicle position and the traveling direction are stored in time series together with the measurement time, for example.

  For example, the video data receiving unit 17 determines from the current vehicle speed data stored in the speed data storage unit 14 whether the host vehicle is in a slow state where the vehicle is stopped or traveling at a predetermined speed or less ( Step S5). If the vehicle is traveling normally, displaying the video on the display device 24 will give unnecessary information to the driver, and the process returns to step S1 because it is dangerous.

  On the other hand, when it is determined that the host vehicle is stopped or slowing down, the video data receiving unit 17 receives the road video data captured by the camera 4 or 5 and transmitted by the antenna 1 or 2, and the video The data is stored in the data storage unit 18 (step S7). Further, the imaging parameter is received and stored in the video data storage unit 18 (step S9).

The video data storage unit 18 stores road images and shooting parameters as shown in FIGS. 8 to 10, for example. In the video data storage unit 18, road video (for example, video data as shown in FIG. 9 or FIG. 10) and shooting parameters are stored in time series. In the example of FIG. 8, only data at time t-1 and data at time t are shown, but data at other times may be held. The data at each time includes the data of each camera. Here, data of the first and second cameras are included. The data of each camera includes video data, camera installation position, camera shooting direction, angle of view, shooting time, _gazing point position, etc., for example, two real coordinates P _RnS and P _RnE on the center line of the shot road. , _And imaging parameters including coordinates I _RnS and I _RnE on the image corresponding to each of the two real coordinates. Note that the shooting parameter may include a flag indicating whether the road being shot is a priority road. Here, n represents a camera number.

  Next, the shear angle calculation unit 19 receives from the camera installation position included in the imaging parameters related to the received video data and the current traveling direction data stored in the traveling direction data storage unit 16. It is determined whether or not the image data is an image of an intersection road at the nearest intersection in the traveling direction of the host vehicle (step S11). This can be determined by whether or not the camera installation position exists in a predetermined area ahead of the traveling direction stored in the traveling direction data storage unit 16. Whether it is the latest or not may be determined by, for example, executing steps S7 to S11 several times to identify the closest one. For example, if the map data held by the navigation system or the like can be used, the camera is installed. It is determined whether it is installed at the nearest intersection from the position. Furthermore, when a flag indicating whether or not the shooting road is a priority road exists in the shooting parameters, based on the flag, it is determined whether or not the road according to the video data is a priority road. If so, it may be determined to use video data.

  If it is determined that the image is not an image of an intersection road at the nearest intersection in the traveling direction of the host vehicle, the process returns to step S7. On the other hand, if it is determined that the image is the image of the intersection road at the nearest intersection in the traveling direction of the host vehicle, the shear angle calculation unit 19 and the image conversion unit 21 perform the image conversion process, and the processing result is stored in the shear angle data storage unit. 20 and the converted video data storage unit 22 (step S13). The video conversion process will be described in detail later. By performing the video conversion process, the video 201 or 202 in FIG. 3 and the video 203 or 204 in FIG. 4 are generated.

  And the shear angle calculation part 19 judges whether all the video data required at the nearest intersection in the advancing direction of the own vehicle were received (step S15). Basically, it is assumed that there are left and right data in two directions, but in actuality, there are cases where there are two or more and cases where there is one. Therefore, for example, when there are a plurality of cameras installed at the same intersection, if the distribution timing of the video data is shifted for each camera, Steps S7 to S11 are performed a plurality of times, and the installation positions are taken by the same camera. When the video data at the time is received, it can be determined that all the video data has been received.

  If it is determined in step S15 that not all video data required at the nearest intersection in the traveling direction of the host vehicle has been received, the process returns to step S7. On the other hand, when it is determined that all necessary video data has been received at the nearest intersection in the traveling direction of the host vehicle, the video display processing unit 23 converts the converted video stored in the converted video data storage unit 22. The data is arranged according to the intersection relationship between the traveling road and the intersection road (step S17) and displayed on the display device 24 (step S19). For example, display data as shown in FIG. 3 or 4 is generated and displayed on the display device 24. That is, the converted video data including the left portion of the priority road is arranged on the left side, and the converted video data including the right portion of the priority road is arranged on the right side.

  The above processing is carried out until an end instruction is given (step S21). In steps S7 to S11, if the vehicle starts running before receiving the video data to be used, or if the video to be used cannot be received even after waiting for a predetermined time, the process returns to step S1 as a timeout. Like that.

  By carrying out the processing as described above, it is possible to present the state of the intersection road that is difficult to see directly at the intersection to the driver in a manner that allows easy understanding of the intersection state and positional relationship of the road. Become.

Next, the video conversion process in step S13 will be described in detail with reference to FIGS. The shear angle calculation unit 19 stores data on the traveling direction V _V of the host vehicle stored in the traveling direction data storage unit 16 and two actual coordinates P on the road stored in the video data storage unit 18. The intersection angle between the traveling road and the intersection road is calculated from _RnS and P _RnE (n is a positive integer) and stored in a storage device such as a main memory (step S31).

Traveling direction vector V _V, as shown in FIG. 12, the traveling direction setting unit 15, a vector from the vehicle position P _Vt-1 at time t-1 to the vehicle position P _Vt at time t. Further, a photographing parameters associated with the video data including the right-hand portion R101b priority road R101 by two real coordinates P _R1S and P _R1E, first road outflow direction vector V from the real coordinate P _R1S to real coordinate P _{R1E Specify R1} . Similarly, the second road outflow direction vector from the real coordinates P _R2S to the real coordinates P _R2E from the two real coordinates P _R2S and P _R2E that are the shooting parameters associated with the video data including the left portion R101a of the priority road R101. _Specify V _R2 .

Then, the intersection angle is calculated by vector calculation of the traveling direction vector V _V and V _R1 or V _R2 . Specifically, θ is calculated based on the following equation.
cos θ = V _V · V _Ri / (| V _V || V _Ri |)

Incidentally, the traveling direction vector V _V positive in the counterclockwise relative to, as shown in FIG. 13 (a), the crossing angle theta ₂ between the traveling direction vector V _V and a second road runoff vector V _R2 is It can be seen that 0 <θ ₂ <180 and the left portion R101a of the priority road R101 is present on the left hand side of the host vehicle. As shown in FIG. 13B, the intersection angle θ ₁ between the traveling direction vector V _V and the first road outflow vector V _R1 is 0> θ ₁ > −180, and the priority road R101. It can be seen that the right portion R101b of the vehicle is present on the right hand of the host vehicle.

  Next, the shear angle calculation unit 19 calculates the display angle β based on the crossing angle θ, and stores it in a storage device such as a main memory (step S33). At this time, for example, the crossing angle θ is converted into the display angle β using a conversion straight line as shown in FIG. The display angle β is the tilt angle of the center line in the final video. In the graph of FIG. 14A, the horizontal axis represents the crossing angle θ (°), and the vertical axis represents the display angle β (°). In this embodiment, the display angle β is the crossing angle θ. Proportional. However, since the video is distorted if the display angle β is too large, the proportionality coefficient is set to be small. As described above, since the counterclockwise angle is positive, the right half of the horizontal axis of the graph in FIG. 14A indicates the left hand as the crossing angle. Further, the left half of the horizontal axis of the graph indicates the right hand as the crossing angle.

  If conversion using such a conversion line is performed, as shown in FIG. 14B, when the crossing angle θ increases in the positive direction, the display angle β in the video increases in the left-hand direction. However, as described above, since the distortion increases when the display angle β is excessively large, the display angle β is not so large as compared with the crossing angle θ. Similarly, when the crossing angle θ increases in the negative direction, the display angle β in the video increases in the right-hand direction.

Returning to the description of FIG. 11, the shear angle calculation unit 19 calculates the inclination angle α of the road in the video from the road image coordinates I _RiS and I _RiE in the video stored in the video data storage unit 18. For example, it is stored in a storage device such as a main memory (step S35). As shown in FIG. 15, a vector V _IV from the road image coordinate I _RiS on the center line to the road image coordinate I _RiE is specified, and an inclination angle α between the vertical line 301 in the video and the vector V _IV is calculated.

  Then, the shear angle calculation unit 19 calculates the shear angle γ or tan γ from the display angle β stored in step S33 and the display angle α stored in step S35, and stores it in the shear angle data storage unit 20 (step S36). As shown in FIG. 16 (a), when the road is inclined by the inclination angle α in the captured image, the shear angle γ is set so that the road is inclined by the display angle β as shown in FIG. 16 (b). decide.

  A method of calculating the shear angle γ will be described with reference to FIG. In FIG. 17, the height of a thin solid line rectangle representing the outer shape of the video before conversion is h, and a thin dotted line 401 represents the center line of the road in the video. The center line is inclined by an inclination angle α. If this thin solid rectangle is tilted by the shear angle γ, it becomes a thick parallelogram. As a result, the center line 401 of the road in the video is converted into a thick dotted line 402. The thick dotted line 402 is inclined by the display angle β. In addition, since it is a shearing conversion, the horizontal length does not change, so a before conversion becomes a after conversion. Specifically, the length a between the side of the parallelogram after conversion and the thick dotted line 402 at the height h is the distance between the side of the rectangle before conversion and the thin dotted line 401 at the height h. The length a is between.

In such a case, the following equation is established.
a = h × tan α
b = h × tan β
tan γ = c / h = (ba) / h
= (H × tan β−h × tan α) / h
= Tan β-tan α
γ = atan (tan β−tan α)

上記の式において、ｈは映像の高さを表し、横方向をｘ方向、縦方向をｙ方向とする。 Then, the video conversion unit 21 performs shear conversion on the video data using the video data stored in the video data storage unit 18 and the shear angle γ or tan γ stored in the shear angle data storage unit 20. Then, the video data after the shear transformation is stored in the post-conversion video data storage unit 22 (step S37). The following matrix transformation is performed.

In the above formula, h represents the height of the video, the horizontal direction is the x direction, and the vertical direction is the y direction.

Then, the video conversion unit 21 cuts out a rectangular region including the road portion to the maximum from the video data after shear conversion stored in the post-conversion video data storage unit 22, and expands it together with the display region (step S39). At the time of clipping, the largest rectangular area including I _RnS and I _RnE stored in the video data storage unit 18 and inscribed in the video after the shear transformation is specified. The process of this step is schematically shown in FIGS. 18 (a) to 18 (c). In FIG. _18A , a specific region 601 including I _RnS and I _RnE is cut out from the image after shear transformation, and an intermediate image as shown in FIG. 18B is generated. Then, as shown in FIG. 18C, the display area is enlarged to a predetermined size. Then, the process returns to the original process.

  By performing the processing as described above, converted video data to be displayed is generated. If the arrangement shown in FIG. 3 or FIG. 4 is performed, important information about safety can be presented to the driver in an easy-to-understand manner.

[Embodiment 2]
Next, a second embodiment will be described. In 2nd Embodiment, it is related to the process of the vehicle-mounted terminal at the time of a right turn. As shown in FIGS. 19A and 19B, when the host vehicle C502 moves from the position C502a to the position C502b and makes a right turn at a crossroad or the like, the situation where the oncoming vehicle C501 is traveling is photographed by the camera 503. . Video data captured by the camera 503 is wirelessly notified to the host vehicle C502 by a video transmission device (not shown). At this time, the video imaged by the camera 503 is the video image as shown in the upper part of FIG. 19A. However, there is a problem that it is difficult for the driver to know where the vehicle is located in the video image. Therefore, when the in-vehicle terminal is at the position C502a, it is converted into an image as shown in the middle part of FIG. 19A, and when it is at the position C502b, it is converted into an image as shown in the lower part of FIG. . A mark 511 is a marker indicating the vehicle position and the traveling direction, and is displayed as an overlay on the converted image described below.

  In this way, by displaying images such as the middle and lower in FIG. 19A on the display device of the in-vehicle terminal, the driver can easily grasp the relative positional relationship in the image.

  Next, FIG. 20 shows a functional block diagram of the in-vehicle terminal 500 that displays such an image. The in-vehicle terminal 500 is installed in a position detection unit 521 such as a GPS (Global Positioning System) that detects a vehicle position, a position data storage unit 522 that stores position data detected by the position detection unit 521, and the in-vehicle terminal 500. A speed detection unit 525 that acquires speed data from a speedometer of the vehicle that is being used, a speed data storage unit 535 that stores speed data acquired by the speed detection unit 525, and position data stored in the position data storage unit 522 Based on the travel direction determination unit 523 that determines the travel direction of the vehicle, the travel direction data storage unit 524 that stores the data of the travel direction determined by the travel direction determination unit 523, and the turn indicator instructs the right or left turn. It is acquired by the direction indicator state acquisition unit 526 that acquires data about whether or not it is present, and the direction indicator state acquisition unit 526 A direction indicator state data storage unit 527 for storing the direction indicator state data received, a video data reception unit 528 for receiving video data taken by the camera 503, and video data received by the video data reception unit 528. The video data storage unit 529, the traveling direction data storage unit 524, the video data storage unit 529, and the data stored in the position data storage unit 522 are used to perform perspective transformation and are stored in the video data storage unit 529. A perspective conversion unit 530 that calculates the vehicle position and the traveling direction angle in the image being viewed, a position and angle data storage unit 531 that stores data of the position and angle calculated by the perspective conversion unit 530, and position and angle data Affine transformation of video received using data stored in storage unit 531 and video data storage unit 529 An affine transformation unit 532 to be implemented, an affine transformation video data storage unit 533 that stores video data that has been affine transformed by the affine transformation unit 523, and an overlay of markers that represent the vehicle position and angle with respect to the video data after the affine transformation An overlay display processing unit 534 that performs display and a display device 536 that displays a processing result of the overlay display processing unit 534 are included.

Next, processing of the in-vehicle terminal 500 shown in FIG. 20 will be described using FIG. 21 to FIG. First, the direction indicator state acquisition unit 526 acquires, as the direction indicator state data, data about whether a right turn or a left turn is instructed from a direction indicator or the like, and the direction indicator state data is the direction indicator state data. Store in the storage unit 527 (step S41). The turn indicator state data storage unit 527 holds data for turning left or on, or turning right or left. Further, the speed detection unit 525 detects the own vehicle speed at predetermined time intervals from a speedometer of the own vehicle and stores the detected vehicle speed in the speed data storage unit 535 (step S43). In the speed data storage unit 535, the own vehicle speed is stored in time series together with, for example, the measurement time. The position detection unit 521 detects the vehicle position at predetermined time intervals, stores the vehicle position data in the position data storage unit 522, and the traveling direction determination unit 523 is stored in the position data storage unit 522. The traveling direction vector is determined using the data of the position P _{Vt at} the current time t and the data of the position P _Vt-1 at time (t−1), and stored in the traveling direction data storage unit 524 (step S45). ). In the position data storage unit 522 and the traveling direction data storage unit 524, the data of the own vehicle position and the traveling direction are stored in time series with, for example, the measurement time.

  For example, the video data receiving unit 528 determines whether or not the direction indicator state data stored in the direction indicator state data storage unit 527 indicates a right turn instruction ON (step S47). If the right turn instruction is not ON, the process returns to step S41. On the other hand, if the state is a right turn instruction ON, it is determined from the current vehicle speed data stored in the speed data storage unit 535 whether the host vehicle is in a slow state where the vehicle is stopped or traveling at a predetermined speed or less. (Step S49). If the image is displayed on the display device 536 during normal driving, unnecessary information is given to the driver, and the process returns to step S41 because it is dangerous.

  On the other hand, when it is determined that the host vehicle is stopped or slowing down, the video data receiving unit 528 receives the road video data captured by the camera 503 and stores it in the video data storage unit 529 (step S51). ). Also, the imaging parameters are received and stored in the video data storage unit 529 (step S53). The data stored in the video data storage unit 529 is the same as that shown in FIG.

For example, the perspective conversion unit 530 includes a current position of the host vehicle stored in the position data storage unit 522, a travel direction stored in the travel direction data storage unit 524, and a camera stored in the video data storage unit 529. It is determined whether or not the road image data received in step S51 is an image of the oncoming lane closest to the traveling direction from the installation position and the actual road coordinates P _RnS and P _RnE (step S55). It can be seen that the distance is equal to or less than a predetermined threshold from the current vehicle position of the own vehicle and the installation position of the camera. When installation video data for a plurality of cameras is received, the above determination is made for the closest camera. Furthermore, from the actual road coordinates P _RnS and P _RnE , the current position of the _host vehicle and the traveling direction of the _host vehicle, the road vector specified by the actual road coordinates P _RnS and P _RnE and the vector of the traveling direction of the _host vehicle are parallel. Alternatively, it is confirmed whether the road actual coordinates P _RnS and P _RnE are substantially parallel and located in a region ahead of the current position of the host vehicle. If video data that satisfies the above-described conditions has not been received, the process returns to step S41.

When it is determined that the conditions as described above are satisfied, the perspective conversion unit 530 is based on the camera position, shooting direction, field angle, and gaze point position stored in the video data storage unit 529. Coordinate transformation (known perspective transformation) is performed on the vehicle position P _Vt stored in the position data storage unit 522, and a coordinate value I _Vt in the coordinate system in the video data is acquired. Further, coordinate conversion is similarly performed on the own vehicle position P _Vt-1 at time t-1 to obtain a coordinate value I _Vt-1 in the coordinate system in the video data, and from I _Vt-1 to I _An angle δ between the vector to _Vt and the horizontal direction in the video data is calculated (step S57). Data on the coordinate position I _Vt and the angle δ are stored in the position and angle data storage unit 531.

The process of this step will be described with reference to FIG. In the real coordinate system XY, the own vehicle C502 exists at the position P _Vt . The positional relationship between the oncoming vehicle C501 and the camera 503 is the same as that in FIG. In this state, if the known perspective transformation is performed on the position P _Vt based on the data as described above, the coordinate value I _Vt of the own vehicle in the coordinate system xy of the video data can be acquired. Furthermore, if a known perspective transformation is also performed at the position P _Vt−1 , the coordinate value I _Vt−1 of the own vehicle in the coordinate system xy of the video data can be acquired. Then, as shown in FIG. 22B, the angle δ formed by the vector from the coordinate values I _Vt−1 to I _Vt and the horizontal line in the image data can be specified.

Next, the affine transformation unit 532 performs affine transformation on the video data stored in the video data storage unit 529 with reference to the vehicle position I _Vt , and converts the converted video data into the affine transformation video data storage unit 533. (Step S59). Specifically, as shown in FIG. 23A, even if the video data 601 and the vehicle position I _Vt are displayed at the same scale, the size of the display screen 701 is exceeded. Therefore, translating the image data 601 as while maintaining the positional relationship between the image data 601 and the vehicle position I _Vt is the vehicle position I _Vt comes at the bottom of the display screen 701. Thereafter, as shown in FIG. 23B, the video data 601 is reduced so as to enter the display screen 701, and the position I _{Vt ′} on the display screen 701 corresponding to the own vehicle position I _Vt is set in accordance with the reduction. calculate. The converted video data 601 ′ and position I _{Vt ′} data are stored in the affine transformation video data storage 533.

After that, the overlay display processing unit 534 displays the vehicle mark that is tilted by the angle δ stored in the position and angle data storage unit 531 with respect to the video data after the affine transformation stored in the affine transformation video data storage unit 533. Overlaying at the position I _{Vt ′} is displayed on the display device 536 (step S61). Accordingly, as shown in FIG. 24, the vehicle mark 611 for representing the positional relationship in the space in the video data received from the camera 503 is displayed in an inclined form by an angle δ. Thereafter, the process returns to step S41 until the process is completed (step S63).

  When the display as shown in FIG. 24 is performed by the processing as described above, the driver can grasp the positional relationship between the own vehicle position and the oncoming vehicle more accurately and easily only by the display contents. If the mark is not attached, the driver may take time to grasp the vehicle position, but according to the present embodiment, the driver can grasp instantly. Further, since the direction in which the vehicle is facing is also shown, it is easy to grasp the traveling direction based on the road condition and the relationship with the oncoming vehicle. However, it is not necessary to change the angle of the mark.

[Embodiment 3]
In the second embodiment, an example in which the own vehicle mark 611 is displayed in an overlay manner has been shown. However, if the driver is not guided to the geography near the intersection, the display content is displayed only by the own vehicle mark 611. It may be difficult to grasp the correspondence with the actual geography. In order to cope with such a case, it is easy to grasp the actual road conditions and the positional relationship of landmarks etc. from the displayed contents by appropriately overlaying the arrow to the road to the right turn, and landmarks etc. It is. In addition, if you know where you are going, you will not be confused when you turn right even in unfamiliar lands.

The main processing contents in the present embodiment are similar to those in the second embodiment. However, the data received from the camera 503 in step S53 of FIG. 21 is not the data shown in FIG. 8 but data as shown in FIG. In FIG. 25, data 1001 is data from the first camera that is additionally distributed in the present embodiment, and is the first camera right turn point image coordinate I _pin1 , the first camera mark image 1, and the first camera mark image. 1 image coordinate I _m1 , first camera mark image 2, and first camera mark image 2 image coordinate I _m2 are included. The first camera right turn point image coordinate I _pin1 is the coordinate of a specific point (for example, the center) of the right turn destination road in the video. The first camera mark images 1 and 2 are image data representing respective landmarks, destinations and road names. The first camera mark image 1 image coordinate _Im1 is a coordinate on the video where the first camera mark image 1 is to be arranged. Similarly, the first camera mark image 2 image coordinate _Im 2 is a coordinate on the video where the first camera mark image 2 is to be placed. The data 1002 is the same as that for the second camera, the data 1003 is the same as the data 1001, and the data 1004 is the same as the data 1002. The data about the mark image is not limited to two sets, and may be more or less.

Further, steps S57 to S61 in FIG. 21 are replaced with a processing flow as shown in FIG. Steps S57 and S61 are the same as those in FIG. After step S57, the affine transformation unit 532 performs affine transformation on the video data or the like stored in the video data storage unit 529 with reference to the vehicle position I _Vt , and converts the converted video data to affine transformation video data. The affine transformation video data storage section is specified by storing the vehicle position I _{Vt ′} after the affine transformation, the right turn point position I _{pin ′} after the affine transformation, the mark position I _{m1 ′} after the affine transformation, etc. It stores in 533 (step S71).

For example, as shown in FIG. 27A, it is assumed that video data 601, own vehicle position I _Vt , right turn point image coordinate I _pin , mark image 1 image coordinate I _m1, and mark image 2 image coordinate I _m2 are arranged. The video data 601 is reduced so as to enter the display screen 701 while maintaining the positional relationship such as the video data 601 and the vehicle position I _Vt, and each coordinate is calculated in accordance with the reduction. That is, the position I _{Vt ′} on the display screen 701 corresponding to the vehicle position I _Vt , the position I _{pin ′} on the display screen 701 corresponding to the right turn point image coordinate I _pin , and the mark image 1 image coordinate I _m1 to position I _m1 on the display screen 701 _', and corresponds to the mark image 2 image coordinates I _m2, the position I _m2 on the display screen _701' obtained. As a result, the positional relationship as shown in FIG. 27B is obtained. Note that the mark image 1 image coordinate I _m1 and the mark image 2 image coordinate I _m2 may not exist, and in this case, conversion is not performed.

Next, the overlay display processing unit 534 generates a curve from the vehicle position I _{Vt ′} after the affine transformation to the right turn point I _{pin ′} after the affine transformation using, for example, a Bezier curve or a spline curve, and an arrow based on the curve Is overlaid on the video after the affine transformation and displayed on the display device 536 (step S73). Specifically, a curve is generated as shown in FIG. 28, an arrow 621 along the curve is generated, and an overlay display is performed on the video after affine transformation as shown in FIG.

Further, the overlay display processing unit 534 overlays the mark images corresponding to the mark positions I _{m1 ′} and I _{m2 ′} after the affine transformation on the video after the affine transformation, and displays them on the display device 536 (step S75). That is, as shown in FIG. 30, the mark 623 is overlaid on the mark position I _{m1 ′} after the affine transformation, and the mark 624 is overlaid on the mark position I _{m2 ′} after the affine transformation.

  Finally, the overlay display processing unit 534 performs step S61 as in the second embodiment. Then, an image as shown in FIG. 30 is generated and displayed on the display device 536. Note that if overlaying is performed in the order described above, display is performed correctly.

  By performing the display as shown in FIG. 30, the user's own position can be grasped by the own vehicle mark 611, the right turn direction is indicated by the arrow 621, the landmark is indicated by the mark 624, and the destination and road name in the right turn direction are indicated by the mark 623. Because it can be grasped, even on the first road, it will be possible to drive without any trouble when turning right. However, landmarks, destinations and road names may not exist.

  Although the embodiment of the present invention has been described above, the present invention is not limited to this. For example, the functional block diagram is an example, and a program module and a functional block do not necessarily correspond.

  Further, the processing flow can be changed as long as the processing result does not change. For example, in FIG. 7 and FIG. 21, the reception of the road video data and the shooting parameters is performed when the condition is satisfied, but may be always received.

(Appendix 1)
A receiving step for receiving video data of an intersecting road that intersects with the traveling road of the vehicle;
A conversion step of performing shear conversion on the video data at a shear angle determined based on an intersection angle between the intersecting road related to the received video data and the traveling road, and storing the data in a storage device;
A display step of generating display video data from the video data after the shear transformation stored in the storage device and displaying the display video data on a display device;
Video processing method.

(Appendix 2)
The converting step comprises:
A crossing angle calculating step of calculating a crossing angle between the crossing road related to the received video data and the traveling road;
An inclination angle calculating step of calculating an inclination angle of an intersection road related to the video data in the received video data;
A shear angle calculating step of calculating data on the shear angle based on the intersection angle and the inclination angle;
Performing a shear transformation on the video data using the calculated data relating to the shear angle, and storing in a storage device;
The video processing method according to appendix 1, which includes:

(Appendix 3)
The shear angle calculating step includes:
A display angle calculation step of calculating a display angle according to the intersection angle;
Calculating data relating to the shear angle from the display angle and the tilt angle;
The video processing method according to appendix 2, including:

(Appendix 4)
In the receiving step, further receiving data representing the outflow direction of the crossing road related to the video data together with the video data,
The intersection angle calculating step includes:
Identifying the traveling direction of the host vehicle;
Calculating a crossing angle from the traveling direction and the outflow direction of the crossing road;
The video processing method according to appendix 2, including:

(Appendix 5)
In the receiving step, further receiving data indicating the direction in the video of the intersection road related to the video data together with the video data;
The video processing method according to claim 2, wherein, in the tilt angle calculating step, the tilt angle is calculated from data representing a direction in the video.

(Appendix 6)
The video processing method according to claim 3, wherein, in the display angle calculation step, the intersection angle is converted into a display angle according to a predetermined function.

(Appendix 7)
The displaying step comprises:
When the intersection road exists on both sides of the traveling road, the display image data relating to the left intersection road is arranged on the left side, and the display image data relating to the right intersection road is arranged on the right side. The video processing method according to appendix 1, which includes:

(Appendix 8)
The displaying step comprises:
The video processing method according to claim 1, further comprising a step of cutting the video data after the shear transformation into a rectangle or a square and performing an enlargement or reduction process.

(Appendix 9)
The video processing method according to claim 1, wherein the processing after the receiving step or the converting step is executed when the host vehicle speed is equal to or less than a threshold value.

(Appendix 10)
The video processing method according to supplementary note 1, wherein when the travel road is a non-priority road, the processing after the reception step or the conversion step is executed.

(Appendix 11)
Receiving means for receiving video data of an intersecting road that intersects the traveling road of the host vehicle;
Conversion means for performing shear transformation on the video data at a shear angle determined based on an intersection angle between the intersecting road related to the received video data and the traveling road, and storing the data in a storage device;
Display means for generating display video data from the video data after the shear transformation stored in the storage device, and displaying on the display device;
A video processing apparatus.

(Appendix 12)
Receiving video data of the traveling direction of the host vehicle, including the opposite lane of the traveling road of the host vehicle;
Identifying the vehicle position in the coordinate system of the received video data;
A display step of generating display image data by arranging the video data and a mark for representing the host vehicle while maintaining a positional relationship in the coordinate system, and displaying the display image data on a display device;
Video processing method.

(Appendix 13)
Further comprising the step of specifying the traveling direction of the host vehicle in the coordinate system of the received video data,
The displaying step comprises:
The video processing method according to appendix 12, including a step of generating a mark facing in the traveling direction.

(Appendix 14)
The displaying step comprises:
Additional remark 12 including the step of generating a right turn arrow from the own vehicle position in the display image data to a predetermined position on a road on the right side of the own vehicle that intersects the traveling road of the own vehicle, and placing the arrow on the display image data. The video processing method described.

(Appendix 15)
The displaying step comprises:
The video data and the host vehicle position so as to maintain a positional relationship in the coordinate system with respect to the video data, the host vehicle position, and a predetermined position of a road on the right side of the host vehicle that intersects the traveling road of the host vehicle. And affine transformation for the predetermined position;
Generating a right turn arrow from a curve connecting the vehicle position after the affine transformation and the predetermined position after the affine transformation, and placing the right turn arrow at the position of the curve on the display image data; ,
The video processing method according to appendix 12, including:

(Appendix 16)
Means for receiving video data of the traveling direction of the host vehicle, including an opposite lane of the traveling road of the host vehicle;
Means for specifying the vehicle position in the coordinate system of the received video data;
Display means for generating display image data by arranging the video data and a mark for representing the host vehicle while maintaining a positional relationship in the coordinate system, and displaying the display image data on a display device;
A video processing apparatus.

It is a figure showing the road and system outline | summary in the 1st Embodiment of this invention. It is a functional block diagram of the vehicle-mounted terminal in the 1st Embodiment of this invention. It is a figure which shows the example of the 1st display screen in the 1st Embodiment of this invention. It is a figure which shows the 2nd example of a display screen in the 1st Embodiment of this invention. It is the schematic diagram explaining the video conversion corresponding to an example of the road intersection in the first embodiment of the present invention. It is the schematic diagram demonstrated about the video conversion corresponding to the other example of the crossing aspect of a road in the 1st Embodiment of this invention. It is a figure which shows the processing flow of the main process in the 1st Embodiment of this invention. It is a figure which shows an example of the data stored in the video data storage part. It is a figure which shows an example of the image | video image | photographed with the camera of road installation. It is a figure which shows the other example of the image | video image | photographed with the camera of road installation. It is a figure which shows the processing flow of the video conversion process in the 1st Embodiment of this invention. It is a figure for demonstrating a road runoff vector. (A) And (b) is a figure for demonstrating the intersection angle of a advancing direction vector and a road outflow vector. (A) is a figure showing the conversion function at the time of calculating the display angle (beta) from an intersection angle, (b) is a figure which shows typically the relationship between an intersection angle and a display angle. It is a figure for demonstrating inclination | tilt angle (alpha). (A) is a figure showing inclination-angle (alpha), (b) is a figure showing the relationship between display angle (beta) and shear angle (gamma). It is a figure for demonstrating calculation of the shear angle (gamma). (A) thru | or (c) is a figure for demonstrating the process of cutting-out of an image | video, and expansion. (A) And (b) is a figure for demonstrating the 2nd Embodiment of this invention. It is a functional block diagram of the vehicle-mounted terminal in the 2nd Embodiment of this invention. It is a figure which shows the processing flow in the 2nd Embodiment of this invention. (A) And (b) is a figure for demonstrating the coordinate transformation in the 2nd Embodiment of this invention. (A) And (b) is a figure for demonstrating the affine transformation in the 2nd Embodiment of this invention. It is a figure which shows the example of a screen in the 2nd Embodiment of this invention. It is a figure which shows an example of the reception data in the 3rd Embodiment of this invention. It is a figure which shows the processing flow in the 3rd Embodiment of this invention. (A) And (b) is a figure for demonstrating the process in the 3rd Embodiment of this invention. It is a figure for demonstrating the process in the 3rd Embodiment of this invention. It is a figure for demonstrating the process in the 3rd Embodiment of this invention. It is a figure which shows the example of a screen in the 3rd Embodiment of this invention. It is a figure for demonstrating a prior art.

Explanation of symbols

C1, C2, C3, C4, C5 Vehicle 1, 2 Antenna 3 Video transmission device 4, 5 Camera 11, 521 Position detection unit 12, 522 Position data storage unit 13, 525 Speed detection unit 14, 535 Speed data storage unit 15. 523 Travel direction determination unit 16,524 Travel direction data storage unit 17,528 Video data reception unit 18,529 Video data storage unit 19 Shear angle calculation unit 20 Shear angle data storage unit 21 Video conversion unit 22 Converted video data storage unit 23 Video display processing unit 24 Display device 526 Direction indicator state acquisition unit 527 Direction indicator state data storage unit 530 Perspective conversion unit 531 Position and angle data storage unit 532 Affine conversion unit 533 Affine conversion video data storage unit 534 Overlay display processing unit 536 Display device

Claims (8)

A receiving step for receiving video data of an intersecting road that intersects with the traveling road of the vehicle;
A conversion step of performing shear conversion on the video data at a shear angle determined based on an intersection angle between the intersecting road related to the received video data and the traveling road, and storing the data in a storage device;
A display step of generating display video data from the video data after the shear transformation stored in the storage device and displaying the display video data on a display device;
Video processing method. The converting step comprises:
A crossing angle calculating step of calculating a crossing angle between the crossing road related to the received video data and the traveling road;
An inclination angle calculating step of calculating an inclination angle of an intersection road related to the video data in the received video data;
A shear angle calculating step of calculating data on the shear angle based on the intersection angle and the inclination angle;
Performing a shear transformation on the video data using the calculated data relating to the shear angle, and storing in a storage device;
The video processing method according to claim 1, further comprising: The displaying step comprises:
When the intersection road exists on both sides of the traveling road, the display image data relating to the left intersection road is arranged on the left side, and the display image data relating to the right intersection road is arranged on the right side. The video processing method according to claim 1, further comprising: Receiving means for receiving video data of an intersecting road that intersects the traveling road of the host vehicle;
Conversion means for performing shear transformation on the video data at a shear angle determined based on an intersection angle between the intersecting road related to the received video data and the traveling road, and storing the data in a storage device;
Display means for generating display video data from the video data after the shear transformation stored in the storage device, and displaying on the display device;
A video processing apparatus. Receiving video data of the traveling direction of the host vehicle, including the opposite lane of the traveling road of the host vehicle;
Identifying the vehicle position in the coordinate system of the received video data;
A display step of generating display image data by arranging the video data and a mark for representing the host vehicle while maintaining a positional relationship in the coordinate system, and displaying the display image data on a display device;
Video processing method. Further comprising the step of specifying the traveling direction of the host vehicle in the coordinate system of the received video data,
The displaying step comprises:
The video processing method according to claim 5, further comprising: generating a mark facing in the traveling direction. The displaying step comprises:
A step of generating a right turn arrow from the position of the host vehicle in the display image data to a predetermined position of a road on the right side of the host vehicle that intersects with a traveling road of the host vehicle and arranging the arrow on the display image data. 5. The video processing method according to 5. Means for receiving video data of the traveling direction of the host vehicle, including an opposite lane of the traveling road of the host vehicle;
Means for specifying the vehicle position in the coordinate system of the received video data;
Display means for generating display image data by arranging the video data and a mark for representing the host vehicle while maintaining a positional relationship in the coordinate system, and displaying the display image data on a display device;
A video processing apparatus.

Images