JP2013201707A - Image reception and display device, and image transmission and reception system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image reception and display device and an image transmission and reception system which accelerates driver's recognition of another vehicle and a pedestrian hidden at positions which become blind spots in a view from the vehicle driven by the driver.SOLUTION: The image reception and display device receives an image captured using a camera installed outside a vehicle such as on a road and an imaging parameter relating to the image, geometrically transforms the image, on the basis of the image parameter, to an image viewed from a predetermined position outside the vehicle, and displays the transformed image to the driver. An image capturing and transmitting device of an image to be displayed by the image reception and display device transmits to the image reception device the image data and the imaging parameter representing a situation in which the image data was captured.

Description

  The present invention relates to an image reception display device and an image transmission / reception system, and more particularly, to an image reception display device and an image transmission / reception system for displaying an image of a blind spot on a driver before the host vehicle approaches.

  As a background art of this technical field, there is Patent Document 1. In this publication, “an image processing unit that captures an image in front of a vehicle, an image processing unit that performs image processing on an image captured by the image capturing unit in association with a curved shape of a curve mirror, and the image processing unit are provided. Display means for displaying an image processed image on a mirror portion in a figure imitating a curve mirror ”(refer to claim 1). Here, the “imaging means” is described as “a camera as an imaging means attached to a vehicle body” (see paragraph 0019) in the embodiment of the publication.

Japanese Patent No. 4312258

When driving a vehicle, an accident may occur due to a delay in recognition of other vehicles or pedestrians lurking in the position corresponding to the blind spot when viewed from the own vehicle. For example, an accident is likely to occur at an intersection with poor visibility. .
In Patent Document 1, since a camera is installed on the vehicle body (especially the foremost part) of the host vehicle, for example, at an intersection with poor visibility, if the host vehicle is not advanced to the extent that the vehicle body approaches the intersection, The situation cannot be captured by the camera, and recognition is delayed.
Accordingly, an object of the embodiment of the present invention is to provide an image reception display device and an image transmission / reception system for displaying an image of a blind spot on a driver before the host vehicle approaches.

  In order to solve the above object, the configuration described in the claims is adopted.

  ADVANTAGE OF THE INVENTION According to this invention, the image receiving display apparatus and image transmission / reception system which display the condition of a blind spot on a driver before the own vehicle approaches can be provided, and there exists an effect that it can contribute to accident prevention.

1 is a configuration diagram of a vehicle image transmission / reception system in Embodiment 1 of the present invention. FIG. The figure which shows the example of the form of the image transmission / reception system for vehicles in Example 1 of this invention. The figure which shows the example of the positional relationship of the intersection periphery at the time of copying a blind spot using the conventional curve mirror. The figure which shows the example of the positional relationship of the periphery of the intersection in Example 1 of this invention. The figure which shows the coordinate before the geometric transformation of the received image data in Example 1 of this invention. The figure which shows the coordinate after the geometric transformation of the received image data in Example 1 of this invention. FIG. 3 is a diagram illustrating an example of an output image in Embodiment 1 of the present invention. The figure which shows the example of the positional relationship of the periphery of the intersection in Example 2 of this invention. The figure which shows the coordinate before the geometric transformation of the received image data in Example 2 of this invention. The figure which shows the coordinate after the geometric transformation of the received image data in Example 2 of this invention. FIG. 6 is a diagram illustrating an example of an output image in Embodiment 2 of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Hereinafter, Example 1 in the present invention will be described in detail.
FIG. 1 is a configuration diagram illustrating a vehicular image transmission / reception system 100 according to a first embodiment of the present invention.
The vehicular image transmission / reception system 100 includes an image capturing / transmitting device 110 and an image receiving / displaying device 120.

The image capturing / transmitting device 110 is a device installed at a place other than the host vehicle, such as on the road or other vehicle, and image data obtained by taking a wide-angle image of a surrounding image and an imaging parameter indicating a situation when the image data is captured. This is a device that wirelessly transmits the data to the image reception display device 120. The imaging parameters will be described in detail later.
The image capturing and transmitting apparatus 110 includes an optical system unit 111, an image sensor unit 112, a signal processing unit 114, an image compression unit 115, a position / orientation output unit 118, a transmission control unit 116, and a transmission unit 117.

  The optical system unit 111 is an optical system configured so that a wide-angle optical image around the apparatus is formed on the sensor surface of the image sensor unit 112. As an actual configuration of the optical system unit 111, for example, a configuration in which a fisheye lens is included in the optical system to form an optical image in a wide angle range can be taken. In the case of this configuration, an optical image with less distortion at the portion (center portion) close to the optical axis of the camera and greater distortion at the periphery can be obtained. The degree of distortion of the optical image is determined by the optical characteristics of the entire optical system 111. In this embodiment, a case using a fisheye lens will be described as an example, but the method of this embodiment can be applied regardless of the configuration of the optical system unit 111.

The image sensor unit 112 photoelectrically converts the optical image formed by the optical system unit 111, performs A / D conversion, and outputs the converted image data to the signal processing unit 114 as digital data. The distortion of the optical image generated by the optical system unit 111 is inherited as the distortion of the image in the output image data.
The signal processing unit 114 performs a series of processes such as black level correction, white balance correction, demosaicing, and gamma correction on the image data as necessary, and outputs the generated image data to the image compression unit 115.
The image compression unit 115 compresses the image data and outputs it to the transmission unit 117.

The position / orientation output unit 118 outputs information indicating the position and direction where the image capturing and transmitting apparatus 110 is installed to the transmission control unit 116. Any method may be used for confirming the position and direction. For example, a method of detecting a position using GPS (Global Positioning System), a method of detecting an azimuth by a magnetic sensor, and an output of the signal processing unit 114 A method of obtaining image data by image recognition, a method of inputting manually at the time of installation, and the like can be used, but methods other than those described above may be used.
The transmission control unit 116 receives information on the optical characteristics of the optical system unit 111 from the optical system unit 111, and receives information indicating the position and direction where the image capturing / transmitting device 110 is installed from the position / orientation output unit 118. Is output to the transmission unit 117 as data called imaging parameters indicating the situation when the image is captured.
The transmission unit 117 wirelessly transmits the image data output from the image compression unit 115 and the imaging parameter output from the transmission control unit 116.

  The transmission image processing unit 113 includes a signal processing unit 114 and an image compression unit 115. Generally, since the bandwidth required for wireless transmission can be saved, image compression is performed before transmission, but there may be a case where image compression is not performed to simplify the circuit. In that case, the image compression unit 115 may be deleted from the components of the transmission image processing unit 113 and the output of the signal processing unit 114 may be output to the transmission unit 117 as it is. In this case, the cost of the compression unit can be saved. Further, the transmission image processing unit 113 may have a configuration in which elements for performing other image processing such as image recognition are added. Further, the signal processing unit 114 may have a configuration in which an element for performing a geometric transformation process that reduces distortion of image data is added. When a geometric transformation process that changes image distortion is applied to the inside of the signal processing unit 114, the degree of image distortion described in the imaging parameters so as to represent the degree of distortion of image data to be transmitted. Is appropriately corrected in the transmission control unit 116 and transmitted.

The image receiving / displaying device 120 is a device installed in the host vehicle, and performs image processing based on imaging parameters that are also wirelessly transmitted on the image data wirelessly transmitted from the image capturing / transmitting device 110, so that an image is displayed to the driver. It is a device to display.
The image reception display device 120 includes a reception unit 121, an image decoding unit 123, a geometric conversion unit 124, a reception control unit 125, a display unit 126, and a position / orientation output unit 127.
The receiving unit 121 outputs the image data wirelessly transmitted from the image capturing / transmitting device 110 to the image decoding unit 123. In addition, the reception unit 121 outputs imaging parameters wirelessly transmitted from the image imaging transmission device 110 to the reception control unit 125.
The image decoding unit 123 decodes the image data output from the receiving unit 121 and outputs the decoded image data to the geometric transformation unit 124.

The position / orientation output unit 127 outputs information indicating the position and direction of the image reception display device 120 or the boarding position and direction of the driver of the host vehicle estimated from the position and direction to the reception control unit 125. Any method for confirming the position and orientation may be used. For example, a method for detecting a position using GPS (Global Positioning System), a method for detecting a change in position using an acceleration sensor, or a magnetic sensor. A method of detecting the orientation can be used, but other methods may be used.
The reception control unit 125 outputs the imaging parameter output from the reception unit to the geometric conversion unit 124.
The geometric conversion unit 124 performs a geometric conversion process based on the imaging parameter output from the reception control unit 125 and the position and direction output from the position / orientation output unit 127, and has a desired geometric characteristic. An image is created and output to the display unit 126.
The display unit 126 displays an image based on the image data output from the geometric conversion unit 124.

  The received image processing unit 122 includes, for example, an image decoding unit 123 and a geometric conversion unit 124. Generally, since the bandwidth of wireless transmission can be saved, image compression is performed before transmission, but there is a case where image compression is not performed for simplification of the circuit. In that case, the image decoding unit 123 may be deleted from the received image processing unit 122 and the output of the receiving unit 121 may be output to the geometric transformation unit 124 as it is. In this case, the circuit cost of the image decoding unit 123 can be suppressed, and image quality degradation when a compressed image is decoded can be prevented. The reception image processing unit 122 may have a configuration in which elements for performing other image processing such as image recognition are added. In addition, an element for superimposing a display screen for GUI (Graphical User Interface) may be added to the inside of the received image processing unit 122.

  As described above with reference to FIG. 1, the image capturing / transmitting device 110 transmits data called imaging parameters together with image data, and the image receiving / displaying device 120 applies geometric transformation processing to the image data based on the imaging parameters. . The image receiving / displaying device 120 can display any desired video by processing the image distortion generated by the image capturing / transmitting device 110 by using the imaging parameter in the geometric transformation process.

FIG. 2 is a diagram illustrating an example of a form of the vehicle image transmission / reception system 100. This figure is the figure which looked at the mode when the own vehicle 202 is moving forward approaching the intersection from the back of the own vehicle. In this figure, the image capturing and transmitting apparatus 110 shown in FIG. 1 is fixed on the road, for example, and the image receiving and displaying apparatus 120 shown in FIG.
In this figure, the image capturing / transmitting device 110 is installed at a high position so as to share the column 200 with the traffic light 201. By capturing a wide-angle image from such a high position, the situation around the intersection can be captured in a wide range. The image capturing / transmitting device 110 may be installed in a place other than that shown in FIG. 2, such as being fixed to a building, installed in a vehicle other than the host vehicle, or installed in an airplane.

The image reception display device 120 is mounted on the host vehicle 202, and in particular, the display unit 126 is installed at a position that is easy for the driver to see. The display unit 126 may be installed in any manner such as a form shared with a display of another in-vehicle device such as an in-vehicle car navigation system or a form projected on a part of the windshield.
As described above, the vehicular image transmission / reception system 100 in FIG. 1 is installed in a form as shown in FIG. In this embodiment, an image including the driver's blind spot condition can be captured by the image capturing device 110, and the image can be processed and displayed by the image receiving display device 120 so that the image can be easily viewed by the driver. This makes it easier for the driver to grasp the blind spot and prevent accidents.
From the next section, an example of an intersection with a poor view will be given as an example, and the problem of the conventional method and the superiority of this embodiment will be described with respect to the method of confirming the blind spot situation.

  FIG. 3 is a diagram illustrating an example of a conventional technique in which a blind mirror is confirmed from the host vehicle by installing a curve mirror at an intersection where visibility is poor. This figure shows a state immediately before the host vehicle 202 moves forward and enters an intersection. It is assumed that the pedestrian 303 and the other vehicle 304 correspond to the blind spots of the buildings (obstacles) 301 and 302 when viewed from the driver of the host vehicle 202. Even if the traveling direction signal 305 of the host vehicle is blue, the pedestrian 303 may forcibly cross the road, and the other vehicle 304 may overlook the signal and go straight ahead. There is a possibility of getting involved. However, the dangerous situation occurring in the blind spot cannot be recognized by the driver of the own vehicle 202 with the naked eye in advance.

As a conventional method for showing such a situation to the driver, for example, there is a method of installing a curve mirror 310. However, in the situation where the curve mirror 310 and the host vehicle 202 are separated as shown in FIG. 3, the situation of the intersection seen through the curve mirror 310 appears small from the driver of the host vehicle 202, so the driver recognizes the blind spot situation. Is difficult. Also, regarding the installation position of the curve mirror 310, of course, there are physical restrictions such as size, position, and angle, so it is not possible to freely select and arrange a position that can be easily viewed by the driver.
Further, even with the method of Patent Document 1, since the camera is installed particularly at the forefront of the host vehicle 202, the situation of the left and right of the intersection must be advanced unless the host vehicle is advanced to the extent that the vehicle body reaches the intersection. Cannot be captured by the camera, and recognition is delayed.

FIG. 4 is a diagram illustrating an example of a method for visually recognizing a blind spot when the method of the present embodiment is used. Under the same conditions as in FIG. 3, the own vehicle 202, a pedestrian 303, another vehicle 304, and buildings (obstacles) 301 and 302 exist near the intersection. In FIG. 4, an image capturing / transmitting device 110 is mounted on the signal 305 in the traveling direction of the host vehicle as in the case of the traffic light 201 in FIG. 2, and the image receiving display device 120 is mounted on the host vehicle 202 as in FIG. 2. It is installed. In the present embodiment, based on the image data captured by the image capturing and transmitting apparatus 110, the image receiving and displaying apparatus 120 provides the blind spot situation in a form that is easy for the driver to visually recognize.
First, in FIG. 4, assuming that the image capturing / transmitting device 110 is a wide-angle camera installed at a high place, all of the own vehicle 202, a pedestrian 303, another vehicle 304, and buildings (obstacles) 301 and 302 are imaged. It is assumed that it is installed with a possible angle of view. That is, the image data captured by the image capturing / transmitting device 110 includes information regarding the blind spot for the driver of the host vehicle 202.

  In this embodiment, first, a virtual mirror 406 and a virtual viewpoint 405 are defined in order to convert an image captured by the image capturing / transmitting device 110 into an image in which the driver can easily confirm the blind spot. The virtual mirror 406 is a virtual curve mirror having a predetermined curvature, and is assumed to be installed on the roadway at the position shown in FIG. 4, for example. The curvature, position, direction, and size of the virtual mirror 406 can be arbitrarily set. The virtual viewpoint 405 is a virtual camera that captures an image reflected on the virtual mirror 406, and is assumed to be installed at a position shown in FIG. The position of the virtual viewpoint 405 can be arbitrarily set, but the direction is assumed to be directed to the virtual mirror 406. In this embodiment, a virtual viewpoint 405 and a virtual mirror 406 are assumed, and a driver can recognize a wide-angle surrounding situation including a pedestrian 303 and another vehicle 304 as indicated by a virtual optical axis 407. For this reason, the display unit 126 of the image receiving display device 120 (FIG. 1 or FIG. 2), when the image reflected at the virtual mirror 406 and captured at the virtual viewpoint 405 is still not directly visible to the driver at the intersection blind spot. (FIG. 1).

  In the present embodiment, the actual image data captured by the image capturing / transmitting device 110 is subjected to a geometric conversion process in the geometric conversion unit 124 of the image receiving and displaying device 120, and the image and the geometric image captured at the virtual viewpoint 405 are used. Create and display scientifically corresponding image data. In the example of FIG. 4, the virtual viewpoint 405 is located in the traveling direction of the host vehicle 202 and exists in a direction that reflects the traveling direction of the host vehicle 202, so when the host vehicle approaches the point of the virtual viewpoint 405. When the virtual mirror 406 is viewed, the scenery seen from the driver's seat is displayed, so that the driver can easily grasp the positional relationship. Further, since the position closer to the intersection than the own vehicle 202 can be set as the viewpoint, the blind spot can be confirmed in a large image in advance. Therefore, accidents can be prevented beforehand.

The video viewed from the virtual viewpoint 405 illustrated in FIG. 4 is displayed on the display unit 126 through the geometric transformation process in the image reception display device 120 based on the image data captured by the image capturing / transmitting device 110. .
Hereinafter, an example of the geometric transformation process in the case of FIG. 4 will be described with reference to FIGS.
FIG. 5 shows an example of the coordinates of an image taken in the situation of FIG. 4 before geometric transformation. An output image range 501 of the image sensor unit indicates a range of an image output from the image sensor unit 112 of FIG. In the example of this figure, the optical system unit 111 is configured to include a fisheye lens, and as in the case of FIG. 4, the image capturing and transmitting apparatus 110 is arranged so that the optical axis is vertically downward at a position as high as a traffic light. Is assumed to be installed. A broken line in the figure indicates a line captured when a mesh-like grid line is drawn in the east-west and north-south direction on the ground level, and represents the degree of image distortion. Since a fisheye lens is used for the optical system 111, the image is distorted toward the periphery as shown in the figure. In the case of this example, the XY axes of the output image 501 of the image sensor unit are (+ X) north, (−X) south, (+ Y) east, (−Y) west, respectively. It is tailored to suit the direction.

  Of the output image range 501 of the image sensor unit, the range that can be seen through the virtual mirror 406 from the virtual viewpoint 405 in FIG. In the positional relationship of FIG. 4, the virtual mirror 406 is located slightly in the northeast direction when viewed from the image capturing / transmitting device 110. The virtual viewpoint 405 is located in the east direction as viewed from the position of the virtual mirror. Therefore, the range that can be imaged from the virtual viewpoint 405 via the virtual mirror 406 is widely distributed in the south, east, and north directions as seen from the image imaging transmission device 110, as indicated by the virtual optical axis 407. The center of the optical axis is slightly north. The image in such a range is a range indicated by 504 on the image data in FIG. 5 (geometric transformation processing range; indicated by a bold line in the figure). Here, it is assumed that the four end points of the geometric transformation processing range 504 are an A point 505, a B point 506, a C point 507, and a D point 508.

FIG. 6 shows an example of the coordinates of the image after geometric transformation when the pixels in the geometric transformation processing range 504 in FIG. 5 are mapped to the geometric transformation processing range 504 in FIG. 6 by the geometric transformation processing. Indicates. In FIG. 6, the coordinates are represented by the X2-Y2 axes, the virtual viewpoint coordinate 503 is the center, the downward direction of the virtual camera is (+ Y2), the upward direction is (−Y2), the right direction is (+ X2), and the left The direction matches (−X2). An output image range 600 (indicated by a thick broken line in the figure) for the display unit is defined so as to overlap with the geometric transformation processing range 504.
A point 505, B point 506, C point 507, and D point 508 in FIG. 5 are mapped to A point 505, B point 506, C point 507, and D point 508 in FIG. In FIG. 5, the C point 507 and the D point 508 exist in the east direction from the A point 505 and the B point 506. Since the optical axis coming from the object existing on the east side is reflected by the virtual mirror 406 and enters the virtual camera 406 from the upper side, the C point 507 and the D point 508 in FIG. The image is mapped above the image 505 and the B point 506.
In FIG. 5, the point A 505 and the point C 507 are present on the south side of the point B 506 and the point D 508. 4, the left direction of the virtual camera 405 corresponds to the south, and the right direction corresponds to the north. Therefore, in FIG. 6, the point A 505 and the point C 507 are mapped to the south side of the point B 506 and the point D 508.

FIG. 7 is an example of the output image 700 of the display unit. This image shows an example of a video image that is displayed when pixels mapped in the output image range 600 of the display unit as shown in FIG. 6 are displayed as image data on the display unit 126 of FIG. In FIG. 4, the pedestrian 303 and the other vehicle 304 that existed at the blind spot are displayed in a large size in FIG. 7. Also, an object that is located far from the virtual mirror 406 in FIG.
In addition, an actual curve mirror is designed so that a wide-angle range can be projected at a time by being curved. Similarly, the virtual mirror 406 is also curved with a predetermined curvature, and a wide-angle range can be displayed by adding distortion to the peripheral portion. By displaying such a virtual image to the driver of the host vehicle 202, the driver can confirm in advance the blind spot situation as an image captured from the immediate vicinity of the blind spot, thereby preventing an accident in advance. it can.

A conversion for mapping from the output image range 501 of the image sensor unit in FIG. 5 to the output image range 600 of the display unit in FIG. 6 is performed in the geometric conversion unit 124. Such conversion will be described below using mathematical expressions.
As shown in FIG. 5 (X, Y), the coordinates on the image data captured by the image sensor unit 112 of the image capturing / transmitting device 110 are on the image data output to the display unit 126 of the image receiving display device 120. Assuming that the coordinates are (X2, Y2) as shown in FIG. 6, the geometric transformation process f _all in the geometric transformation unit 124 is a process in which five types of geometric transformation processes from f ₁ to f ₅ are integrated. They have the relationship shown in (Formula 1).
(X2, Y2) = f _all (X, Y) = f ₅ f ₄ f ₃ f ₂ f ₁ (X, Y) (Formula 1)
However,
f ₁ ; Non-linear conversion processing for correcting an image distortion caused by the optical characteristics of the optical system unit to obtain an image without distortion or an image with reduced distortion.
f ₂ ; movement and rotation conversion processing for removing the movement and rotation of the viewpoint caused by the change in the position and direction of the image capturing and transmitting apparatus 110.
f ₃ ; mirror inversion conversion processing generated by the virtual mirror 406.
f ₄ ; Nonlinear conversion processing for reproducing the distortion of the virtual mirror 406 having a predetermined curvature and adding distortion to display a wide-angle region.
f ₅ ; movement and rotation conversion processing for converting the video viewed from the position and direction of the virtual viewpoint 405.

Of the geometric transformation processing of (Formula 1), f ₃ , f _4, and f ₅ are related to the position and orientation of the viewpoint to be displayed, and the distortion of the image at the time of display, and therefore arbitrary within the reception control unit 125 of FIG. Can be specified. On the other hand, f ₁ and f ₂ are items determined by the optical characteristics, the installation position, and the direction of the image capturing and transmitting apparatus 110. Geometric conversion unit 124, because it contains the image receiving display device 120, for f ₁ and f _2, only the information present in the image receiving display device 120 can not be determined parameters (variables).
Therefore, the present embodiment is characterized in that the mathematical formulas of f ₁ and f ₂ or parameters constituting those mathematical formulas are wirelessly transmitted from the imaging imaging transmission device 110 to the image reception display device 120 as imaging parameters. . By doing so, the mathematical expression of the geometric transformation process performed by the geometric transformation unit 124 can be confirmed, and an appropriate distortion correction process can be performed. Therefore, the situation of the portion corresponding to the blind spot of the driver can be displayed in a form that is easy to visually recognize, and an accident can be prevented.

In the present embodiment, the geometric transformation process in which the five types of geometric transformation processes f ₁ to f ₅ are combined has been described as an example. However, in practice, any nonlinear transformation may be used, and the order of the geometric transformation process May be replaced, a part of the geometric transformation process may be removed, or another geometric transformation process may be added. For example, in the signal processing unit 114 shown in FIG. 1, a part of the geometric transformation processing of f ₁ or f ₂ may be performed in order to reduce image distortion or normalize the displacement of the position or direction. good. In that case, the information about the optical characteristics, position, and direction at the time of image capturing is corrected by the transmission control unit 116 so as to match the transmitted image data, and is transmitted as an image capturing parameter.

Hereinafter, Example 2 in the present invention will be described in detail.
In the second embodiment, as in the first embodiment, when the host vehicle 202 is about to enter an intersection with poor visibility, it is an object to display a blind spot image of the driver in advance so that it can be easily seen. In the second embodiment, the configuration and the configuration are the same as those in the first embodiment, the configuration is the same as that of the vehicle image transmission / reception system 100 in FIG. 1 of the first embodiment, and the configuration is the same as that in the first embodiment. Same as 2. As a means, as in the case of the first embodiment, the image receiving display device 120 applies geometric transformation processing to the image data captured by the image capturing / transmitting device 110 to create a video that is easy for the driver to see. However, in the second embodiment, as will be described below, the viewpoint of the video to be created is different from that in the first embodiment, and the geometric transformation process for creating the video from that viewpoint is different from that in the first embodiment. The difference will be described below.

FIG. 8 is a diagram illustrating an example of a method for visually recognizing a blind spot when the method of the second embodiment is used. The image pickup transmission device 110, the image reception display device 120, the own vehicle 202, the building (obstacle) 301, the building (obstacle) 302, the pedestrian 303, the other vehicle 304, and the traveling direction 305 of the own vehicle in FIG. This is the same as FIG. In the first embodiment, the virtual viewpoint 405 and the virtual mirror 406 in FIG. 4 are defined, and an image corresponding to the video viewed from the virtual viewpoint 405 is created by geometric transformation processing. In contrast, in the second embodiment, a virtual viewpoint 800 is defined near the intersection as shown in FIG. Assuming that the virtual viewpoint 800 is installed in a shape that looks down from the height to the ground, in the case of the second embodiment, an image of the viewpoint looking down from the sky with the virtual viewpoint 800 as the center of the screen is displayed. Create and display to the driver. At this time, geometrical transformation in the rotational direction is applied as necessary so that the traveling direction of the host vehicle 202 is in the upward direction of the displayed image. By displaying an image looking down from the sky, the driver can know in advance the situation around the intersection through live action, so accidents can be prevented. Further, for example, the position of the virtual viewpoint 800 may be superimposed on the position of the host vehicle 202. In order to reflect the position or orientation of the host vehicle 202 in the displayed image, information on the position and orientation of the host vehicle 202 is output from the position / orientation output unit 127 in FIG. 1, and the reception control unit 125 in FIG. The information is transmitted to the geometric conversion unit 124 in FIG.
The video viewed from the virtual viewpoint 800 illustrated in FIG. 8 is generated from the image data captured by the image capturing / transmitting device 110 by the geometric transformation process in the image receiving display device 120. Hereinafter, an example of the geometric transformation process in the case of FIG. 8 will be described with reference to FIGS.

  FIG. 9 shows an example of the coordinates of an image taken in the situation of FIG. 8 before geometric transformation. An output image range 501 of the image sensor unit indicates a range of an image output from the image sensor unit 112 of FIG. In the example of this figure, the optical system unit 111 is configured to include a fisheye lens, and as in the case of FIG. 4, the image capturing and transmitting apparatus 110 is arranged so that the optical axis is vertically downward at a position as high as a traffic light. Is assumed to be installed. A broken line in the figure indicates a line captured when a mesh-like grid line is drawn in the east-west and north-south direction on the ground level, and represents the degree of image distortion. Since a fisheye lens is used for the optical system 111, the image is distorted toward the periphery as shown in the figure. In the case of this example, the XY axes of the output image 501 of the image sensor unit are (+ X) north, (−X) south, (+ Y) east, (−Y) west, respectively. It is tailored to suit the direction. Of the output image range 501 of the image sensor unit, a range including the coordinates 905 of the virtual viewpoint in FIG.

In the positional relationship of FIG. 8, the virtual viewpoint 800 is located slightly in the northeast direction when viewed from the image capturing / transmitting device 110. The range to be subjected to the geometric transformation process is a range indicated by 900 in the image data of FIG. 9 (geometric transformation processing range; indicated by a bold line in the figure). Here, it is assumed that the four end points of the geometric transformation processing range 900 are an A2 point 901, a B2 point 902, a C2 point 903, and a D2 point 904.
FIG. 10 shows an example of the coordinates of the image after geometric transformation when the pixels in the geometric transformation processing range 900 of FIG. 9 are mapped to the geometric transformation processing range 900 of FIG. 10 by the geometric transformation processing. In FIG. 10, the coordinates are represented by the X3-Y3 axes, the virtual viewpoint coordinate 905 is the center, the downward direction of the virtual camera is (+ Y3), the upward direction is (−Y3), the right direction is (+ X3), and the left The direction matches (−X3). An output image range 1000 (shown by a thick broken line in the figure) of the display unit is defined so as to overlap with the geometric transformation processing range 900.

A2 point 901, B2 point 902, C2 point 903, and D2 point 904 in FIG. 9 are mapped to A2 point 901, B2 point 902, C2 point 903, and D2 point 904 in FIG. The virtual viewpoint coordinate 905 is the central portion of the geometric transformation processing range 900.
FIG. 11 is an example of the output image 1100 of the display unit. This image shows an example of a video image that is displayed when pixels mapped in the output image range 1000 of the display unit as shown in FIG. 10 are displayed as image data on the display unit 126 of FIG. By displaying such a virtual image to the driver of the host vehicle 202, the driver can confirm the blind spot situation as an image captured from above, so that an accident can be prevented.

A conversion for mapping from the output image range 501 of the image sensor unit in FIG. 9 to the output image range 900 of the display unit in FIG. 10 is performed in the geometric conversion unit 124. Such conversion will be described below using mathematical expressions.
As shown in FIG. 9 (X, Y), the coordinates on the image data captured by the image sensor unit 112 of the image capturing / transmitting device 110 are on the image data output to the display unit 126 of the image receiving display device 120. _Assuming that the coordinates are (X3, Y3) as shown in FIG. 10, the geometric transformation processing f _all2 in the geometric transformation unit 124 _combines four types of geometric transformation processing of f ₁ , f ₂ , f ₆ , and f ₇ . These are the processes, and they have the relationship shown in (Formula 2).
(X3, Y3) = f _all2 (X, Y) = f ₇ f ₆ f ₂ f ₁ (X, Y) (Formula 2)
However,
f ₁ ; Non-linear conversion processing for correcting an image distortion caused by the optical characteristics of the optical system unit to obtain an image without distortion or an image with reduced distortion.
f ₂ ; movement and rotation conversion processing for removing the movement and rotation of the viewpoint caused by the change in the position and direction of the image capturing and transmitting apparatus 110.
f ₆ ; movement and rotation conversion processing for converting the video viewed from the position and direction of the virtual viewpoint 800.
f ₇ ; perspective projection conversion processing or parallel projection conversion processing.

F ₆ and f ₇ of geometric transformation processing of (Equation 2), the position of the viewpoint to be displayed, orientation, since according to the projection method, can be arbitrarily designated within the reception controller 125 of FIG. On the other hand, f ₁ and f ₂ are items determined by the optical characteristics, the installation position, and the direction of the image capturing and transmitting apparatus 110. Geometric conversion unit 124, because it contains the image receiving display device 120, for f ₁ and f _2, only the information present in the image receiving display device 120 can not be determined parameters (variables).
Therefore, the present embodiment is characterized in that the mathematical formulas of f ₁ and f ₂ or parameters constituting those mathematical formulas are wirelessly transmitted from the imaging imaging transmission device 110 to the image reception display device 120 as imaging parameters. . By doing so, the mathematical expression of the geometric transformation process performed by the geometric transformation unit 124 can be confirmed, and an appropriate distortion correction process can be performed. Therefore, the situation of the portion corresponding to the blind spot of the driver can be displayed in a form that is easy to visually recognize, and an accident can be prevented.

In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.
In addition, each of the above-described configurations may be configured such that some or all of them are configured by hardware, or are implemented by executing a program by a processor. Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

  DESCRIPTION OF SYMBOLS 100: Image transmission / reception system for vehicles, 110: Image pick-up transmission apparatus, 111: Optical system part, 112: Image sensor part, 113: Transmission image processing part, 114: Signal processing part, 115: Image compression part, 116: Transmission control 117: Transmission unit 118: Position / orientation output unit 120: Image reception display device 121: Reception unit 122: Received image processing unit 123: Image decoding unit 124: Geometric transformation unit 125: Reception control Part: 126: display part, 127: position / orientation output part, 200: prop, 201: traffic light, 202: own vehicle, 301: building (obstacle), 302: building (obstacle), 303: pedestrian, 304 : Other vehicle, 305: Signal of traveling direction of own vehicle, 310: Conventional curve mirror, 311: Optical axis, 405: Virtual viewpoint, 406: Virtual mirror, 407: Virtual optical axis, 501: Image Output image range of the disensor unit, 502: vertical coordinate of the image sensor unit, 503: coordinate of the virtual viewpoint, 504: geometric transformation processing range, 505: point A, 506: point B, 507: point C, 508: point D , 600: Output image range of the display unit, 700: Output image of the display unit, 900: Geometric transformation processing range, 901: A2 point, 902: B2 point, 903: C2 point, 903: D2 point, 1000: Display unit Output image range, 1100: output image of the display unit.

Claims (8)

An image reception display device that receives image data of an image obtained by imaging a subject and an imaging parameter indicating a situation when the image is captured, and displays an image based on the image data,
A receiving unit for receiving the image data and the imaging parameters;
A geometric transformation unit that is supplied with the image data received by the receiving unit, performs image processing including geometric transformation on the image data, and outputs the processed image data;
A display unit for displaying an image based on the image data output by the geometric conversion unit;
An image reception display device comprising: a reception control unit that is supplied with imaging parameters received by the reception unit and controls the image processing unit to perform image processing including the geometric transformation based on the imaging parameters. The image reception display device according to claim 1,
The image receiving apparatus is an image receiving apparatus mounted on a moving body, and the receiving unit receives the image data and the imaging parameter via a wireless communication path. The image reception display device according to claim 1,
The imaging parameter includes information representing optical image distortion of the image data received by the receiving unit,
The image reception display device, wherein the reception control unit controls the geometric conversion unit based on the information. The image reception display device according to claim 1,
The imaging parameter includes information related to a position and a direction when the image data is captured,
The image reception display device, wherein the reception control unit controls the geometric conversion unit based on the information. The image reception display device according to claim 4,
A position and direction output unit that detects and outputs information related to the position and direction of the image reception display device itself,
The reception control unit includes information on both the position and direction information when the image data is captured, and the position and direction information output from the position and direction output unit of the image reception display device. The image receiving and displaying apparatus characterized by controlling the geometric conversion unit based on the above. The image reception display device according to claim 5,
The reception control unit includes a virtual mirror provided for the image data received by the receiving unit from a virtual viewpoint at a position different from the position of the image receiving apparatus, at a position different from the position of the image receiving apparatus. An image receiving and displaying apparatus, wherein the geometric conversion unit is controlled to convert into image data when viewed. The image reception display device according to claim 5,
The receiving control unit is configured to convert the image data received by the receiving unit into image data when looking down from the virtual viewpoint at a position different from the position of the image receiving device into the ground direction. An image receiving display device characterized by controlling. An image receiving device according to claim 1;
An image transmission / reception system comprising: an image imaging transmission device that generates image data of an image obtained by imaging a subject, generates imaging parameters indicating a situation when the image is captured, and transmits the imaging parameters to the image reception device Because
The image capturing and transmitting apparatus is
An imaging unit that outputs an image signal of an image obtained by imaging the subject, and information relating to an optical situation when the image is captured;
A transmission image processing unit that processes image signals output by the imaging unit and outputs image data to be transmitted;
A position and direction output unit for detecting and outputting information related to the position and direction of the imaging transmission device itself;
Information relating to the optical situation and information relating to the position and direction are supplied, and a transmission control unit that outputs both as the imaging parameters;
An image transmission / reception system comprising: a transmission unit which is supplied with image data output from the transmission image processing unit and imaging parameters output from the transmission control unit and transmits both to the image reception device.

Images