JP7076896B2 - Display control device, display control method and camera monitoring system - Google Patents

Description

The present invention relates to a display control device, a display control method, and a camera monitoring system.

Conventionally, there has been known a display control technique in which a camera monitoring system is mounted on a vehicle and a photographed image behind the vehicle is displayed on a display device in the vehicle to function as an electronic mirror.

For example, Patent Document 1 below discloses a camera monitoring system that functions as an electronic side mirror by installing an image pickup device at a position of a side mirror of a vehicle and displaying a captured image on a display device. Further, in Patent Document 1 below, the photographed image displayed on the display device is made visible to the driver as a virtual image through a concave mirror, whereby the focal length from the driver is lengthened and the burden on the driver's eyes is increased. The configuration that reduces the problem is disclosed.

Japanese Unexamined Patent Publication No. 2005-329768

However, in the case of the camera monitoring system, the position of the virtual image visually recognized by the driver does not match the position where the image pickup device is installed. Therefore, the driver may feel a sense of discomfort with the visually recognized virtual image.

The present invention has been made in view of the above problems, and an object of the present invention is to reduce a sense of discomfort with respect to a virtual image visually recognized in a camera monitoring system.

According to one aspect, the display control device has the following configuration. That is,
It is a display control device connected to the image pickup device installed at the mounting position of the side mirror of the vehicle.
A conversion unit that converts the coordinates of each pixel of the image data captured by the image pickup device, and
It has an output unit that outputs display image data generated by converting coordinates by the conversion unit to the display unit.
The conversion unit
The coordinates of each pixel of the image data when the image pickup apparatus takes an image at a position corresponding to the mirror surface portion of the side mirror, in which the display image data displayed on the display unit is visually recognized as a virtual image via an optical member. And, at the mounting position of the side mirror of the vehicle , based on the correspondence with the coordinates of each pixel of the image data when the image pickup device takes a picture at the mounting position of the side mirror of the vehicle in which the image pickup device is installed. It is characterized in that the coordinates of each pixel of the image data captured by the image pickup apparatus are converted.

It is possible to reduce the discomfort with respect to the virtual image visually recognized in the camera monitoring system.

It is a figure which shows an example of the system configuration of the camera monitoring system in 1st Embodiment. It is a figure which shows the position where the image pickup device is installed, the shooting range of the image pickup device, and the shooting direction. It is a figure for demonstrating the function of an optical member. It is a figure which shows the positional relationship between the position where an image pickup apparatus is installed, and the position of a virtual image visually recognized by a driver. It is a figure which shows an example of conversion information. It is a figure which shows an example of the hardware composition of a display control device. It is a figure which shows an example of the functional structure of the viewpoint conversion part of the display control apparatus in 1st Embodiment. It is a flowchart which shows the flow of the viewpoint conversion processing by the viewpoint conversion part of the display control apparatus in 1st Embodiment. It is a figure which shows an example of the functional structure of the viewpoint conversion part of the display control apparatus in 2nd Embodiment. It is a flowchart which shows the flow of the viewpoint conversion processing by the viewpoint conversion part of the display control apparatus in 2nd Embodiment. It is a figure which shows an example of the system configuration of the camera monitoring system in 3rd Embodiment. It is a figure which shows the position where the image pickup apparatus is installed, and the photographing range of the image pickup apparatus in the camera monitoring system in 3rd Embodiment.

Hereinafter, each embodiment will be described with reference to the attached drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to omit duplicate explanations.

[First Embodiment]
<System configuration of camera monitoring system>
First, the system configuration of the camera monitoring system mounted on the vehicle will be described. FIG. 1 is a diagram showing an example of a system configuration of a camera monitoring system according to the first embodiment. As shown in FIG. 1, the camera monitoring system 100 includes image pickup devices 110 and 111, display control devices 120, display devices 130 and 131, and optical members 140 and 141. In the present embodiment, the camera monitoring system 100 is activated when the ignition switch of the vehicle is turned on, and is stopped when the ignition switch of the vehicle is turned off. However, the camera monitoring system 100 may be configured to start and stop based on the instruction of the driver of the vehicle.

The image pickup devices 110 and 111 are installed at the positions of the side mirrors of the vehicle and photograph the rear of the vehicle. The image pickup devices 110 and 111 transmit image data (left side image data, right side image data) obtained by inverting the captured image left and right to the display control device 120.

A viewpoint conversion program is installed in the display control device 120, and when the viewpoint conversion program is executed, the display control device 120 functions as the viewpoint conversion unit 121. The viewpoint conversion unit 121 reads the conversion information from the conversion information storage unit 122 and converts the left side image data and the right side image data to generate the left side display image data and the right side display image data. Further, the viewpoint conversion unit 121 outputs the generated left side display image data and right side display image data to the display devices 130 and 131, respectively.

The display devices 130 and 131 display the left side display image data or the right side display image data, respectively. The left side display image data displayed by the display device 130 is visually recognized as a virtual image by the driver of the vehicle via the optical member 140. Further, the right side display image data displayed by the display device 131 is visually recognized as a virtual image by the driver of the vehicle via the optical member 141.

<Installation example of imaging device>
Next, an installation example of the image pickup devices 110 and 111 (position where the image pickup devices 110 and 111 are installed, a shooting range and a shooting direction of the image pickup devices 110 and 111) will be described. FIG. 2 is a diagram showing a position where an image pickup device is installed, a shooting range of the image pickup device, and a shooting direction.

As shown in FIG. 2, the image pickup apparatus 110 is installed at or near the position where the side mirror on the left side of the vehicle 200 is originally attached. The image pickup apparatus 110 takes a picture with the shaded area 210 behind the vehicle 200 as a picture taking range. The shooting direction at this time is as shown by the arrow 220. Similarly, the image pickup apparatus 111 is installed at or near the position where the side mirror on the right side of the vehicle 200 is originally attached. The image pickup apparatus 111 takes a picture with the shaded area 211 behind the vehicle 200 as a picture taking range. The shooting direction at this time is as shown by arrow 221.

<Function of optical member>
Next, the functions of the optical members 140 and 141 for visualizing the left side display image data or the right side display image data displayed on the display devices 130 and 131 to the driver of the vehicle 200 as a virtual image will be described. Since the function of the optical member 140 and the function of the optical member 141 are basically the same, the function of the optical member 141 will be described here.

FIG. 3 is a diagram for explaining the function of the optical member. As shown in FIG. 3, the optical member 141 includes a concave mirror 301 and a concave mirror 302. The concave mirror 301 reflects the emitted light of the right side display image data emitted by the display device 131. The concave mirror 302 further reflects the emitted light of the right side display image data reflected by the concave mirror 301. The emitted light of the right-side display image data reflected by the concave mirror 302 is captured by the visual sense of the driver 300 of the vehicle 200.

Here, the concave mirror 301 is installed at a position separated by a distance L1 from the position where the display device 131 is installed. Therefore, the emitted light of the right-side display image data reflected by the concave mirror 301 is visually recognized as a virtual image 311 at a position at a distance L1 behind the concave mirror 301 at the position on the reflection side of the concave mirror 301.

Similarly, the concave mirror 302 is installed at a position separated from the position of the virtual image 311 by a distance L2. Therefore, the emitted light of the right-side display image data reflected by the concave mirror 302 is a virtual image 312 at a position on the reflection side of the concave mirror 302 (that is, by the driver 300 of the vehicle 200) at a distance L2 behind the concave mirror 302. Will be visually recognized as.

<Positional relationship between the position where the image pickup device is installed and the position of the virtual image>
Next, the positional relationship between the positions where the image pickup devices 110 and 111 are installed and the position of the virtual image visually recognized by the driver 300 of the vehicle 200 will be described. FIG. 4 is a diagram showing the positional relationship between the position where the image pickup device is installed and the position of the virtual image visually recognized by the driver.

As described above, the left side display image data displayed by the display device 130 is visually recognized as a virtual image by the driver 300 via the optical member 140. Further, the right side display image data displayed by the display device 131 is visually recognized as a virtual image by the driver 300 via the optical member 141.

Here, the optical member 140 is configured so that the virtual image 400 is visually recognized at a position slightly away from the left side surface of the vehicle 200. Similarly, the optical member 141 is configured such that the virtual image 401 is visible at a position slightly distant from the right side surface of the vehicle 200. This takes into consideration that the mirror surface portion of the side mirror in a general vehicle is located at a position slightly away from the left side surface (or right side surface) of the vehicle.

Assuming that the mirror surface portion of the side mirror in a general vehicle is at the position of the virtual image 400, the direction and region when the driver 300 of the vehicle visually recognizes the rear of the vehicle through the side mirror is an arrow. The direction indicated by 420 and the shaded area 410. In other words, for the left side display image data visually recognized as a virtual image 400 by the driver 300 of the vehicle 200, the image pickup device 110 is installed at the position of the virtual image 400, and the arrow 420 is photographed in the photographing direction and the shaded area 410 is photographed as the photographing range. It is desirable that it has been done.

However, in reality, as described with reference to FIG. 2, the image pickup apparatus 110 is installed on the left side surface of the vehicle 200, and the arrow 220 is the shooting direction and the shaded area 210 is the shooting range. That is, in a general vehicle, the position of the virtual image 400 and the position where the image pickup device 110 is installed do not match. Therefore, the driver 300 of the vehicle 200 feels a sense of discomfort in the direction and region visually recognized as the virtual image 400. Therefore, the viewpoint conversion unit 121 converts the left side image data and generates the left side display image data so as to reduce the discomfort.

Similarly, assuming that the mirror surface portion of the side mirror in a general vehicle is at the position of the virtual image 401, the direction and region when the driver 300 of the vehicle visually recognizes the rear of the vehicle through the side mirror is determined. The direction indicated by the arrow 421 and the area indicated by the shaded area 411. In other words, for the right-side display image data visually recognized as a virtual image 401 by the driver 300 of the vehicle 200, the image pickup device 111 is installed at the position of the virtual image 401, and the arrow 421 is photographed with the shooting direction and the shaded area 411 as the shooting range. It is desirable that it has been done.

However, in reality, as described with reference to FIG. 2, the image pickup apparatus 111 is installed on the right side surface of the vehicle 200, and the arrow 221 is the shooting direction and the shaded area 211 is the shooting range. That is, in a general vehicle, the position of the virtual image 401 and the position where the image pickup device 110 is installed do not match. Therefore, the driver 300 of the vehicle 200 feels a sense of discomfort in the direction and region visually recognized as the virtual image 401. Therefore, the viewpoint conversion unit 121 converts the right image data and generates the right display image data so as to reduce the discomfort.

<Details of conversion information>
Next, the conversion information of the conversion information storage unit 122, which the viewpoint conversion unit 121 refers to when converting the left side image data and the right side image data into the left side display image data and the right side display image data, will be described. FIG. 5 is a diagram showing an example of conversion information.

Of these, FIG. 5A is a diagram showing an example of conversion information 510 referred to when the viewpoint conversion unit 121 converts the left side image data into the left side display image data. As shown in FIG. 5A, the conversion information 510 includes the "viewpoint position", "viewpoint direction", and "image coordinates" of the image pickup apparatus, and the "viewpoint position" and "viewpoint direction" of the virtual image as information items. , Includes "image coordinates".

In the conversion information 510, the "viewpoint position" of the image pickup device stores the coordinates ("coordinates _PLC ") indicating the position where the image pickup device 110 is installed. Further, in the "viewpoint direction" of the image pickup device, a 6-axis lens parameter ("lens parameter _DLC ") indicating the shooting direction of the image pickup device 110 is stored. Further, in the "image coordinates" of the image pickup device, the coordinates of each pixel of the left image data captured by the image pickup device 110 at the corresponding viewpoint position and viewpoint direction are stored.

Further, in the conversion information 510, the coordinates ("coordinates _PLI ") indicating the position of the virtual image 400 are stored in the "viewpoint position" of the virtual image. Further, in the "viewpoint direction" of the image pickup device, a 6-axis lens parameter ("lens parameter D _LC ") indicating the shooting direction when the image pickup device 110 is installed at the position of the virtual image 400 is stored. Further, in the "image coordinates" of the virtual image, the coordinates of each pixel of the image data taken by the image pickup apparatus 110 at the corresponding viewpoint position and viewpoint direction are stored.

In FIG. 5A, the "image coordinates" of the image pickup apparatus = "x _L1 , y _L1 ", ... "X _Ln , y _Lm " and the "image coordinates" of the virtual image = "X _L1 , Y _L1 ", ... It is associated with "X _Ln , Y _Lm ". The association between the "image coordinates" of the image pickup device and the "image coordinates" of the virtual image is as follows in the case of "x _L1 , y _L1 " and " _XL1 , Y _L1 ", for example. It is in.
-On the image data of the subject A behind the vehicle 200 when the image pickup device 110 takes a picture of the rear of the vehicle 200 with the viewpoint position = "coordinates _PLC " and the viewpoint direction = "lens parameter _DLC ". The coordinates are "x _L1 , y _L1 ".
-On the image data of the subject A behind the vehicle 200 when the image pickup device 110 takes a picture of the rear of the vehicle 200 with the viewpoint position = "coordinates P _LI " and the viewpoint direction = "lens parameter D _LI ". The coordinates are " _{XL1, Y L1 "} .

That is, in the conversion information 510, the "image coordinates" of the image pickup device and the "image coordinates" of the virtual image store the coordinates of the corresponding points on the image data when the same subject is photographed at different viewpoint positions and viewpoint directions. The coordinates.

Similarly, FIG. 5B is a diagram showing an example of conversion information 520 referred to when converting the right side image data into the right side display image data. As shown in FIG. 5B, the conversion information 520 contains the "viewpoint position", "viewpoint direction", and "image coordinates" of the image pickup apparatus, and the "viewpoint position" and "viewpoint direction" of the virtual image as information items. , Includes "image coordinates".

In the conversion information 520, the "viewpoint position" of the image pickup device stores the coordinates ("coordinates P _RC ") indicating the position where the image pickup device 111 is installed. Further, in the "viewpoint direction" of the image pickup device, a 6-axis lens parameter ("lens parameter D _RC ") indicating the shooting direction of the image pickup device 111 is stored. Further, in the "image coordinates" of the image pickup device, the coordinates of each pixel of the right image data captured by the image pickup device 111 at the corresponding viewpoint position and viewpoint direction are stored.

Further, in the conversion information 520, the coordinates ("coordinates _PRI ") indicating the position of the virtual image 401 are stored in the "viewpoint position" of the virtual image. Further, in the "viewpoint direction" of the image pickup device, a 6-axis lens parameter ("lens parameter D _RC ") indicating the shooting direction when the image pickup device 111 is installed at the position of the virtual image 401 is stored. Further, in the "image coordinates" of the virtual image, the coordinates of each pixel of the image data taken by the image pickup apparatus 111 at the corresponding viewpoint position and viewpoint direction are stored.

In FIG. 5B, the "image coordinates" of the image pickup apparatus = "x _R1 , y _R1 ", ... "X _Rn , y _Rm " and the "image coordinates" of the virtual image = "X _R1 , Y _R1 ", ... It is associated with "X _Rn , Y _Rm ". In the case of "x _R1 , y _R1 " and "X _R1 , Y _R1 ", the relationship between the "image coordinates" of the image pickup device and the "image coordinates" of the virtual image is as follows. It is in.
-On the image data of the subject B behind the vehicle 200 when the image pickup device 111 takes a picture of the rear of the vehicle 200 with the viewpoint position = "coordinates P _RC " and the viewpoint direction = "lens parameter D _RC ". The coordinates are "x _R1 , y _R1 ".
_{-On the image data of the subject B behind the vehicle 200 when the image pickup device 111 takes a picture of the rear of the vehicle 200 with the viewpoint position = "coordinates PRI" and the viewpoint direction = "lens parameter D RI "} . The coordinates are "X _R1 , Y _R1 ".

That is, in the conversion information 520, the "image coordinates" of the image pickup device and the "image coordinates" of the virtual image store the coordinates of the corresponding points on the image data when the same subject is photographed at different viewpoint positions and viewpoint directions. The coordinates.

<Hardware configuration of display control device>
Next, the hardware configuration of the display control device 120 will be described. FIG. 6 is a diagram showing an example of the hardware configuration of the display control device. As shown in FIG. 6, the display control device 120 includes a CPU (Central Processing Unit) 601, a ROM (Read Only Memory) 602, and a RAM (Random Access Memory) 603. Further, the display control device 120 includes a GPU (Graphic Processing Unit) 604 and an I / F (Interface) device 605. Each part of the display control device 120 is connected to each other via a bus 606.

The CPU 601 is a device that executes various programs (for example, a viewpoint conversion program) installed in the ROM 602. The ROM 602 is a non-volatile memory and functions as a main storage device for storing various programs executed by the CPU 601. The RAM 603 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or a SRAM (Static Random Access Memory). The RAM 603 functions as a main storage device that provides a work area that is expanded when various programs installed in the ROM 602 are executed by the CPU 601.

The GPU 604 is a dedicated integrated circuit that performs image processing. In the present embodiment, the left side display image data and the right side display image data displayed on the display devices 130 and 131 are based on instructions from the CPU 601 that executes the viewpoint conversion program. Is generated. The I / F device 605 is a connection device for connecting the image pickup devices 110 and 111, the display devices 130 and 131, and the display control device 120.

<Functional configuration of display control device>
Next, the functional configuration of the display control device 120 will be described. FIG. 7 is a diagram showing an example of the functional configuration of the viewpoint conversion unit of the display control device according to the first embodiment. As described above, the viewpoint conversion program is installed in the display control device 120, and when the camera monitoring system 100 is activated, the display control device 120 executes the viewpoint conversion program. As a result, the display control device 120 has the left side image data acquisition unit 701, the right side image data acquisition unit 711, the left side image data conversion unit 702, the right side image data conversion unit 712, the left side display image data generation unit 703, and the right side display image data generation. It functions as a unit 713.

The left image data acquisition unit 701 and the right image data acquisition unit 711 acquire the left image data and the right image data transmitted from the image pickup devices 110 and 111. The left image data acquisition unit 701 and the right image data acquisition unit 711 transmit the acquired left image data and right image data to the left image data conversion unit 702 and the right image data conversion unit 712 in frame units.

The left image data conversion unit 702 and the right image data conversion unit 712 are examples of the conversion unit, and each pixel of the left image data and the right image data is based on the conversion information 510 and 520 stored in the conversion information storage unit 122. Convert the coordinates of. The conversion of the left image data and the right image data is obtained by calculation using a general conversion method (for example, coordinate conversion of the projection surface, affine transformation, etc.) instead of referring to the conversion information 510 and 520. May be good.

The left side display image data generation unit 703 and the right side display image data generation unit 713 are examples of output units. The left side display image data generation unit 703 and the right side display image data generation unit 713 extract areas suitable for display on the display devices 131 and 132 from the left side image data and the right side image data in which the coordinates of each pixel are converted. , Left side display image data, right side display image data are generated. Further, the left side display image data generation unit 703 and the right side display image data generation unit 713 output the generated left side display image data and right side display image data to the display device 130 and the display device 131.

<Flow of viewpoint conversion process by viewpoint conversion unit>
Next, the flow of the viewpoint conversion process by the viewpoint conversion unit 121 of the display control device 120 will be described. FIG. 8 is a flowchart showing the flow of the viewpoint conversion process by the viewpoint conversion unit of the display control device according to the first embodiment.

Of these, FIG. 8A is a flowchart showing the flow of the viewpoint conversion process for the left image data. In step S801, the left image data acquisition unit 701 reads the conversion information 510 from the conversion information storage unit 122. In step S802, the left image data acquisition unit 701 acquires the left image data from the image pickup apparatus 110.

In step S803, the left image data conversion unit 702 converts the coordinates of each pixel of the left image data based on the conversion information 510. In step S804, the left side display image data generation unit 703 generates left side display image data based on the left side image data in which the coordinates of each pixel are converted, and outputs the left side display image data to the display device 130.

In step S805, the left image data acquisition unit 701 determines whether or not to end the viewpoint conversion process, and if it is determined not to end (if No in step S805), the process returns to step S802. On the other hand, if it is determined in step S805 to end (yes in step S805), the left image data acquisition unit 701 ends the viewpoint conversion process for the left image data.

On the other hand, FIG. 8B is a flowchart showing the flow of the viewpoint conversion process for the right image data. In step S811, the right image data acquisition unit 711 reads the conversion information 520 from the conversion information storage unit 122. In step S812, the right image data acquisition unit 711 acquires the right image data from the image pickup apparatus 111.

In step S813, the right image data conversion unit 712 converts the coordinates of each pixel of the right image data based on the conversion information 520. In step S814, the right side display image data generation unit 713 generates right side display image data based on the right side image data in which the coordinates of each pixel are converted, and outputs the right side display image data to the display device 131.

In step S815, the right image data acquisition unit 711 determines whether or not to end the viewpoint conversion process, and if it is determined not to end (if No in step S815), returns to step S812. On the other hand, if it is determined in step S815 to end (yes in step S815), the right image data acquisition unit 711 ends the viewpoint conversion process for the right image data.

<Summary>
As is clear from the above description, the display control device according to the first embodiment is
-The coordinates of each pixel of the image data captured by the image pickup device are converted and displayed on the display device (display unit).
-At that time, based on the positional relationship (viewpoint position, viewpoint direction) between the position where the display image data displayed by the display device is visually recognized as a virtual image via the optical member and the position where the image pickup device is installed. Convert the coordinates of each pixel of the image data.

Thereby, according to the display control device in the first embodiment, it is possible to absorb the difference between the position of the virtual image visually recognized by the driver and the position set by the image pickup device. As a result, in the camera monitoring system that functions as an electronic side mirror, it is possible to reduce the driver's discomfort with respect to the virtual image.

[Second Embodiment]
In the first embodiment, it has been described that the viewpoint direction of the virtual image is fixed. On the other hand, in the second embodiment, a case where the viewpoint direction of the virtual image is variable based on the instruction of the driver of the vehicle will be described. This is because the mirror surface portion of the side mirror in a general vehicle can also change the viewpoint direction based on the instruction of the driver of the vehicle. Hereinafter, the second embodiment will be described focusing on the differences from the first embodiment.

<Functional configuration of display control device>
FIG. 9 is a diagram showing an example of the functional configuration of the viewpoint conversion unit of the display control device according to the second embodiment. The difference from FIG. 7 is that it has a left side instruction receiving unit 901, a left side image data conversion unit 902, a right side instruction receiving unit 911, and a right side image data conversion unit 912.

The left side instruction receiving unit 901 receives an instruction to change the viewpoint direction for the left side display image data from the driver 300 of the vehicle 200. Further, the left side instruction receiving unit 901 notifies the left side image data conversion unit 902 of the received instruction.

The left image data conversion unit 902 reads the conversion information 510 from the conversion information storage unit 122, and converts the coordinates of each pixel of the left image data based on the read conversion information 510. The conversion information storage unit 122 stores a plurality of conversion information having different "viewpoint directions" of the virtual image, and the left image data conversion unit 702 responds to the instruction notified from the left instruction reception unit 901. Read the conversion information that has the "viewpoint direction" of the virtual image.

The right side instruction receiving unit 911 receives an instruction to change the viewpoint direction for the right side display image data from the driver 300 of the vehicle 200. Further, the right side instruction receiving unit 911 notifies the right side image data conversion unit 912 of the received instruction.

The right image data conversion unit 912 reads the conversion information 520 from the conversion information storage unit 122, and converts the coordinates of each pixel of the right image data based on the read conversion information 520. The conversion information storage unit 122 stores a plurality of conversion information having different "viewpoint directions" of the virtual image, and the right image data conversion unit 912 responds to the instruction notified from the right instruction reception unit 911. Read the conversion information that has the "viewpoint direction" of the virtual image.

<Flow of viewpoint conversion process by viewpoint conversion unit>
Next, the flow of the viewpoint conversion process by the viewpoint conversion unit 121 of the display control device 120 will be described. FIG. 10 is a flowchart showing the flow of the viewpoint conversion process by the viewpoint conversion unit of the display control device according to the second embodiment.

Of these, FIG. 10A is a flowchart showing the flow of the viewpoint conversion process for the left image data, and the difference from FIG. 8A is step S1001. In step S1001, the left side instruction receiving unit 901 determines whether or not the driver 300 of the vehicle 200 has instructed to change the viewpoint direction on the left side. If it is determined in step S1001 that there is no instruction (No in step S1001), the process proceeds to step S805.

On the other hand, if it is determined in step S1001 that an instruction has been given (yes in step S1001), the instruction is accepted, notified to the left image data conversion unit 902, and then the process returns to step S801. As a result, the left image data conversion unit 902 reads the conversion information having the "viewpoint direction" of the virtual image from the conversion information storage unit 122 according to the notified instruction. As a result, the left image data conversion unit 902 can realize conversion of the left image data according to the instruction of the driver 300 of the vehicle 200.

FIG. 10B is a flowchart showing the flow of the viewpoint conversion process for the right image data, and the difference from FIG. 8B is step S1011. In step S1011, the right side instruction receiving unit 911 determines whether or not the driver 300 of the vehicle 200 has instructed to change the viewing direction on the right side. If it is determined in step S1011 that there is no instruction (No in step S1011), the process proceeds to step S815.

On the other hand, if it is determined in step S1011 that an instruction has been given (yes in step S1011), the instruction is accepted, notified to the right image data conversion unit 912, and then the process returns to step S811. As a result, the right image data conversion unit 912 reads the conversion information having the "viewpoint direction" of the virtual image from the conversion information storage unit 122 according to the notified instruction. As a result, the right image data conversion unit 912 can realize conversion of the right image data according to the instruction of the driver 300 of the vehicle 200.

<Summary>
As is clear from the above description, the display control device according to the second embodiment is
-Based on the positional relationship (viewpoint position, viewpoint direction) between the position where the display image data displayed by the display device is visually recognized as a virtual image via the optical member and the position where the image pickup device is installed, the image data Convert the coordinates of each pixel.
-At that time, the coordinates of each pixel of the image data are converted based on the positional relationship (viewpoint direction) according to the instruction of the driver of the vehicle.

As a result, according to the display control device according to the second embodiment, it is possible to realize the same processing as operating the mirror surface portion of the side mirror in a general vehicle with the electronic side mirror.

[Third Embodiment]
In the first and second embodiments described above, a camera monitoring system in which two image pickup devices are installed in the vehicle 200 has been described. However, the system configuration of the camera monitoring system is not limited to this. Hereinafter, the system configuration of another camera monitoring system will be described with reference to FIGS. 11 and 12.

FIG. 11 is a diagram showing an example of the system configuration of the camera monitoring system according to the third embodiment. Further, FIG. 12 is a diagram showing a position where the image pickup device is installed and a shooting range of the image pickup device in the camera monitoring system according to the third embodiment.

FIG. 11A shows how one image pickup device 1101 is installed at the rear center position of the vehicle 200 as shown in FIG. 12A, and the viewpoint conversion process is realized in the camera monitoring system 1100. ing. In this case, the camera monitoring system 1100 functions as an electronic rearview mirror.

As shown in FIG. 12A, the image pickup apparatus 1101 takes a picture with the shaded area 1210 behind the vehicle 200 as the picture taking range. Then, the display control device 120 generates the central display image data based on the central image data captured by the image pickup device 1101, and the display device 1102 displays the central display image data. The central display image data displayed by the display device 1102 is visually recognized by the driver at the central position in the vehicle interior as a virtual image via the optical member 1103.

On the other hand, in FIG. 11B, three image pickup devices 110, 111, and 1101 are installed at the position of the side mirror of the vehicle 200 or the rear center position as shown in FIG. 12B, and the viewpoint conversion process is performed. , The state of realization in the camera monitoring system 1110 is shown. The camera monitoring system 1110 shall function as an electronic side mirror.

As shown in FIG. 12 (b), the image pickup apparatus 1101 performs imaging with a region outside the imaging range of the imaging devices 110 and 111 as the photographing range. Therefore, in the display control device 120, the central image data (central image data of the region outside the shooting range of the image pickup device 110) taken by the image pickup device 1101 is combined with the left side image data taken by the image pickup device 110. Then, the left side display image data is generated.

Similarly, in the display control device 120, the center image data (center image data of the region outside the shooting range of the image pickup device 110) taken by the image pickup device 1101 is combined with the right image data taken by the image pickup device 111. Then, the image data displayed on the right side is generated. Thereby, according to the camera monitoring system 1110, the driver 300 of the vehicle 200 can visually recognize the left side display image data and the right side display image data including the area outside the shooting range of the image pickup devices 110 and 111 as virtual images. ..

[Other embodiments]
In the first to third embodiments, the coordinates of each pixel of the image data are converted in order to absorb the difference between the position of the virtual image visually recognized by the driver and the position where the image pickup device is installed. explained. However, the method for absorbing the difference between the position of the virtual image visually recognized by the driver and the position where the image pickup device is installed is not limited to this, and the position of the virtual image visually recognized by the driver and the image pickup device are limited to this. The optics may be designed to match the installation position. This eliminates the need to convert the coordinates of each pixel of the image data.

The present invention is not limited to the configurations shown here, such as combinations with other elements in the configurations and the like described in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form thereof.

100: Camera monitoring system 110, 111: Imaging device 120: Display control device 121: Viewpoint conversion unit 130, 131: Display device 140, 141: Optical member 200: Vehicle 300: Driver 301, 302: Concave mirror 400, 401: Fictitious image 510: 520: Conversion information 701: Left side image data acquisition unit 702: Left side image data conversion unit 703: Left side display image data generation unit 711: Right side image data acquisition unit 712: Right side image data conversion unit 713: Right side display image data generation unit 901: Left instruction reception unit 902: Left image data conversion unit 911: Right instruction reception unit 912: Right image data conversion unit

Claims (5)

It is a display control device connected to the image pickup device installed at the mounting position of the side mirror of the vehicle.
A conversion unit that converts the coordinates of each pixel of the image data captured by the image pickup device, and
It has an output unit that outputs display image data generated by converting coordinates by the conversion unit to the display unit.
The conversion unit
The coordinates of each pixel of the image data when the image pickup apparatus takes an image at a position corresponding to the mirror surface portion of the side mirror, in which the display image data displayed on the display unit is visually recognized as a virtual image via an optical member. And, at the mounting position of the side mirror of the vehicle , based on the correspondence with the coordinates of each pixel of the image data when the image pickup device takes a picture at the mounting position of the side mirror of the vehicle in which the image pickup device is installed. A display control device characterized by converting the coordinates of each pixel of image data captured by the image pickup device. Further, it has a storage unit for storing conversion information indicating the correspondence relationship, and has a storage unit.
The first aspect of the present invention is characterized in that the conversion unit reads the conversion information from the storage unit and converts the coordinates of each pixel of the image data captured by the image pickup device at the mounting position of the side mirror of the vehicle. The display control device described. It also has a reception unit that accepts instructions to change the correspondence.
The display control device according to claim 2, wherein the conversion unit reads the conversion information in response to an instruction received by the reception unit from the storage unit. It is a display control method of the display control device connected to the image pickup device installed at the mounting position of the side mirror of the vehicle.
A conversion step of converting the coordinates of each pixel of the image data captured by the image pickup device, and
It has an output step of outputting display image data generated by converting coordinates in the conversion step to a display unit.
The conversion step is
The coordinates of each pixel of the image data when the image pickup apparatus takes an image at a position corresponding to the mirror surface portion of the side mirror, in which the display image data displayed on the display unit is visually recognized as a virtual image via an optical member. And, at the mounting position of the side mirror of the vehicle , based on the correspondence with the coordinates of each pixel of the image data when the image pickup device takes a picture at the mounting position of the side mirror of the vehicle in which the image pickup device is installed. A display control method comprising converting the coordinates of each pixel of image data captured by the image pickup apparatus. An image pickup device installed at the mounting position of the side mirror of the vehicle and taking a picture of the rear of the vehicle,
A display control device connected to the image pickup device and
A display device that displays display image data output from the display control device, and a display device that displays the display image data.
A camera monitoring system comprising an optical member that reflects the emitted light of the display device.
The display control device is
A conversion unit that converts the coordinates of each pixel of the image data captured by the image pickup device, and
It has an output unit that outputs display image data generated by converting coordinates by the conversion unit to the display device.
The conversion unit
Each pixel of the image data when the image pickup device takes an image at a position corresponding to the mirror surface portion of the side mirror, in which the display image data displayed on the display device is visually recognized as a virtual image through the optical member. The mounting position of the side mirror of the vehicle is based on the correspondence between the coordinates and the coordinates of each pixel of the image data when the image pickup device takes a picture at the mounting position of the side mirror of the vehicle in which the image pickup device is installed. A camera monitoring system characterized in that the coordinates of each pixel of the image data captured by the image pickup apparatus are converted.

Images