JP5464741B2 - Image composition apparatus and image composition method - Google Patents

Description

  The present invention relates to a technique for adjusting the luminance of a plurality of camera images to be combined.

A technique for processing an image input from a camera and displaying it on a display screen is known. For example, there is camera image processing called AGC (Automatic Gain Control).

  The AGC is usually composed of an electric circuit or the like. When the average brightness of the image signal input from the camera is weak, the overall brightness is increased. When the average brightness is strong, the brightness is totally bright. This is control that performs processing to reduce the image so that the image displayed on the display screen is not too dark and not too bright. A technique for processing a camera image in this way is disclosed in Patent Document 1, for example.

  Further, in recent years, a vehicle peripheral image is generated by combining images input from a plurality of cameras mounted on the vehicle to generate a vehicle peripheral image showing a state of the periphery of one vehicle and displaying the generated vehicle peripheral image on a display screen. A display system has been developed.

Japanese Patent Laid-Open No. 11-136571

  However, in the vehicle peripheral image display system, if one composite image is generated after AGC is performed for each camera image in a plurality of camera images, there is a possibility that brightness unevenness occurs in the composite image. Also, if AGC is applied after combining a plurality of camera images into a single vehicle periphery image, the brightest image may become brighter and difficult to see due to the effect of the darkest image. is there.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique that makes it easy to view a composite image of a plurality of camera images captured by a plurality of cameras mounted on a vehicle.

In order to solve the above-described problem, the invention of claim 1 is an image composition device that is mounted on a vehicle and synthesizes a plurality of images, and includes a plurality of image pickup means mounted on the outside of the vehicle in different directions. Input means for inputting a plurality of images, determination means for determining one of the plurality of images as a representative image based on the luminance of the plurality of input images, and other information based on the representative image. First adjusting means for adjusting the brightness of the image of the image, and combining means for combining the representative image and the other image whose brightness has been adjusted into one vehicle peripheral image showing a state of the periphery of the vehicle,
It is characterized by providing.

  According to a second aspect of the present invention, in the image synthesizing apparatus according to the first aspect, the plurality of images are configured by a predetermined number of pixels, and the determining unit is configured to calculate all pixels in the image for each of the plurality of images. An average luminance is calculated, and an image having the highest average luminance is determined as a representative image.

  The invention according to claim 3 is the image synthesizing apparatus according to claim 2, wherein the determining means does not include the pixel in the calculation of the average luminance when the pixel in the image has a luminance of a certain value or more. It is characterized by that.

  According to a fourth aspect of the present invention, in the image synthesizing device according to the third aspect, the difference between the highest average luminance and the lowest average luminance among the average luminances calculated for each of the plurality of images is further determined. Control means for enabling the first adjusting means when it is equal to or greater than a predetermined value is provided.

  According to a fifth aspect of the present invention, in the image composition device according to the fourth aspect, the luminance of the pixel in the image in which the average luminance is calculated based on the average luminance calculated for each of the plurality of images. A second adjusting unit that adjusts the difference between the highest average luminance and the lowest average luminance among the average luminances calculated for each of the plurality of images. The second adjusting means is enabled.

  According to a sixth aspect of the present invention, there is provided an image composition method for combining a plurality of images mounted on a vehicle, wherein a plurality of images are input from a plurality of imaging means mounted on the outside of the vehicle in different directions. An input step, a determination step of determining one of the plurality of images as a representative image based on the input luminance of the plurality of images, and a luminance of another image based on the representative image An adjusting step for adjusting, and a combining step for combining the representative image and the other image whose luminance has been adjusted into one vehicle peripheral image showing a state of the periphery of the vehicle. .

  According to the first to sixth aspects of the present invention, in order to image the periphery of the vehicle, a plurality of images are input from a plurality of imaging units attached to different directions, and based on the luminance of the input plurality of images. The brightness of other images is adjusted based on the determined representative image, and these images are combined into one vehicle surrounding image, so there is little unevenness in brightness and the viewer feels uncomfortable as a single vehicle surrounding image. Can be prevented.

  According to the invention of claim 2, in the invention of claim 1, the average brightness of all pixels in the image is calculated for each of a plurality of images, and the image with the highest average brightness is determined as the representative image. Since the brightness of the other image is adjusted based on the average brightness of the image, the other image is adjusted based on the ideal image having the highest average brightness, so that the visibility for the viewer can be improved.

According to the invention of claim 3, in the invention of claim 2, when a pixel in the image has a luminance equal to or higher than a certain value, the pixel is not included in the calculation of the average luminance. The average luminance can be prevented from becoming extremely high, and the visibility for the viewer can be improved.
According to the invention of claim 4, in the invention of claim 3, the difference between the highest average luminance and the lowest average luminance among the average luminances calculated for each of the plurality of images is equal to or greater than a predetermined value. In this case, since the image adjustment is enabled, when a bright image and a dark image are mixed, the adjustment is performed based on the bright image, and the dark image can be brightened.

  According to the invention of claim 5, further, the second adjustment is performed to adjust the luminance of the pixel in the image for which the average luminance is calculated with reference to the average luminance calculated for each of the plurality of images. Since the second adjustment is effective when the difference between the highest average luminance and the lowest average luminance among the average luminances calculated for each of the plurality of images is less than a predetermined value, the bright image and the dark image are not mixed. Further, the luminance can be adjusted for each image based on the average luminance of each image, and image deterioration can be prevented.

FIG. 1 is a diagram illustrating a vehicle. FIG. 2 is a system block diagram of the vehicle. FIG. 3 is a flowchart for explaining the image processing. FIG. 4 is a flowchart for explaining the image processing. FIG. 5 is a diagram illustrating a vehicle. FIG. 6 is a diagram illustrating a vehicle and a vehicle-mounted camera image. FIG. 7 is a flowchart for explaining image processing. FIG. 8 is a diagram illustrating a vehicle-mounted camera image. FIG. 9 is a diagram showing a vehicle-mounted camera image. FIG. 10 is a flowchart for explaining image processing. FIG. 11 is a diagram illustrating a vehicle-mounted camera image. FIG. 12 is a flowchart for explaining image processing. FIG. 13 is a flowchart for explaining image processing. FIG. 14 is a diagram for explaining image processing. FIG. 15 is a flowchart for explaining image processing. FIG. 16 is a flowchart for explaining image processing.

A vehicle-mounted vehicle peripheral image display system that is a representative embodiment of the present invention will be described below with reference to the accompanying drawings.
<Representative examples>
[vehicle]
First, a vehicle periphery image display system mounted on a vehicle equipped with an engine or a traveling electric motor as a drive source will be described with reference to FIG.

  The vehicle periphery image display system S1 is configured by electrically connecting an image processing system, the in-vehicle device 100, various switches, and sensors to the in-vehicle network N.

  The image processing system includes a control device 10, a first imaging unit 200 that is an imaging system, a second imaging unit 201, a third imaging unit 202, and a fourth imaging unit 203.

The control device 10 is installed under the passenger seat of the vehicle 1. The in-vehicle device 100 is installed between a driver seat and a passenger seat on the dashboard of the vehicle 1. The first imaging unit 200 is installed on the front of the vehicle 1, the second imaging unit 201 is installed on the left side mirror 2 of the vehicle 1, the third imaging unit 202 is installed on the right side mirror 3 of the vehicle 1, and the fourth The imaging unit 203 is installed on the back of the vehicle 1. Various switches and sensors are appropriately arranged at positions for achieving the purpose. The in-vehicle network N is disposed inside the body of the vehicle 1 such as the engine room or pillar of the vehicle 1.
(Vehicle surrounding image display system)
As shown in FIG. 2, the vehicle periphery image display system S <b> 1 includes an image processing system S <b> 2 and an in-vehicle device 100.
(Image processing system)
As illustrated in FIG. 2, the image processing system S2 includes a control device 10 and an imaging system S3.
(Imaging system)
As shown in FIG. 2, the imaging system S3 includes a first imaging unit 200, a second imaging unit 201, a third imaging unit 202, and a fourth imaging unit 203.

  The first imaging unit 200 is a front camera that is a vehicle-mounted camera, the second imaging unit 201 is a left-side camera that is a vehicle-mounted camera, and the third imaging unit 202 is a right-side camera that is a vehicle-mounted camera, The 4th imaging part 203 is a back camera which is a vehicle-mounted camera. Each of these imaging units includes an imaging element such as a CCD or a CMOS, and acquires an image electronically.

  The first imaging unit 200 is provided in the vicinity of the license plate mounting position at the front end of the vehicle 1, and the optical axis thereof is directed in the straight traveling direction of the vehicle 1. The second imaging unit 201 is provided in the left side mirror 2, and its optical axis is directed to the outside along the left direction with reference to the straight traveling direction of the vehicle 1. The third imaging unit 202 is provided in the light side mirror 3, and its optical axis is directed to the outside along the right direction with reference to the straight traveling direction of the vehicle 1. The fourth imaging unit 203 is provided in the vicinity of the license plate mounting position at the rear end of the vehicle 1, and the optical axis thereof is directed in the direction opposite to the straight traveling direction of the vehicle 1. Note that the attachment position of the first imaging unit 200 and the fourth imaging unit 203 is desirably the substantially horizontal center, but may be a position slightly shifted in the horizontal direction from the horizontal center.

A fish-eye lens or the like is employed as the lens of the first imaging unit 200 to the fourth imaging unit 203, and the first imaging unit 200 to the fourth imaging unit 203 have an angle of view of 180 degrees or more. For this reason, it is possible to capture the entire periphery of the vehicle 1 by using these four imaging units.
(Control device)
As illustrated in FIG. 2, the control device 10 includes a control unit 11, an AGC unit 12, a nonvolatile storage unit 13, a volatile storage unit 14, and an image adjustment unit 15.

  The control unit 11 includes a valid / invalid control unit 16, a first image luminance adjustment unit 17, a second image luminance adjustment unit 18, an image composition unit 19, and an image display unit 20. That is, the control unit 11 exhibits an effective / ineffective control function, a first image brightness adjustment function, a second image brightness adjustment function, an image composition function, and an image display function, which will be described later.

  Specifically, the control unit 11 is a microcomputer including a CPU, a ROM, a RAM, and the like. The CPU executes a program for demonstrating the function stored in the ROM while using the RAM serving as a working area.

  The AGC unit 12 includes an image luminance calculation unit 22. That is, the AGC unit 12 exhibits an image luminance calculation function, which will be described later.

  Specifically, the AGC unit 12 is configured by an electronic circuit or the like. An image captured by the imaging system S3 is input to an electronic circuit or the like, and the electronic circuit or the like performs its function based on the input image.

  The nonvolatile storage unit 13 is composed of an EEPROM. The controller 11 serves as a backup memory that stores parameters when the controller 11 performs various functions.

  The volatile storage unit 14 includes a RAM. When the control unit 11 stores a fixed amount of the synthesized image and displays the image stored by the image display function, the controller 11 deletes the displayed image and stores the latest image. That is, it plays the role of a video memory.

  The image adjustment unit 15 is a contour correction function that corrects the contour of one vehicle peripheral image combined from a plurality of images by the image combining function of the control unit 11, and a contrast correction function that corrects the contrast of one vehicle peripheral image. It exhibits a saturation correction function that corrects the saturation of a single vehicle periphery image.

  Specifically, the image adjustment unit 15 is configured by an electronic circuit or the like. One vehicle periphery image synthesized by the control unit 11 is input to an electronic circuit or the like, and the electronic circuit or the like performs its function based on the input image.

As described above, the image processing system S2 including the control device 10 and the imaging system S3 can be considered as an image composition device. .
(In-vehicle device)
As shown in FIG. 2, the in-vehicle device 100 includes a display / operation unit 101 and a control unit 112.

  The display / operation unit 101 exhibits a display function and an operation function.

  Specifically, the display / operation unit 101 includes a touch panel display. The display / operation unit 101 performs data display when display data such as an image is controlled by the control unit 11. When the icon to be displayed is touched by the user, the display / operation unit 101 causes the control unit 11 to perform the function intended by the icon.

  The control unit 102 exhibits the above-described display control function, navigation function, audio function, broadcast data communication function, and the like.

  Specifically, the control unit 102 includes a CPU, a ROM, a RAM, and the like. The CPU executes a program for demonstrating the function stored in the ROM while using the RAM serving as a working area.

The in-vehicle device 100 also exhibits a large-capacity storage device that stores map data necessary for performing a navigation function, an audio playback device necessary for performing an audio function, and a broadcast data communication function. It has a communication unit that is necessary for this.
(Switch / sensor)
Various switches and sensors are connected to the in-vehicle network N together with the control device 10 and the in-vehicle device 100. The switches and sensors include, for example, a lighting switch 300, a wiper switch 301, an illuminance sensor 302, and a setting switch 303.

  The lighting switch 300 is a switch operated by the user to brighten the front of the vehicle 1 at night or when passing through a tunnel, and is provided near the driver's seat. When the illumination switch 300 is operated by the user, illumination (not shown) is turned on and the front of the vehicle 1 is brightened. That is, the control part 11 of the control apparatus 10 estimates that the surroundings of the vehicle 1 are dark when the lighting switch 300 is operated by the user.

  The wiper switch 301 is a switch that is operated to remove water droplets on the windshield and rear glass of the vehicle 1 on which the user travels in rainy weather or dense fog, and is provided near the driver's seat. When the user operates the wiper switch 301, a wiper (not shown) is activated, and water droplets on the windshield and rear glass of the vehicle 1 are removed. That is, when the wiper switch 300 is operated by the user, the control unit 11 of the control device 10 estimates that the outside of the vehicle 1 is rainy or dense fog.

  The illuminance sensor 302 is for detecting the degree of light that enters the outside of the vehicle 1 or the inside of the vehicle 1, and is appropriately provided at a position where the purpose can be achieved. That is, the control unit 11 of the control device 10 estimates the brightness around the vehicle 1 based on a signal indicating the degree of light detected by the illuminance sensor 302 received at a predetermined period or at a predetermined timing.

  The setting switch 303 is an operation means for the user to set which of the first image luminance adjustment processing or the second image luminance adjustment processing described later is to be performed by the control unit 11 of the control device 10.

[Image processing]
Next, image processing executed by the control device 10 will be described with reference to FIG. The functions exhibited by the control device 10 include an image luminance calculation function exhibited by the AGC unit 12, and an effective / ineffective control function, a first image luminance adjustment function, a second image luminance adjustment function exhibited by the control unit 11, There are an image composition function and an image display function, and further there is an image adjustment function exhibited by the image adjustment unit 15. The image adjustment function includes a contour correction function, a contrast correction function, and a saturation correction function.

  In order to perform these functions, the control device 10 performs image luminance calculation processing S1000, valid / invalid processing S2000, first image luminance adjustment processing S3000, second image luminance adjustment processing S4000, image composition processing S5000, image shown in FIG. An adjustment process S6000 and an image display process S7000 are executed.

  The control unit 11, the AGC unit 12, and the image adjustment unit 15 of the control device 10 start executing image processing at a predetermined cycle when the control device 10 is activated by the user. That is, the process proceeds to step S1000 in FIG.

Since the brightness of an image can be expressed by luminance, it will be described below as “luminance”.
(Image brightness calculation processing)
In step S1000, the AGC unit 12 of the control device 10 executes an image luminance calculation process. Details of the image luminance calculation processing will be described with reference to FIG.

  The AGC unit 12 starts executing the image luminance calculation process, and proceeds to step S1001.

  In step S <b> 1001, the AGC unit 12 captures (receives) an image captured by the first imaging unit 200 to the fourth imaging unit 203 and transmitted to the control unit 11.

  Here, an image captured by the first imaging unit 200 to the fourth imaging unit 203 will be described. As illustrated in FIG. 5, the first imaging unit 200 transmits an image in the imaging range H <b> 1 to the control unit 11. The second imaging unit 201 transmits an image in the imaging range H2 to the control unit 11. The third imaging unit 202 transmits an image in the imaging range H3 to the control unit 11. The third imaging unit 202 transmits an image in the imaging range H3 to the control unit 11. The fourth imaging unit 203 transmits an image in the imaging range H4 to the control unit 11. As described above, a fish-eye lens or the like is employed as the lens included in the first imaging unit 200 to the fourth imaging unit 203, and the imaging ranges H1 to H4 of the first imaging unit 200 to the fourth imaging unit 203 illustrated in FIG. As shown, it has an angle of view of 180 degrees or more.

  Therefore, the image displayed by the first imaging unit 200 is a vehicle peripheral image of 180 degrees or more in the straight traveling direction of the vehicle 1 as shown in an image Z1 of FIG. The image displayed by the second imaging unit 201 is a vehicle image of 180 degrees or more in the left direction with reference to the straight traveling direction of the vehicle 1 as shown in an image Z2 in FIG. The image displayed by the third imaging unit 202 is a vehicle image of 180 degrees or more in the right direction with reference to the straight traveling direction of the vehicle 1 as shown in an image Z3 in FIG. The image projected by the fourth imaging unit 203 is a vehicle image of 180 degrees or more in the direction opposite to the straight traveling direction of the vehicle 1 as shown by an image Z4 in FIG. Next, the process proceeds to step S1002.

  In step S1002, the AGC unit 12 calculates an average luminance for each image in the images Z1 to Z4 received from the first imaging unit 200 to the fourth imaging unit 203. The calculation of the average luminance of the image means that the total luminance assigned to each pixel in the image is divided by the total number of pixels. At that time, the AGC unit 12 determines whether or not the luminance assigned to the pixel exceeds a threshold value that is a constant value, and if it exceeds, the luminance is the sum of the luminance in the calculation of the average luminance. Not included, and not included in the total number of pixels. The threshold value is a value for excluding the high luminance from the calculation of the average luminance of the image.

  That is, when there is an extremely bright portion in the image, there is a possibility that overexposure (saturation) or the like has occurred. In this case, the average luminance is increased due to the high luminance applied to the pixels in the bright portion, and the image corrected based on the correction coefficient obtained by the average luminance in the subsequent processing is the high luminance in the bright portion. It prevents the overall brightness from being affected. Next, the process proceeds to step S1003.

In step S1003, the control unit 11 calculates a correction coefficient (gain adjustment value) based on the maximum average luminance among the average luminances calculated for each of the images Z1 to Z4 received from the AGC unit. The correction coefficient is a value obtained by dividing the maximum average luminance by the respective average luminance. For example, as shown in FIG. 8, if the average luminance of the image Z1 is the maximum, the correction coefficient of the image Z1 is the average luminance of the image Z1 / the average luminance of the image Z1, and the correction coefficient of the image Z2 is The average brightness of Z1 / the average brightness of image Z2, the correction coefficient of image Z3 is the average brightness of image Z1 / the average brightness of image Z3, and the correction coefficient of image Z4 is the average brightness of image Z1 / the average brightness of image Z4. Average brightness. Next, the process proceeds to step S2000 in FIG.
(Valid / invalid processing)
In step S2000, the valid / invalid control unit 16 of the control unit 11 executes a valid / invalid process. Details of the image luminance calculation processing will be described with reference to FIG.

  In step S2001, the control unit 11 receives a signal from the setting switch 303 via the in-vehicle network N. The control unit 11 determines whether the received signal indicates setting 1 or setting 2. If the control unit 11 determines that the received signal indicates setting 1, the control unit 11 proceeds to step S2002 (setting 1 in step S2001). If the control unit 11 determines that the received signal indicates setting 2, the control unit 11 proceeds to step S2003 (setting 2 in step S2001).

  In step S2002, the first flag set in the RAM included in the control unit 11 is turned on. Next, the process proceeds to return.

  In step S2003, the second flag set in the RAM included in the control unit 11 is turned on. Next, the process proceeds to return.

  Note that the first flag and the second flag set in the RAM are set to 1 at the time of manufacture, and the control unit 11 sets the values set in the end process in the initial process. Next, the process proceeds to step S3000 in FIG.

  Now, as shown in FIG. 8, when the overall brightness is different between the images Z1 to Z4, the brightness of the image Z1 having the highest overall brightness and the image Z2 having the lowest overall brightness. When the difference is large, that is, when the difference between the average luminance of the image Z1 and the average luminance of the image Z2 is greater than or equal to a predetermined value, these images are combined into one image later, so that the luminance is regarded as one image. When adjustment (luminance correction) is performed, a problem occurs in image quality.

  The problem will be specifically described. After the control unit 11 calculates the average luminance of the images Z1 to Z4 captured as one image, the luminance assigned to all the pixels of all the images is set to the luminance of the dark pixel based on the average luminance. If the adjustment (correction) is performed so that the brightness of a bright pixel is darkened by a predetermined percentage, the image Z2 that is too dark and difficult to see as shown in FIG. Therefore, there is a problem that the image Z1 that is easy to view becomes brighter and difficult to view. In other words, if correction based on the same average brightness is performed on an extremely dark image and the brightest image, the outline of the object in the bright image becomes thin or disappears due to being dragged by the extremely dark image, making it difficult to see. End up.

  As described above, in the case of the vehicle periphery image, a part of the image before the composition may be extremely darker than the other images, and the occurrence of such a problem is sufficiently considered. For example, when only the rear part of the vehicle 1 is in a dark covered parking lot, or when the side surface of the vehicle 1 is covered by a shade such as a building.

Therefore, in order to prevent such a problem from occurring, the subsequent processing is executed so that there is no sense of incongruity when one composite image is generated so that the luminance in the images Z1 to Z4 is uniform.
(First image brightness adjustment processing)
In step S3000, the first image luminance adjustment unit of the control unit 11 executes a first image luminance adjustment process. Details of the first image luminance calculation processing will be described with reference to FIG.

  In step S3001, the control unit 11 determines whether or not the first flag set in the provided RAM is on. When it is determined that the first flag is on, the control unit 11 proceeds to step S3002 (YES in step S3001). If control unit 11 does not determine that the first flag is off, the control unit 11 proceeds to return (NO in step S3001).

  In step S3002, the calculated correction coefficient is multiplied by the luminance in the images Z1 to Z4. Multiplying a correction coefficient for an image means multiplying each luminance assigned to each pixel in the image by a correction coefficient.

  That is, in step S1003 in FIG. 4 and step S3002 in FIG. 10, the control unit 11 determines the image having the maximum average luminance as a representative image and adjusts the luminance of other images based on the determined representative image. By doing so, it is possible to unify the brightness of the images as in the images Z1 to Z4 shown in FIG. 11 and to improve the visibility of extremely dark images. It is possible to provide an image that does not give the viewer a sense of incongruity.

  Since the correction coefficient is 1 for the image with the maximum average luminance, that is, the image Z1, the multiplication process may be omitted. Further, the multiplication of the image with the maximum average luminance, that is, the image Z2 to the image Z4 which are images other than the image Z1, is applied to the luminance assigned to the pixels exceeding the threshold value (pixels close to saturation). The multiplication process may be omitted. Next, the process proceeds to step S3004.

  In step S3004, the control unit 11 transmits the multiplied (corrected) image Z1 to the volatile storage unit including the image Z4.

  Here, the reason for correcting the image Z1 to the image Z4 based on the correction coefficient calculated based on the image having the maximum average luminance will be described.

  First, among images Z1 to Z4, an ideal image having the most suitable luminance distribution when expressed in a histogram is used as a representative image, and the luminance of other images is adjusted (corrected) based on this representative image. ) However, in that case, it is necessary to provide a circuit or software with logic to realize the processing, which may increase the cost, and in order to avoid such disadvantages, it is easy to guess an ideal image. There is a need to.

  Therefore, the aperture of each imaging unit is pre-adjusted to the level (luminance) that the viewer feels the image is most comfortable to see, based on the brightness of the surroundings of the vehicle in the daytime, in the captured image. Therefore, the control unit 11 estimates that the image having the highest average brightness among the images captured by the aperture is the ideal image that is most easily viewed by the viewer.

Therefore, the reason is that by correcting other images based on an ideal image that is easily estimated, that is, an image having the maximum average luminance, an increase in cost is suppressed while performing appropriate luminance correction. Is based on being able to. Next, the process proceeds to step S4000 in FIG.
(Second image brightness adjustment processing)
In step S4000, the second image luminance adjustment unit 18 of the control unit 11 executes a second image luminance adjustment process. Details of the second image luminance calculation processing will be described with reference to FIG.

  In step S4001, the control unit 11 determines whether or not the second flag set in the RAM included in the control unit 11 is on. When it is determined that the second flag is on, control unit 11 proceeds to step S4002 (YES in step S4001). When determining that the second flag is off, the control unit 11 proceeds to return (NO in step S4001).

  In step S4002, the control unit 11 adjusts (corrects) the image for which the average luminance is calculated based on the average luminance of each of the images Z1 to Z4 received from the AGC unit 12. Specifically, the control unit 11 obtains a correction coefficient (gain adjustment value) by dividing the predetermined value by the value obtained by subtracting the increase rate from the average luminance, and is applied to the pixel in the image for which the average luminance is calculated. Multiply (correct) the luminance by the correction coefficient. Since the increase rate is set in association with the average luminance in the register table provided in the control unit 11, the increase rate is referred to from the register table based on the average luminance. As a result, the dark image as a whole becomes brighter as a viewer and can be adjusted to an easy-to-see image. Next, the process proceeds to step S4003.

In step S4003, the control unit 11 transmits the image to the volatile storage unit including the image Z1 to the image Z4 subjected to the multiplication (correction). The process proceeds to step S5000 in FIG.
(Image composition processing)
In step S5000, the image composition unit 19 of the control unit 11 performs image composition processing. Details of the image composition processing will be described with reference to FIG.

  In step S5001, the control unit 11 projects the corrected images Z1 to Z4 stored in the volatile storage unit provided on the three-dimensional curved surface P in the virtual three-dimensional space shown in FIG. The three-dimensional curved surface P has, for example, a substantially hemispherical shape (a bowl shape), and a central portion (a bottom portion of the bowl) is determined as a position where the vehicle 1 exists. The correspondence between the positions of the pixels included in the images Z1 to Z4 and the positions of the pixels on the solid curved surface P is determined in advance. For this reason, the value of each pixel of the three-dimensional curved surface P is determined based on this correspondence and the value of each pixel included in the images Z1 to Z4.

  The correspondence between the positions of the pixels of the image Z1 to the image Z4 and the positions of the pixels of the three-dimensional curved surface P is determined by the arrangement of the first imaging unit 200 to the fourth imaging unit 203 that are four on-vehicle cameras in the vehicle 1 (between each other). Distance, ground height, optical axis angle, etc.). Therefore, the table data indicating this correspondence is included in the vehicle type data stored in the nonvolatile storage unit 13.

  Further, polygon data indicating the shape and size of the vehicle body included in the vehicle type data is used, and a vehicle image which is a polygon model indicating the three-dimensional shape of the vehicle 1 is virtually configured. The configured vehicle image is arranged in a substantially hemispherical central portion defined as the position of the vehicle 1 in the three-dimensional space in which the three-dimensional curved surface P is set.

  Further, the virtual viewpoint AGL is set by the control unit 11 for the three-dimensional space where the solid curved surface P exists. The virtual viewpoint AGL is defined by the viewpoint position and the viewing direction, and is set at an arbitrary viewpoint position corresponding to the periphery of the vehicle 1 in this three-dimensional space toward an arbitrary viewing direction. Next, the process proceeds to step S5002.

  In step S5002, the control unit 11 cuts out a necessary area on the three-dimensional curved surface P as an image according to the set virtual viewpoint AGL. The relationship between the virtual viewpoint AGL and the necessary area on the three-dimensional curved surface P is determined in advance, and is stored in advance in the nonvolatile storage unit 13 or the like as table data. On the other hand, rendering is performed on a vehicle image composed of polygons according to the set virtual viewpoint AGL, and the resulting two-dimensional vehicle image is superimposed on the cut out image. Thereby, the composite image which shows a mode that the vehicle 1 and the periphery of the vehicle 1 were seen from arbitrary virtual viewpoints is produced | generated.

  For example, when the virtual viewpoint AGL1 in which the viewpoint position is a position just above the center of the position of the vehicle 1 and the visual field direction is a direction immediately below the vehicle 1 is set, the vehicle 1 ( In practice, a composite image G1 showing the vehicle image) and the surroundings of the vehicle 1 is generated. Further, as shown in the figure, when the virtual viewpoint AGL2 in which the viewpoint position is the left rear of the position of the vehicle 1 and the visual field direction is substantially in front of the vehicle 1 is set, the entire periphery from the left rear of the vehicle 1 is set. As seen, a composite image G2 showing the vehicle 1 (actually a vehicle image) and the surroundings of the vehicle 1 is generated.

  Note that in the case of actually generating a composite image, it is not necessary to determine the values of all the pixels of the three-dimensional curved surface P, and only the values of the pixels in the area necessary corresponding to the set virtual viewpoint AGL are imaged. By determining based on Z1 to image Z4, the processing speed can be improved. Next, the process proceeds to step S5003.

In step S5003, the control unit 11 transmits the generated composite image G1 or G2 to the image adjustment unit 15. The process proceeds to step S6000 in FIG.
(Image adjustment processing)
In step S6000, the image adjustment unit 15 performs image adjustment processing. Details of the image adjustment processing will be described with reference to FIG.

  In step S6001, the image adjustment unit 15 corrects the contour of the object in the composite image G1 or G2 received from the control unit 11. The contour correction is, for example, adjusting the strength of the contour of an object according to the distance or performing smooth correction of a curve. Next, the process proceeds to step S6002.

  In step S <b> 6002, the image adjustment unit 15 corrects the contrast between objects or object portions in the composite image G <b> 1 or G <b> 2 received from the control unit 11. The contrast correction is, for example, performing contrast strength adjustment between objects or between object parts according to the distance. Next, the process proceeds to step S6003.

In step S6003, the image adjustment unit 15 corrects the saturation between objects in the composite image G1 or G2 received from the control unit 11. Saturation correction is, for example, adjusting the strength of the color of an object according to the distance. Next, the process proceeds to step S7000 in FIG.
(Image display processing)
In step S7000, the image display unit 20 of the control unit 11 executes image display processing. Details of the image display processing will be described with reference to FIG.

  In step S <b> 7001, the image display unit 20 reads the composite image stored in the volatile storage unit 14. At that time, when the composite image G1 or G2 stored in the volatile storage unit 14 exceeds a certain amount, the image display unit 20 deletes the oldest composite image and stores the latest composite image, or Then, storage control for overwriting the oldest synthesized image with the latest synthesized image is executed. Next, the process proceeds to step S7002.

  In step S <b> 7002, the image display unit 20 controls the display operation unit 101 provided in the in-vehicle device 100 to display the composite image G <b> 1 or G <b> 2 read from the volatile storage unit 14. Specifically, the image display unit 20 updates and displays the composite image G1 or G2 stored in the volatile storage unit 14 from an old image to a new image at a predetermined cycle so that it can be seen as a moving image. The process proceeds to the return in FIG.

<Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible. Other embodiments will be described below. Of course, you may combine the form demonstrated below suitably.

<Modification 1>
In step S1002 in FIG. 4 for explaining the representative embodiment, the AGC unit 12 calculates the average luminance for each of the images Z1 to Z4 received from the first imaging unit 200 to the fourth imaging unit 203. The AGC unit 12 determines whether or not the luminance allocated to the pixel exceeds a threshold value that is a constant value, and if it exceeds, the luminance is not included in the total luminance in the calculation of the average luminance. Although not described in the total number of pixels, the control unit 11 of the control device 10 determines the brightness outside the vehicle and uses the average brightness described above based on the threshold value according to the brightness outside the vehicle. It may be calculated.

  For example, when the control unit 11 determines that the vehicle exterior is dark, the control unit 11 sets the threshold value to a threshold value higher than the above threshold value, and when it is determined that the vehicle exterior is bright, the threshold value is set to the above-described threshold value. Set the threshold value lower than the threshold value.

  Specifically, the control unit 11 determines that the vehicle exterior is dark when receiving an on signal of the lighting switch 300 connected to the in-vehicle network N, and the vehicle when receiving the off signal. Judge that the outside is bright. Further, the control unit 11 determines that the vehicle exterior is dark when receiving the ON signal of the wiper switch 301 connected to the in-vehicle network N, and the vehicle exterior is bright when receiving the OFF signal. Judge. Further, when the control unit 11 receives a signal from the illuminance sensor 302 connected to the in-vehicle network N, the control unit 11 determines the brightness outside the vehicle according to the signal level, and the threshold value is variable according to the brightness. To.

  Note that the threshold value may be arbitrarily set by the user.

  In this way, by changing the threshold value according to the brightness outside the vehicle, the average brightness, which is the basis of the coefficient for correcting the image brightness, is darker when the vehicle exterior is brighter, and when the vehicle exterior is darker Since it is set to the bright side, the visibility of the image corrected based on this average luminance can be improved.

<Modification 2>
In step S2001 in FIG. 7 for explaining the representative embodiment, in step S2001, the control unit 11 receives a signal from the setting switch 303 via the in-vehicle network N, and whether the received signal indicates setting 1 or not. It is determined whether setting 2 is indicated. When the control unit 11 determines that the received signal indicates setting 1, the control unit 11 performs the first image luminance adjustment processing in FIG. 10, and when it determines that the received signal indicates setting 2, the control unit 11 performs the first image luminance adjustment process in FIG. Although it has been described that the two-image luminance adjustment process is executed, the control unit 11 determines that the difference between the highest average luminance and the lowest average luminance among the average luminances of the images Z1 to Z4 received from the AGC unit 12 is a predetermined value or more. If the difference is greater than or equal to a predetermined value, the first flag is turned on, the first image brightness adjustment process is executed in FIG. 10, and the difference is greater than or equal to the first predetermined value. If it is not determined that, the second flag may be turned on, and the second image brightness adjustment process in FIG. 12 may be executed.

  It should be noted that the first predetermined value is set in advance as a value for determining a combination of images that cause the above-described problems. The first flag and the second flag are turned off during the initial process and the end process, and are turned off when the purpose of the function is finished.

  That is, when the combination of images captured by the first imaging unit 200 to the fourth imaging unit 203 is a pattern in which the above-described problem occurs, the first image brightness adjustment process is executed. On the other hand, when the pattern does not cause the above-described problem, it is possible to prevent image degradation by executing the second image brightness adjustment process without executing the first image brightness adjustment process.

<Modification 3>
In step S2001 in FIG. 7 for explaining the representative embodiment, in step S2001, the control unit 11 receives a signal from the setting switch 303 via the in-vehicle network N, and whether the received signal indicates setting 1 or not. It is determined whether setting 2 is indicated. When the control unit 11 determines that the received signal indicates setting 1, the control unit 11 performs the first image luminance adjustment processing in FIG. 10, and when it determines that the received signal indicates setting 2, the control unit 11 performs the first image luminance adjustment process in FIG. Although it has been described that the two-image luminance adjustment process is executed, the control unit 11 has an average luminance exceeding the upper limit value among the average luminances of the images Z1 to Z4 received from the AGC unit 12, and an average exceeding the lower limit value. If the condition that there is brightness is satisfied, the first flag is turned on and the first image brightness adjustment process in FIG. 10 is executed. If the condition is not satisfied, the second flag is turned on and the first flag in FIG. A two-image brightness adjustment process may be executed.

  Note that the upper limit value and the lower limit value are set in advance as values for determining a combination of images that cause the above-described problems.

  That is, when the combination of images captured by the first imaging unit 200 to the fourth imaging unit 203 is a pattern in which the above-described problem occurs, the first image brightness adjustment process is executed. On the other hand, when the pattern does not cause the above-described problem, it is possible to prevent image degradation by executing the second image brightness adjustment process without executing the first image brightness adjustment process.

  Further, in the above-described embodiment, it has been described that various functions are realized in software by the arithmetic processing of the CPU according to the program. However, some of these functions are realized by an electrical hardware circuit. Also good. Conversely, some of the functions realized by the hardware circuit may be realized by software.

  Furthermore, each process in the flowchart for explaining the control of each embodiment is shown as one series for convenience. However, each control unit processes each subdivided process in parallel by the multitask control function. May be.

DESCRIPTION OF SYMBOLS S1 Vehicle periphery image display system S2 Image processing system 10 Control apparatus 11 Control part 12 AGC part 15 Image adjustment part 16 Effective / ineffective control part 17 Image brightness adjustment part 18 Image brightness adjustment part 19 Image composition part 100 Car-mounted machine 101 Display operation part 102 Control unit S3 Imaging system 200 Imaging unit 200 Imaging unit 201 Imaging unit 202 Imaging unit 203 Imaging unit

Claims (5)

An image composition device that is mounted on a vehicle and synthesizes a plurality of images,
Input means for inputting a plurality of images from a plurality of imaging means mounted on the outside of the vehicle toward different directions,
Determining means for determining one of the plurality of images as a representative image based on the luminance of the plurality of input images;
First adjusting means for adjusting the brightness of another image with reference to the representative image;
A synthesizing unit that synthesizes the representative image and the other image of which brightness has been adjusted, into one vehicle peripheral image showing a state of the periphery of the vehicle;
Equipped with a,
The plurality of images are composed of a predetermined number of pixels,
It said determining means, the image synthesizing apparatus characterized that you determine the representative image an image of the calculation to the highest average luminance average luminance of all pixels in the image for each of the plurality of images. The image composition apparatus according to claim 1,
The determining means does not include the pixel in the calculation of the average luminance when the pixel in the image has a luminance of a certain value or more . The image composition device according to claim 2,
And a control unit that enables the first adjustment unit when a difference between the highest average luminance and the lowest average luminance among the average luminances calculated for each of the plurality of images is equal to or greater than a predetermined value. An image composition device characterized by the above. The image composition device according to claim 3.
Furthermore, with the average brightness calculated for each of the plurality of images as a reference, second adjustment means for adjusting the brightness of the pixels in the image for which the average brightness is calculated,
The control means enables the second adjusting means when the difference between the highest average brightness and the lowest average brightness among the average brightness calculated for each of the plurality of images is less than a predetermined value. A featured image synthesizer. An image composition method that is mounted on a vehicle and combines a plurality of images,
An input step of inputting a plurality of images from a plurality of imaging means mounted on the outside of the vehicle toward different directions,
A determination step of determining one of the plurality of images as a representative image based on the luminance of the plurality of input images;
An adjustment step of adjusting the brightness of other images based on the representative image;
Synthesizing the representative image and the other image whose brightness has been adjusted to a single vehicle periphery image showing a state of the periphery of the vehicle;
With
The plurality of images are composed of a predetermined number of pixels,
It said determining means, an image synthesis method and determining an image of the calculated and the highest average luminance average luminance of all pixels in the image for each of the plurality of images representative of the image.