JP2022029346A - Image processing device and image processing program - Google Patents

Abstract

To reduce the amount of deviation and improve the visibility of a target image which is an object to be watched and monitored as compared with an area other than the target image without depending on the skill of an observer who monitors a composite image in a compositing process between images taken by a plurality of cameras.SOLUTION: A correction coefficient A is incorporated in a formula for executing a deformation process for correcting a parallax of images taken by an L camera 102 and an R camera 104. An actual correction target by the correction coefficient A is an image in a region other than the object to be watched (target image) identified under a predetermined condition, and a viewpoint position is shifted when multiplied by the correction coefficient A, to make the image a so-called out-of-focus image. In other words, among images on a virtual screen 114 that are images finally displayed on a monitor unit 106, only the target image is made in focus, and the other regions are made out of focus.SELECTED DRAWING: Figure 12

Description

The present invention relates to an image processing device and an image processing program for synthesizing a vehicle surrounding image divided and captured.

In the technology of providing overlapping areas (overlaps) in images taken by two or more cameras installed at intervals and synthesizing them, the image shift is basically based on a predetermined virtual screen (projection position). To correct.

At this time, there may be a blind spot that is not displayed on the front side of the virtual screen, and there may be a region where the subject becomes a double image on the back side of the virtual screen.

Patent Document 1 describes reducing the amount of displacement of the peripheral edge of an object on the boundary line of a composite image.

More specifically, the image projection screen setting unit sets a plurality of projection screens perpendicular to the virtual line of sight connecting the virtual viewpoint and the vanishing point by changing the distance from the virtual viewpoint. For each projection screen, the image composition unit uses the line of intersection between the boundary surface parallel to the virtual line of sight and a predetermined distance as the boundary, and projects the rear image, the left rear side image, and the right rear side image. Project to to form multiple candidates for the composite image. The image evaluation unit evaluates the amount of deviation of the periphery of the object on the boundary in the composite image for each composite image, and the image selection unit selects the composite image having the smallest deviation amount in each composite image. As a result, the amount of displacement of the peripheral edge of the object on the boundary of the composite image can be reduced.

Japanese Unexamined Patent Publication No. 2013-118508

However, in the technique of Patent Document 1, it is described that the amount of deviation at the joint of images taken by two cameras is reduced, but from the images on the image projection screen in which the amount of deviation is reduced, It is not possible to emphasize the object to be watched and monitored in particular, and the recognition of the object to be watched and monitored is left to the observer who sees the image.

The present invention reduces the amount of deviation in the compositing process between images taken by a plurality of cameras, and is an object of interest that should be watched and monitored without depending on the skill of the observer who monitors the composite image. It is an object of the present invention to obtain an image processing device and an image processing program capable of enhancing visibility as compared with an area other than the region of interest.

The image processing apparatus according to the present invention acquires a plurality of image information captured by a plurality of image pickup devices so as to form an overlapping portion in which mutual image pickup regions overlap, and synthesizes the acquired plurality of image information. At least one virtual viewpoint is set for the overlapped portion, and as an image for the virtual viewpoint, an interpolated image generated by interpolating from images taken by the plurality of image pickup devices and the plurality of images are displayed. By synthesizing images other than the overlapped portion taken by the image pickup device of the above, an image having a focal position on a predetermined virtual screen is generated, and a predetermined condition is obtained from the images taken by the plurality of image pickup devices. The attention image specified in the above is extracted, the distance from each of the plurality of image pickup devices to the attention image is calculated, and when the attention image is extracted, from each of the plurality of image pickup devices to the attention image. It has a control unit that executes a process of providing a difference in the degree of visibility of an image between a region of interest image and a region other than the region of interest based on the distance.

According to the present invention, in a plurality of image pickup devices, a plurality of image information captured so as to form an overlapping portion where mutual imaging regions overlap is acquired, and the acquired plurality of image information are combined and displayed. In addition, at least one virtual viewpoint is set for the overlapping portion, and as an image for the virtual viewpoint, an interpolated image generated by interpolating from an image taken by a plurality of image pickup devices and an image taken by the plurality of image pickup devices. An image having a focal position on a predetermined virtual screen is generated by combining with an image other than the overlapped portion.

Here, in the present invention, the attention image specified by a predetermined condition is extracted from the images taken by the plurality of image pickup devices, and the distance from each of the plurality of image pickup devices to the attention image is calculated. When the attention image is extracted, there is a difference in the degree of visibility of the image between the attention image region and the region other than the attention image region based on the distance from each of the plurality of image pickup devices to the attention image. Is executed. For example, the degree of visibility of the image can be replaced with focus adjustment, the focus image area is focused, and the focus is intentionally shifted except for the focus image area.

Since the image of interest is in focus, the image of interest can be easily identified, and the time required for identification can be shortened.

This reduces the amount of deviation in the compositing process between images taken by multiple cameras, and at the same time, it does not depend on the skill of the observer who monitors the composite image, and it is an object of interest that should be watched and monitored. , The visibility can be improved as compared with the area other than the area of interest.

The present invention is characterized in that the distance to the attention image is calculated by using the parallax information obtained from the plurality of image information captured by the plurality of image pickup devices.

This makes it possible to utilize the peculiar phenomenon that occurs when a plurality of cameras are used for the acquisition of distance information.

In the present invention, the process of providing the difference in the degree of visibility intentionally focuses on a region other than the attention image among the images having a focus on the virtual screen in the synthesis process. It is characterized by being a blurring process.

This makes use of the fact that, for example, in the recognition of information through human vision, an image that is in focus has better visibility than an image that is out of focus, and the focus is on the virtual screen. By intentionally defocusing a region other than the attention image in the image, the attention image can be clearly seen and the visibility can be improved.

The present invention is characterized in that the difference in the degree of visibility includes at least one of contrast adjustment and marking that follows the movement of the attention image, and is a process for emphasizing the attention image.

As a result, a process of emphasizing the image of interest is added to the image that is in focus and is combined on the virtual screen. The emphasizing process includes, for example, at least one of contrast adjustment and marking that follows the movement of the image of interest. By combining with the process of blurring the focus of the area other than the attention image, the visibility of the attention image can be further improved.

In the present invention, the plurality of image pickup devices are image pickup devices that divide and image the rear of the vehicle, and the image synthesized by the control unit is visually displayed by the driver while driving, and the attention image. However, it is characterized by being an object to be watched that exists behind the own vehicle.

As a result, it can be effectively used as a substitute for the rear-view mirror of the vehicle, and the image of interest can be automatically emphasized. For example, an image of interest (for example, an approaching vehicle) can be highlighted and displayed while driving.

In the present invention, the predetermined condition for extracting the attention image is based on a predetermined priority, and as an element for determining the priority, at least the distance to the attention image existing behind the own vehicle. , And the moving speed of the image of interest.

In the present invention, the vehicle is a moving body including an automobile, an agricultural machine, a construction machine, a motorcycle, and a bicycle, and all the moving bodies located behind the own vehicle are candidates for the attention image. Objects that can be the image of interest include moving objects other than vehicles (for example, people, animals, signs, obstacles, etc.). The size of the image of interest may be included as an element for determining the priority.

The image processing program according to the present invention is characterized in that a computer is operated as a control unit of the image processing apparatus.

As described above, in the present invention, in the compositing process between images taken by a plurality of cameras, the amount of deviation is reduced, and the object to be watched and monitored without depending on the skill of the observer who monitors the composite image. The visibility of the attention image is higher than that of the region other than the attention image.

It is a schematic block diagram which shows the structure of the image processing apparatus which concerns on 1st Embodiment. It is a front view which shows the arrangement state of the virtual cameras when the two cameras applied to the image processing apparatus which concerns on 1st Embodiment are seen from the direction which intersects the optical axis. It is a schematic block diagram which shows the detail of the viewpoint interpolation image generation part in the image processing apparatus shown in FIG. 1. (A) is a control flowchart showing the processing flow of the image processing apparatus according to the first embodiment, and (B) is a control flowchart showing the attention image enhancement processing subroutine executed in step 156 of FIG. 4 (A). be. (A) is a conceptual diagram showing the results of the synthesis processing and the viewpoint generation processing of the overlap portion of the virtual camera array method according to the first embodiment, and (B) is a comparative example thereof, and images taken by two cameras. It is a conceptual diagram which shows the result of image composition processing. It is a front view when the camera which explains the principle of the image composition of the virtual camera array system of this invention is seen from the direction which intersects the optical axis, and (A) the double at the time of composition of the image taken by two real cameras. The image shift width W0 is shown, (B) shows the double image shift width W1 when one virtual camera is set between two real cameras, and (C) shows the shift width W1 of the two real cameras. The deviation width W2 of the double image when three virtual cameras are set between them is shown. It is a front view when the camera for showing the principle of blind spot elimination in 1st Embodiment is seen from the direction which intersects the optical axis. The vehicle according to the second embodiment is shown, (A) is a plan view of the vehicle, and (B) is a side view of the vehicle. It is the schematic when the front seat is seen from the center of the rear seat in the interior of the vehicle which concerns on 2nd Embodiment. (A) is a front view showing the arrangement state of virtual cameras when the three cameras applied to the vehicle according to the second embodiment are viewed from the direction intersecting the optical axis, and (B) is the second. It is a conceptual diagram which shows the result of the synthesis processing of the virtual camera array system and the viewpoint generation processing of the overlap part which concerns on embodiment of. FIG. 10B is a conceptual diagram showing the result of performing a process of defocusing an image in a region other than the image of interest with respect to FIG. 10B. (A) is a front view of the image displayed on the monitor of the rearview mirror portion when the focus of the image in the area other than the attention image is blurred in the second embodiment, and (B) is the image before the focus blur. The front view of (C) is a front view of an image in which only the overlapped portion is blurred as a modification of the focus blur.

"First embodiment"

FIG. 1 shows an image processing device 100 using a virtual camera array according to the first embodiment of the present invention.

The image processing apparatus 100 includes an L camera 102 and an R camera 104 as mounting cameras. In the image processing apparatus 100, the images taken by the L camera 102 and the R camera 104 are combined and output to the monitor unit 106.

The monitor unit 106 includes a display driver 108 and a display screen 110, and displays a composite image on the display screen 110 based on the image information received from the image processing device 100.

As shown in FIG. 2, the L camera 102 and the R camera 104 are arranged so as to be arranged in the left-right direction when viewed from the front of FIG. 2. In the first embodiment, the L camera 102 and the R camera 104 are distinguished into left (L) and right (R) in relative positional relationship, but in the depth direction (vertical direction) of FIG. It may be arranged so as to line up with. The L camera 102 and the R camera 104 have the visual field ranges WL and WR shown by the alternate long and short dash line in FIG. 2, respectively. Among them, the image reflected on the virtual screen 114 described later is shown by diagonal lines (the same applies to FIG. 5). It is used as a display image.

As shown in FIG. 2, it is preferable that the optical axes of the L camera 102 and the R camera 104 are not parallel to each other but are outward to each other, and the views of the L camera 102 and the R camera 104 are different from each other. .. Although the optical axes of the L camera 102 and the R camera 104 are oriented outward from each other, they may be parallel to each other.

The overlapping portion 112 (see FIG. 2) is present in the images captured by the L camera 102 and the R camera 104.

In the overlap portion 112, when the image information of the L camera 102 and the image information of the R camera 104 are simply combined, the subject behind the virtual screen 114 can be seen twice due to the mutual parallax (double). Image) Phenomenon occurs. The double image depends on the distance (distance) between the L camera 102 and the R camera 104 (details will be described later in FIG. 6).

Further, when the image information of the L camera 102 and the image information of the R camera 104 are simply combined, the blind spot region 116 (see FIG. 2) exists on the front side of the virtual screen 114.

In the first embodiment, an image from a virtual viewpoint adjusted for parallax is generated from the image captured by the L camera 102 and the image captured by the R camera (including the field of view range WL and WR in FIG. 2). , The generation of double images, and the blind spot area 116 were reduced. It should be noted that the viewpoints may be generated at equal intervals, but by narrowing the viewpoint intervals, the joints can be made inconspicuous. Further, on the side connected to other than the overlap portion of the captured image of the L camera 102 of the generated image, the virtual viewpoint is connected to the vicinity of the position of the L camera 102 and other than the overlap portion of the captured image of the R camera 104 of the generated image. By setting the virtual viewpoint near the position of the R camera 104, it is possible to make the joint at the portion connected by the image connecting portion inconspicuous.

As shown in FIG. 1, the L camera 102 and the R camera 104 are connected to the viewpoint interpolation image generation unit 118 of the image processing device 100, respectively. The viewpoint interpolation image generation unit 118 generates a virtual viewpoint between the L camera 102 and the R camera 104, and generates an image viewed from the virtual viewpoint as an interpolated image.

That is, as shown in FIG. 2, it can be said that a virtual camera 120 that captures an interpolated image at each viewpoint is installed between the L camera 102 and the R camera 104. The number of virtual cameras 120 is instructed by the viewpoint position indicating unit 122. The number of virtual cameras 120 is 1 or more, and the practical maximum number depends on the image width of the overlapped portion to be generated. In this case, a different virtual viewpoint is set for each pixel width of the generated image. The number of virtual cameras 120 may be set according to the subject to be photographed by the L camera 102 and the R camera 104, accuracy conditions (for example, limitation of the deviation width of the double image), and the like.

The viewpoint position indicating unit 122 is connected to the viewpoint interpolation image generation unit 118, and determines the number and position of the virtual cameras 120 based on the images taken by the L camera 102 and the R camera 104. The viewpoint position is indicated by a value t in which the internal division ratio between the position of the L camera 102 and the position of the R camera 104 is t: 1-t, and indicates t corresponding to each pixel (x, y) of the overlap portion. The viewpoint position information is carried out as a map t (x, y) to the parallax detection unit 128 and the image mixing unit 136 (both see FIG. 3).

This t is a numerical value (decimal number) from 0 to 1, for example, t = 0 is the position of the L camera 102 and t = 1 is the position of the R camera 104. Therefore, the image connecting portion 124 takes a value close to 0 on the side connected to the area image taken by the L camera 102 alone, and conversely, 1 on the side connected to the area image taken by the R camera 104 alone. It takes a value close to, and changes continuously during that time. For example, at the intermediate position, t = 0.5.

The viewpoint interpolation image generation unit 118 sends the interpolated images (see the virtual camera optical axis (viewpoint) group in FIG. 2) generated according to the set number of virtual cameras 120 to the image connection unit 124.

Images taken by the L camera 102 and the R camera 104 are also input to the image connecting unit 124. The image connecting unit 124 connects the captured image of the L camera 102, the captured image of the R camera 104, and a plurality of interpolated images.

The composite image generated by connecting with the image connecting unit 124 is output to the output unit 126 and displayed on the display screen 110 of the monitor unit 106 as described above.

(Details of viewpoint interpolation image generation unit 118)

FIG. 3 is a schematic configuration diagram showing details of arithmetic processing related to generation of an interpolated image according to a viewpoint instruction from a viewpoint position indicating unit 122 in the viewpoint interpolation image generation unit 118 of FIG. 2.

As shown in FIG. 3, the captured images of the L camera 102 and the R camera 104 are input to the parallax detection unit 128, respectively. The parallax detection unit 128 is input with the viewpoint instruction information from the viewpoint position instruction unit 122.

The parallax detection unit 128 calculates the parallax map d of the captured image of the L camera 102 and the R camera 104 based on the following equations (1) and (2). Note that x and y are the coordinates of the two-dimensional image. The parallax map d (x, y) with respect to the L camera 102 is obtained as follows.

This parallax map d (x, y) is a map having a value that matches the R image when each pixel L (x, y) of the L image is moved laterally by −d (x, y) pixel width. In reality, they do not match exactly mathematically, but are calculated by a technique known as stereomatching. There are various methods such as block matching and semi-global block matching in stereo matching, but the present invention does not limit the stereo matching method.

Similarly, the parallax map d (x, y) with respect to the R camera 104 is obtained as follows.

Equation (1) is the parallax based on the L camera 102, and equation (2) is the parallax based on the R camera 104. Both have almost the same calculation results, but are photographed only by the L camera 102. It is preferable to use the equations (1) and (2) together due to the existence of the region, the region captured only by the R camera 104, and the like.

In the calculation result of the parallax detection unit 128, the calculation result of the formula (1) is sent to the parallax adjustment unit 130 for the L camera 102, and the calculation result of the formula (2) is the parallax adjustment unit 132 for the R camera 104. Is sent to.

The viewpoint position information from the viewpoint position indicating unit 122 is input to the parallax adjusting units 130 and 132, respectively.

(Parallax adjustment of L camera 102)

The parallax adjusting unit 130 for the L camera 102 executes the calculation of the deformation amount dL for parallax adjustment by the equation (3) based on the calculation result of the equation (1) and the viewpoint position information.

The calculation result in the parallax adjustment unit 130 is sent to the image deformation unit 134 for the L camera 102. Image information taken by the L camera 102 is input to the image deformation unit 134.

The image deformation unit 134 performs a transformation process for correcting the parallax of the image taken by the L camera 102 based on the calculation result of the equation (3) in the parallax adjustment unit 130 and the image information of the L camera 102. 4) Execute according to the formula.

Here, the correction coefficient A is incorporated in the equation (4). The correction coefficient A is a correction coefficient for focus adjustment. The correction coefficient A is integrated into "dL" on the right side of the equation (4), and A = 1 means that there is no correction.

The left side of the equation (4) is the position (coordinates) of the pixels constituting the image, and in the first embodiment, the correction coefficient A is set for each coordinate (pixel).

The actual correction target by the correction coefficient A is an image in a region other than the object to be watched (attention image) specified under predetermined conditions, and the viewpoint position shifts by multiplying the correction coefficient A. The so-called out-of-focus image is obtained.

In other words, of the image of the virtual screen 114, that is, the image finally displayed on the monitor unit 106, only the image of interest is in focus, and the other areas are out of focus (FIG. 12 (A). )reference).

(Functional block diagram for setting the correction coefficient A)

The parallax detection unit 128 is connected to the distance information calculation unit 200L, and the distance information calculation unit 200L obtains the distance information of the subject displayed on the virtual screen 114 based on the parallax information detected by the parallax detection unit 128. calculate. The calculated distance information of the subject is sent to the focus adjustment distance determination unit 202L.

The focus image information extracted by the attention image extraction unit 204L is input to the focus adjustment distance determination unit 202L, and the focus adjustment distance determination unit 202L determines the focal length of the attention image and the focal length of a region other than the attention image. It is determined and sent to the correction coefficient determination unit 206L. The correction coefficient determination unit 206L sets the correction coefficient A to be corrected by multiplying the dL of the equation (4), and sends the correction coefficient A to the image deformation unit 134. At this time, A = 1 for the pixel having the coordinates corresponding to the image of interest, and A ≠ 1 for the image having the coordinates corresponding to the region other than the image of interest.

On the other hand, the image information of the L camera 102 after the transformation process executed by the image transformation unit 134 is sent to the image mixing unit 136.

(Parallax adjustment of R camera 104)

The parallax adjustment unit 132 for the R camera 104 executes the calculation of the deformation amount dR for parallax adjustment by the equation (5) based on the calculation result of the equation (1) and the viewpoint position information.

The calculation result in the parallax adjustment unit 132 is sent to the image deformation unit 138 for the R camera 104. Image information taken by the R camera 104 is input to the image deformation unit 138.

The image transformation unit 138 performs transformation processing for correcting the parallax of the image taken by the R camera 104 based on the calculation result of the equation (5) in the parallax adjustment unit 132 and the image information of the R camera 104. 6) Execute according to the formula.

Here, the correction coefficient A is incorporated in the equation (6). The correction coefficient A is a correction coefficient for focus adjustment. The correction coefficient A is integrated into "dR" on the right side of the equation (6), and A = 1 means that there is no correction.

The left side of the equation (6) is the position (coordinates) of the pixels constituting the image, and in the first embodiment, the correction coefficient A is set for each coordinate (pixel).

The actual correction target by the correction coefficient A is an image in a region other than the object to be watched (attention image) specified under predetermined conditions, and the viewpoint position shifts by multiplying the correction coefficient A. The so-called out-of-focus image is obtained.

In other words, of the image of the virtual screen 114, that is, the image finally displayed on the monitor unit 106, only the image of interest is in focus, and the other areas are out of focus (FIG. 12 (A). )reference).

(Functional block diagram for setting the correction coefficient A)

The parallax detection unit 128 is connected to the distance information calculation unit 200R, and the distance information calculation unit 200R obtains the distance information of the subject displayed on the virtual screen 114 based on the parallax information detected by the parallax detection unit 128. calculate. The calculated distance information of the subject is sent to the focus adjustment distance determination unit 202R.

The focus image information extracted by the attention image extraction unit 204R is input to the focus adjustment distance determination unit 202R, and the focus adjustment distance determination unit 202R determines the focal length of the attention image and the focal length of a region other than the attention image. It is determined and sent to the correction coefficient determination unit 206R. The correction coefficient determination unit 206 sets the correction coefficient A to be corrected by multiplying the dR of the equation (6), and sends the correction coefficient A to the image deformation unit 138. At this time, A = 1 for the pixel having the coordinates corresponding to the image of interest, and A ≠ 1 for the image having the coordinates corresponding to the region other than the image of interest.

The image information of the R camera 104 after the transformation process executed by the image transformation unit 138 is sent to the image mixing unit 136.

The viewpoint instruction information from the viewpoint position instruction unit 122 is input to the image mixing unit 136.

In the image mixing unit 136, an image for each viewpoint (an image of each viewpoint of the virtual camera) is superimposed based on the equation (7).

This synthesis is known as the α-blending process. Adoption ratio (1-t): By setting t to the opposite of the internal division ratio between the position of the L camera 102 and the position of the R camera 104, the step gradually changes and the step due to the difference in image quality between the cameras is eliminated. Further, in the portion where the images are connected, since the camera image which is connected almost 100% is adopted, the step between the connected images is also eliminated.

The interpolated image I (x, y) of the overlapped portion superimposed by the image mixing unit 136 is temporarily stored in the interpolated image storage unit 140, and is stored at a predetermined timing (the image of the L camera 102 and the image of the R camera 104). It is sent to the above-mentioned image connecting unit 124 in a state of being synchronized between the two. It is necessary to perform synchronization between the cameras before the parallax detection unit 128.

Hereinafter, the operation of the first embodiment will be described with reference to the flowchart of FIG.

FIG. 4 is a control flowchart showing a processing flow of the image processing apparatus. The image processing may be controlled by a single processor based on a predetermined control program, or one or a plurality of processes may be processed in parallel by a multiprocessor in the block units of FIGS. 1 and 3. Alternatively, a processing board in which an integrated circuit for a specific application such as an ASIC (Application Specific Integrated Circuit) is combined may be created and processed in parallel.

As shown in FIG. 4, in step 150, the images taken by the L camera 102 and the R camera 104 are captured, then the process proceeds to step 152, the viewpoint position instruction information is captured, and the process proceeds to step 154.

In step 154, the parallax detection process based on the L camera 102 and the parallax detection process based on the R camera 104 are executed.

In the next step 155, the parallax adjustment process based on the parallax based on the L camera 102 and the parallax adjustment process based on the parallax based on the R camera 104 are executed, and the process proceeds to step 156 to execute the attention image enhancement process. , Step 158.

(Subroutine of attention image enhancement processing in step 156)

FIG. 4B is a control flowchart showing the attention image enhancement processing subroutine.

In step 210, the correction coefficient A is set to 1 (A = 1) as an initial setting, and the process proceeds to step 212. In step 212, the distance information of the object is calculated based on the parallax information detected by the parallax detection unit 128, and then the process proceeds to step 214 to determine whether or not there is an object (attention image) to be watched from the object. to decide. If an affirmative decision is made in step 214, the process proceeds to step 216 to determine the focal lengths for regions other than the region of interest image. That is, the focal length of the attention image is set based on the compositing process on the virtual screen 114, but for the region other than the attention image, the focal length to be out of focus on the virtual screen 114 is determined, and the process proceeds to step 218. do. If a negative determination is made in step 214, the process returns to step 156 of the main routine (FIG. 4A) with the correction coefficient A set to 1 (A = 1).

In step 218, the correction coefficient A based on the focal length determined in step 216 is set, and the process returns to step 156 of the main routine (FIG. 4A).

In the setting of the correction coefficient A in step 218, the correction coefficient A of the region of the attention image is set to 1, and the region other than the focus image is set to A ≠ 1, so that the region other than the focus image is out of focus. Become.

In step 158, which shifts after the attention image enhancement process (see FIG. 4B) in step 156, the image transformation process based on the L camera 102 and the image transformation process based on the R camera 104 are executed. The correction coefficient A is incorporated in the equations (4) and (6) used for this image transformation process, and the process of defocusing the coordinates (pixels) in the region other than the image of interest is executed at the same time. ..

In the parallax detection process of step 154, the parallax adjustment process of step 156, and the image deformation process of step 158, each process based on the L camera 102 and the R camera 104 is processed in parallel in order to shorten the processing time. Is preferable.

In the next step 160, the images of the overlapped portion of the L camera 102 reference and the R camera 104 reference, which have been deformed in step 158, are mixed to generate an interpolated image, and the process proceeds to step 162 to move to the interpolated image. Is temporarily stored in the interpolated image storage unit 140 (see FIG. 3).

In the next step 164, the non-overlapping portion of the captured image of the L camera 102 and the image captured by the R camera 104 is captured, and then the interpolated image stored in the interpolated image storage unit 140 (see FIG. 3) in step 166 is read out. Then, the process proceeds to step 168.

In step 168, the captured image of the L camera 102, the captured image of the R camera 104, and the interpolated image are concatenated, and the process proceeds to step 170.

In step 170, the image display process is executed. In the image display processing, the connected image connected by the image connecting unit 124 (see FIG. 1) is output to the monitor unit 106 via the output unit 126, and the connected image is displayed on the display screen 110 by driving the display driver 108. ..

In the next step 172, it is determined whether or not the power of the image processing apparatus 100 has been turned off, and if a negative determination is made, the process returns to step 150 and the above steps are repeated to display the moving image. If an affirmative decision is made in step 172, this routine ends.

FIG. 5 shows the case where the virtual camera is set in the overlap portion 112 in the synthesis process of the L camera 102 and the R camera 104 (see the first embodiment, FIG. 5A), and the two L cameras 102 and R. This is a comparison between the case where the camera 104 is simply synthesized (comparative example, see FIG. 5B). As a comparative example, for example, the image composition process by the α-blending process used in the image mixing unit 136 in the first embodiment can be applied.

In the comparative example of FIG. 5B, the subject 142 of the overlap portion 112 on the back side of the virtual screen 114 is also photographed by the L camera 102 and the R camera 104 and displayed on the virtual screen 114, resulting in parallax. Is generated and becomes a double image.

On the other hand, in the first embodiment of FIG. 5A, the subject 142 is photographed by a part of the virtual cameras 120 set in plurality, so that the double image is eliminated.

It is preferable that the subject is photographed by a single virtual camera 120, but in reality, it is often two virtual cameras 120. However, since the parallax is extremely small, some distortion may occur, but a double image is not obtained.

(Principle of eliminating double image by setting virtual camera 120)

In FIG. 6, the principle that the double image is particularly eliminated when the virtual camera 120 is set between the cameras mounted on the L camera 102 and the R camera 104 will be described.

FIG. 6A shows a case where the captured images of the two mounted cameras (L camera 102 and R camera 104) are combined by, for example, the α-blending process applied in the comparative example of FIG. 5B. , The deviation width of the double image is W0.

FIG. 6B shows a case where one virtual camera 120 is set between the L camera 102 and the R camera 104 in the configuration of FIG. 6A, and the double image is also shown in FIG. 6B. Is occurring.

However, with respect to the configuration in which the virtual camera 120 is not set by setting one virtual camera 120 (see FIG. 6A), the deviation width W1 of the double image in FIG. 6B is shown in FIG. The deviation width W0 of the double image of (A) is 1/2 (W1 = 0.5 × W1).

Subsequently, FIG. 6C shows a case where the virtual camera 120 is further set between the L camera 102, the R camera 104, and the virtual camera 120 in the configuration of FIG. 6B. It means that three virtual cameras 120 are set between the camera 102 and the R camera 104. A double image is also generated in FIG. 6 (C).

However, with respect to the configuration in which the virtual camera 120 is not set (see FIG. 6A) by setting the three virtual cameras 120, the deviation width W2 of the double image in FIG. 6C is FIG. The deviation width W1 of the double image of (B) is 1/4 (W2 = 0.5 × (0.5 × W0) = 0.25 × W0).

That is, as the distance (distance) between the L camera 102 and the R camera 104 becomes shorter (proportional), the deviation width of the double image becomes smaller. Therefore, by setting the required number of virtual cameras 120, the parallax is reduced according to the set number of virtual cameras 120, and the double image can be eliminated.

The virtual camera 120 is effective at the time of setting one (the deviation width of the double image is halved), and can be set depending on the maximum and the width of the overlap image. The number of settings of the virtual camera 120 may be set according to the accuracy conditions of the intended use of the captured image and the like.

In the first embodiment, when the image of the viewpoint of the virtual camera 120 is generated, the image taken by the L camera 102 and the image taken by the R camera 104 are used in combination, and this combination is shown in FIG. 7. As described above, since one of the blind spot of the L camera 102 and the blind spot of the R camera 104 complement each other, the blind spot of the virtual camera 120 can be eliminated.

Further, in the first embodiment, in addition to the compositing process on the virtual screen 114, when the attention image is present, the focus image is emphasized by intentionally blurring the focus of the area other than the attention image. I tried to do it. Therefore, it is possible to make the time until the operator or the like monitoring the image visually recognizes the image of interest shorter than the viewing time when all the images on the virtual screen 114 are in focus (as an example). See FIG. 12B, details below).

The attention image does not have to be a single region, and two or more separated regions may be used as the region of the attention image.

"Second embodiment"

The second embodiment of the present invention will be described below.

8A and 8B show a vehicle 10 according to the present embodiment, FIG. 8A is a plan view of the vehicle 10, and FIG. 8B is a side view of the vehicle 10.

As shown in FIG. 8A, the right camera adapter 12 is attached to the right side surface of the vehicle 10 in the forward direction. The right camera adapter 12 is attached with an RH camera head 14 whose optical axis is directed to the right rear of the vehicle 10. The RH camera head 14 captures the region AR surrounded by the alternate long and short dash line R in FIG. 8 (A).

Further, as shown in FIG. 8A, the left camera adapter 16 projects from the left side surface of the vehicle 10 in the forward direction. The left camera adapter 16 is attached with an LH camera head 18 whose optical axis is directed to the left rear of the vehicle 10. The LH camera head 18 captures the region AL surrounded by the alternate long and short dash line L in FIG. 8 (A).

Further, as shown in FIG. 8A, the RR camera head 20 whose optical axis is directed to the rear of the vehicle 10 (the optical axis is substantially horizontal to the road surface) is located at the center in the vehicle width direction at the rear of the vehicle 10. Is attached. The RR camera head 20 captures the region AC surrounded by the alternate long and short dash line C in FIG. 8 (A).

Here, the region AR and the region AC each partially overlap, and a part of the image to be captured overlaps. Further, the region AL and the region AC each partially overlap, and a part of the image to be captured overlaps.

As shown in FIG. 8B, a reverse-moving camera 22 is attached to the rear of the vehicle 10 in addition to the RR camera head 20. The reverse-moving camera 22 captures an image of the area AB surrounded by the alternate long and short dash line B, and the captured image is displayed on the information display monitor 24 (see FIG. 9) in the vehicle interior when the vehicle 10 moves backward. It is designed to be displayed. The optical axis of the region AB is set so as to include the rear end portion (for example, the rear bumper) of the vehicle 10, and is directed downward from the optical axis of the RR camera head 20.

That is, while the above-mentioned RH camera head 14, LH camera head 18, and RR camera head 20 are used as support for the driver of the vehicle 10 in normal driving (whether forward or backward or stopped), the reverse movement is performed. The dedicated camera 22 is specialized for assisting the driver when the vehicle 10 is moving backward, and has different uses.

FIG. 9 is a schematic view of the interior of the vehicle 10 when the front seats are viewed from the center of the rear seats.

The above-mentioned right camera adapter 12 is located in front of the right side window portion 26 of the vehicle 10, and the RH camera head 14 is attached to the right camera adapter 12.

An RH monitor 32 is attached to the lower part of the right side A pillar 30 located between the right side window portion 26 and the front window portion 28. The RH monitor 32 displays an image of the region AR (see FIG. 9A) on the right rear side of the vehicle captured by the RH camera head 14 in a mirror-reversed state. That is, the RH camera head 14 and the RH monitor 32 have the same role as the optical right side door mirror.

The left camera adapter 16 described above is located in front of the left side window portion 34 of the vehicle 10, and the LH camera head 18 is attached to the left camera adapter 16.

An LH monitor 38 is attached to the lower part of the left side A pillar 36 located between the left side window portion 34 and the front window portion 28. The LH monitor 38 displays an image of the region AL (see FIG. 9A) on the right rear side of the vehicle captured by the LH camera head 18 in a mirror-reversed state. That is, the LH camera head 18 and the LH monitor 38 have the same role as the optical left side door mirror.

Further, an information display monitor 24 is attached to the upper part of the center console unit 40 in the vehicle interior. The information display monitor 24 displays an image captured by the reverse reverse camera 22 when the shift lever 42 of the transmission (not shown) of the vehicle 10 is in the reverse range (R range).

The information display monitor 24 is used as a multi-function monitor, and in addition to displaying the vehicle rear image, for example, in cooperation with a navigation system or an audio system (not shown), a navigation screen (navigation information screen, etc.) can be obtained by switching operations. ) And audio screens (information screens related to music, etc.) are displayed. For example, on the navigation screen, the traveling position of the own vehicle is displayed on the map image together with the map image based on the position information acquired by the GPS function mounted on the vehicle 10. When the information display monitor 24 is a touch panel, it also has a function as an input device.

Here, the rearview mirror portion 44 is attached to the upper center (or ceiling portion) of the front window portion 28 of the vehicle 10 of the second embodiment.

The rearview mirror unit 44 has both a mirror surface function that serves as an optical mirror surface and a monitor function that displays an image (corresponding to the monitor unit 106 (see FIG. 1) shown in the first embodiment).

With the mirror surface function, it is possible to check the situation behind the vehicle within the range of the field of view (movement of the line of sight) of the driver or the like while driving.

On the other hand, in the monitor function, it is possible to display a composite image of the images captured by the RH camera head 14, the LH camera head 18, and the RR camera head 20.

As shown in FIG. 8, the RH camera head 14, the LH camera head 18, and the RR camera head 20 mounted on the vehicle 10 have an overlapping portion (first overlap portion 50A) between the region AL and the region AC. There is an overlapping portion (second overlap portion 50B) between the region AR and the region AC. In other words, there are two overlapping portions where the virtual camera 120 of the present invention can be set.

Replacing this with the configuration of the L camera 102 and the R camera 104 (see FIG. 2) of the first embodiment, the first overlap portion 50A has the LH camera head 18 as the L camera 102 and the RR camera. The head 20 becomes the R camera 104.

On the other hand, in the second overlap portion 50B, the RR camera head 20 is the L camera 102, and the RH camera head 14 is the R camera 104.

In the second embodiment, the virtual camera 120 is set in each of the two overlapping portions (first overlapping portion 50A and second overlapping portion 50B) in FIG. 10 (A).

Also in the second embodiment, the image processing apparatus 100 (see FIG. 3) described in the first embodiment is applied to each of the first overlap portion 50A and the second overlap portion 50B. The parallax is detected, the detected parallax is adjusted, the image of the mounted camera is deformed, the images are superimposed on each overlap portion (first overlap portion 50A and second overlap portion 50B), and the interpolated image is generated. By generating it, as shown in FIG. 11B, it is possible to eliminate (reduce) the generation of the double image on the virtual screen 114 of the subject 142.

As a reference, in Table 1, in the overlap portion 112 of the first embodiment, the cameras (L camera 102, R camera 104) implemented for generating the interpolated image by the virtual camera 120 and the second implementation. Cameras (RH camera head 14, LH camera head 18, RR) mounted for generating an interpolated image by the virtual camera 120 in each of the overlap portions (first overlap portion 50A and second overlap portion 50B) in the form of the above. The contrasting relationship with the camera head 20) is shown.

Here, the image of interest is a part of an image that is displayed only in focus on the virtual screen 114 after performing image composition processing without double image and disappearance.

In the image composition process, the image of the virtual screen 114 is displayed on the rearview mirror unit 44, but in the normal image (see FIG. 12B), all the images (objects) are in focus. Become.

When there is an object (attention image) to be watched in this display image, in the second embodiment, the attention image is emphasized by intentionally blurring the focus of the area other than the attention image. Can be done (see FIG. 12 (A)). As shown in FIG. 12C, only the overlapped portion in which the image of interest is present may be blurred.

In particular, as in the second embodiment, in the rear image of the vehicle, there is an object to be watched for safe driving. For example, the object to be watched includes at least another vehicle close to the own vehicle, another vehicle approaching the own vehicle, and an object having a small appearance (bicycle, pedestrian, etc.), and the surrounding image is a noteworthy image. It can be emphasized by blurring the focus of the image.

In the first embodiment and the second embodiment, as a means for emphasizing the attention image, the focus of the image in the region other than the attention image is intentionally blurred, but the region of the attention image and the attention Differences in contrast, color, etc. from areas other than the image may be provided. For example, only the attention image may be a color image, and the area other than the attention image may be a black-and-white image. Further, a marker may be added to the image of interest.

The image of interest is not limited to an approaching object, and may be extracted according to the type of the object (automobile, agricultural machine, construction machine, bicycle, pedestrian, etc.) or by the size of the object. You may. Further, when there are a plurality of attention images, it is basically preferable to use one attention image for one virtual screen 114, so extraction may be performed based on a predetermined priority.

The attention image does not have to be a single region, and two or more separated regions may be used as the region of the attention image. In particular, since a plurality of risk factors may exist while the vehicle is running, it is effective to set a plurality of pixels of interest and to provide a difference in the degree of visibility between the pixels other than the pixels of interest.

The following is an example of priority elements and rankings when specifying an object to be watched (that is, an image of interest to be extracted).

The elements of the degree of priority include distance, speed (change in relative distance), size of an object, and the like in descending order of priority.

(1st priority) When the element is a distance, a near object has a higher priority than a distant object.

(Second priority) When the element is speed, an object with a high approach speed has a higher priority than an object with a slow approach speed. Furthermore, a negative approach speed (an object moving away) has the lowest priority.

(3rd priority) When the element is the size of an object, a small object has a higher priority than a large object (smaller objects may delay visual recognition).

The above-mentioned priority is a basic order, and it is also possible to deviate from the priority and extract the image of interest after monitoring (tracking) the surrounding conditions or for a certain period of time.

For example, if the image processing device 100 of the first embodiment is mounted on a vehicle, there is another vehicle that follows the vehicle while approaching the rear of the own vehicle, and the motorcycle passes by the other vehicle, the other vehicle However, it is preferable that the speed of the second priority and the combination of the third priority give priority to the first priority and the motorcycle as the attention image.

Further, for example, the image processing device 100 of the first embodiment is mounted on a vehicle, an obstacle or a signboard exists at a relatively close position behind the own vehicle, and another vehicle follows the obstacle or the signboard. If you are driving, the signboard with the closest distance will be given priority, but depending on the combination of the speed of the second priority and the third priority, it will be prioritized over the first priority and pay attention to other vehicles. It is preferable to use an image.

Further, when there are a plurality of attention images, a plurality of attention images may be selected in descending order of priority, or a type of attention image related to the attention image of the first priority may be selected. For example, when a bicycle is selected as the image of interest, all areas that can be recognized as a bicycle in the displayed image may be focused.

It should be noted that all the events may be recorded and the subsequent extraction of the image of interest may be processed by using machine learning. For example, the priority is not fixed, people and bicycles are the first priority during commuting hours, speed is the first priority when the road is wide and there are many lanes, in the industrial zone. It may have a learning function such as setting the size as the first priority in order to pay attention to the dump truck or the like.

In the second embodiment, the cameras to be mounted are the RH camera head 14, the LH camera head 18, and the RR camera head 20, but the number of cameras to be mounted and the installation location are not limited. For example, a pair of RR cameras may be provided by distributing the RR camera heads 20 to the left and right instead of the center of the vehicle 10 (a total of four mounted cameras and three overlapping portions). That is, by widening the area of the overlap portion, it is possible to improve the degree of freedom in the process of providing a difference in the degree of visibility of the image.

Further, in the second embodiment, an image of the rear of the vehicle 10 is taken and displayed on the rearview mirror unit 44 in order to support driving. However, the image processing device of the present invention is a surveillance camera or the like. It is also applicable to.

For example, when a certain area is divided into multiple surveillance cameras for shooting and monitoring, the images of each surveillance camera are displayed in multiples (or are displayed by switching periodically) on the monitor. , The observer had to look at each image in sequence. Therefore, by providing an overlapping portion in the shooting area of a plurality of surveillance cameras and synthesizing them by the image processing apparatus of the present invention, a single wide-field image in which a double image is not generated or the subject is not lost is obtained. Can be generated, and the movement of the observer's viewpoint can be minimized.

"First embodiment"
100 Image processing device 102 L camera 104 R camera 106 Monitor unit 108 Display driver 110 Display screen 112 Overlap unit 114 Virtual screen 116 Blind spot area 118 Viewpoint interpolation image generation unit 120 Virtual camera 122 Viewpoint position indicator unit 124 Image connection unit 126 Output unit 128 Disparity detection unit 130, 132 Disparity adjustment unit 134, 138 Image deformation unit 136 Image mixing unit (interpolated image generation unit)
140 Interpolated image storage unit 142 Subject 144 Distance image generation unit 200L, 200R Distance information calculation unit 202L, 202R Focus adjustment distance determination unit 204L, 204R Attention image extraction unit 206L, 206R Correction coefficient setting unit "Second embodiment"
10 Vehicle 12 Right camera adapter 14 RH camera head 16 Left camera adapter 18 LH camera head 20 RR camera head 22 Reverse camera 24 Information display monitor 26 Right side window part 28 Front window part 30 Right side A pillar 32 RH monitor 34 Left Side window part 36 Left side A pillar 38 LH monitor 40 Center console part 42 Shift lever 44 Room mirror part

Claims (7)

When a plurality of image information captured by a plurality of image pickup devices so as to form an overlap portion in which mutual image pickup areas overlap is acquired, and the acquired plurality of image information are combined and displayed, the overlap is performed. At least one virtual viewpoint is set for the unit, and as an image for the virtual viewpoint, an interpolated image generated by interpolating from images taken by the plurality of imaging devices and an overlapping unit captured by the plurality of imaging devices. By combining with images other than the above, an image with the focal position on a predetermined virtual screen is generated, and at the same time,
From the images taken by the plurality of image pickup devices, the attention image specified under a predetermined condition is extracted, the distance from each of the plurality of image pickup devices to the attention image is calculated, and the attention image is extracted. In this case, a process of providing a difference in the degree of visibility of the image between the attention image region and the region other than the attention image region is executed based on the distance from each of the plurality of image pickup devices to the attention image. An image processing device having a control unit. The image processing apparatus according to claim 1, wherein the distance to the image of interest is calculated using parallax information obtained from a plurality of image information captured by the plurality of image pickup devices. In the compositing process, the process of providing the difference in the degree of visibility is a process of intentionally blurring the focus on a region other than the attention image among the images having a focus on the virtual screen. , The image processing apparatus according to claim 1 or 2. Any one of claims 1 to 3, wherein the difference in the degree of visibility includes at least one of contrast adjustment and marking that follows the movement of the attention image, and is a process of emphasizing the attention image. The image processing apparatus according to item 1. The plurality of image pickup devices are image pickup devices that divide and image the rear of the vehicle, and the image synthesized by the control unit is visually displayed by the driver while driving, and the attention image is the own vehicle. The image processing apparatus according to any one of claims 1 to 4, which is an object to be watched behind. The predetermined condition for extracting the attention image is based on a predetermined priority, and as factors for determining the priority, at least the distance to the object existing behind the own vehicle and the movement of the object. The image processing apparatus according to claim 5, which includes speed. Computer,
It is operated as a control unit of the image processing apparatus according to any one of claims 1 to 6.
Image processing program.

Images