JP6311646B2 - Image processing apparatus, electronic mirror system, and image processing method - Google Patents

Description

  The present invention relates to a technique for displaying an image of a rear side of a vehicle on a display in a vehicle interior.

  Conventionally, for example, Patent Document 1 discloses an image display device that sets a region of interest as a target of user's attention in an image and enlarges and displays this region of interest as a general image processing technique. . This image display device can make an object of interest of the user stand out by enlarging the attention area.

JP 2014-192777 A

  In recent years, instead of door mirrors installed in vehicles, electronic mirror systems have been developed that display an image of the rear side of the vehicle on a display provided in the vehicle interior. While the inventors of the present invention are engaged in the development of such an electronic mirror system, it is possible to present information that is easy for the driver to recognize by applying an image processing technique such as enlargement display as disclosed in Patent Document 1. We examined whether it was possible.

  In such development, the inventors of the present invention have focused on the fact that there is a complicated area in the rear side image that displays a complicated image. Specifically, for example, when a vehicle is moving at a very high speed on a road with no speed limit in Europe, the rear side image shows a road such as a soundproof wall or a guardrail that flows backward at high speed. The upper stationary object can be reflected. Such intense movement of the object to be photographed results in an image that is easy to feel troublesome for a driver who visually recognizes the display.

  The present invention has been made in view of such a problem, and an object of the present invention is to reduce the annoyance of the image on the rear side and to realize information presentation that is easy for a driver or the like to recognize. It is to provide technology related to processing.

  In order to achieve the above-described object, one disclosed invention provides a rear side image (40R, 40L) of a vehicle photographed by an imaging device (51, 52) mounted on the vehicle (A). An image processing unit (31) for displaying on a display (61, 62) provided in the passenger compartment, the rear side captured image (41R, 41L) captured by the imaging device; A complicated area setting unit (32, 232) for setting a complicated area (42) for displaying a complicated image by movement of the object to be photographed in the rear side image, and a vehicle in the rear side image. A region-of-interest setting unit (34, 234) that sets a region of interest (44) that is an object of interest of the driver, and an enlarged image (45) of the region of interest is pasted on the complicated region of the captured images. And an image composition unit (35) for outputting to a display. And that the image processing device.

  In another disclosed invention, the rear side image (40R, 40L) of the vehicle photographed by the imaging device (51, 52) mounted on the vehicle (A) is displayed in the vehicle interior of the vehicle. An image processing method for displaying on a provided display (61, 62), and as a step executed by at least one processor (21, 22), a rear side captured image (41R) imaged by an imaging device , 41L), an image acquisition step (S201), a complicated region setting step (S206) for setting a complicated region (42) in which a complicated image is displayed by the movement of the object to be photographed in the rear side image, and The region of interest setting step (S207) for setting the region of interest (44) of interest of the driver of the vehicle in the rear side image, and the region of interest expanded to the complicated region of the captured image Paste the image (45), an image combining step of outputting to the display unit (S209), the image processing method including the.

  According to these inventions, the enlarged image of the region of interest is pasted in the complicated area that becomes a complicated image due to the movement of the object to be photographed in the rear side image. Therefore, in the rear side image displayed on the display, the image of the complicated area can be covered with the image of the region of interest. In addition, since the video of the region of interest pasted on the complicated region is enlarged and displayed, the object of interest of the driver can be shown in detail. Therefore, it is possible to realize information presentation that is easy for the viewer to recognize while reducing the troublesomeness of the video on the rear side.

  Note that the reference numbers in the parentheses are merely examples of correspondences with specific configurations in the embodiments to be described later in order to facilitate understanding of the present invention, and limit the scope of the present invention. It is not a thing.

It is a figure which shows typically the mounting position of the right-and-left rear side camera and rear side monitor in the own vehicle. It is a figure which shows the layout around the driver's seat in the own vehicle. It is a block diagram which shows the whole structure of an electronic mirror system. It is a figure which shows the functional block constructed | assembled by the control circuit of an image processing apparatus. It is a figure which shows an example of the captured image which imaged the right back side. It is a figure which shows an example of the electronic mirror image displayed on a right back side monitor. It is a figure which shows an example of the captured image which imaged the left rear side. It is a figure which shows an example of the electronic mirror image displayed on a left back side monitor. It is a bird's-eye view for demonstrating the driving | running | working condition in which the electronic mirror image shown in FIG. 10 is displayed. It is a figure which shows an example of the electronic mirror image displayed on a left back side monitor in the driving | running | working condition of FIG. It is a bird's-eye view for demonstrating the driving | running | working condition in which the electronic mirror image shown in FIG. 12 is displayed. It is a figure which shows an example of the electronic mirror image displayed on a left back side monitor in the driving | running | working condition of FIG. It is an overhead view for demonstrating the driving | running | working condition in which the electronic mirror image shown in FIG. 14 is displayed. It is a figure which shows an example of the electronic mirror image displayed on a left back side monitor in the driving | running | working condition of FIG. It is an overhead view for demonstrating the driving | running | working condition in which the electronic mirror image shown in FIG. 16 is displayed. It is a figure which shows an example of the electronic mirror image displayed on a left back side monitor in the driving | running | working condition of FIG. It is a figure which shows an example of the electronic mirror image displayed on a left back side monitor. It is a flowchart which shows the video processing process implemented by the control circuit of an image processing apparatus. It is a figure which shows the functional block of the control circuit in 2nd embodiment. It is a figure which shows the complicated area | region and the region of interest in 2nd embodiment. FIG. 21 is a bird's-eye view comparing and showing ranges shown in each region of interest in FIG. 20. It is a flowchart which shows the image processing process by 2nd embodiment.

  Hereinafter, a plurality of embodiments of the present invention will be described with reference to the drawings. In addition, the overlapping description may be abbreviate | omitted by attaching | subjecting the same code | symbol to the corresponding component in each embodiment. When only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other part of the configuration. Moreover, not only the combination of the configurations explicitly described in the description of each embodiment, but also the configuration of a plurality of embodiments can be partially combined even if they are not explicitly described, as long as there is no problem in the combination. And the combination where the structure described in several embodiment and the modification is not specified shall also be disclosed by the following description.

(First embodiment)
An electronic mirror system 100 to which the present invention is applied is mounted on a host vehicle A as shown in FIGS. 1 and 2. The electronic mirror system 100 displays the rear side images 40L and 40R (see FIGS. 6 and 8) taken by the rear side cameras 51 and 52 on the rear side monitors 61 and 62, respectively. Display. As shown in FIG. 3, the electronic mirror system 100 is connected to the communication bus 99, thereby allowing the operation device 81, the in-vehicle sensor 83, the periphery monitoring device 91, the in-vehicle communication device 93, the navigation device 95, etc. Can be exchanged. The electronic mirror system 100 is configured by the rear side cameras 51 and 52, the rear side monitors 61 and 62, the image processing apparatus 20, and the like.

  The operation device 81 is arranged, for example, on a center console provided between the driver's seat and the passenger seat (see FIG. 2). For example, a setting value related to the electronic mirror system 100 is input to the operation device 81 by a driver or the like. The operation device 81 can output input information based on an input from a driver or the like to the communication bus 99. The operation device 81 may be, for example, a press switch provided on the steering wheel, a press switch provided on the center cluster, a dial, or the like.

  The in-vehicle sensor 83 is various sensors mounted on a vehicle including at least a speed sensor, an angular velocity sensor, and a steering angle sensor, for example. The in-vehicle sensor 83 is connected to the communication bus 99. The in-vehicle sensor 83 can output measurement information related to the traveling state of the host vehicle A such as the traveling speed, yaw rate, and steering angle of the host vehicle A (see FIG. 1) to the communication bus 99.

  The periphery monitoring device 91 has a configuration including at least a plurality of external sensors and a periphery monitoring ECU (Electronic Control Unit). The external sensor can include a peripheral monitoring camera 92, a millimeter wave radar, a quasi-millimeter wave radar, a lidar, a sonar, and the like. The peripheral monitoring camera 92 can include a front camera, a rear camera, left and right cameras, and the like. The perimeter monitoring device 91 is a moving object such as a pedestrian, a non-human animal, a bicycle, a motorcycle, and other vehicles, a fallen object on the road, a traffic signal, a guardrail, a curb, a road sign, a road marking, a lane marking, And detecting static objects such as trees. The peripheral monitoring device 91 is connected to the communication bus 99. The surroundings monitoring device 91 uses other vehicle position information indicating the relative position of the detected object including other vehicles, own lane information indicating whether the traveling lane is the traveling lane or the overtaking lane, and the like as the monitoring information. Can be output.

  The in-vehicle communication device 93 is an in-vehicle device for V2X (vehicle to X) that enables wireless communication between the host vehicle A and the outside. The in-vehicle communication device 93 is connected to the communication bus 99 and an antenna for wireless communication. Specifically, the in-vehicle communication device 93 can perform inter-vehicle communication by wireless communication with the in-vehicle communication device of another vehicle located around the host vehicle A (see FIG. 1). The in-vehicle communication device 93 can perform road-to-vehicle communication by wireless communication with a roadside device installed beside the road. The in-vehicle communication device 93 can receive traffic jam information, traffic regulation information, and the like as VICS (registered trademark) information, for example. The in-vehicle communication device 93 can transmit the information of the host vehicle A output on the communication bus 99 to other vehicles and roadside units. The in-vehicle communication device 93 can output received information received from other vehicles and roadside devices to the communication bus 99.

  The navigation device 95 includes a GNSS (Global Navigation Satellite System) receiver, an inertial sensor such as a gyro sensor, a memory for storing map data, and a navigation ECU. The navigation device 95 measures the position of the host vehicle A (see FIG. 1) by combining signals from a plurality of artificial satellites received by the GNSS receiver and the measurement results of the inertial sensor. The navigation device 95 acquires the shape information and the speed limit information of the road on which the host vehicle A is traveling by reading out map data around the host vehicle A from the memory based on the measured position information. In addition, the navigation device 95 acquires road type information such as whether the road on which the vehicle A is traveling is a highway or a general road. The navigation device 95 can output position information, road shape information, speed limit information, road type information, and the like of the host vehicle A to the communication bus 99.

  The rear side cameras 51 and 52 are configured to include at least an image sensor that converts incident light into an electrical signal, an optical system that causes light to enter the image sensor, and a control unit that controls the image sensor and the optical system. . As the image sensor, a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like can be used. The rear side cameras 51 and 52 are connected to the image processing apparatus 20. The rear side cameras 51 and 52 are disposed in the vicinity of the left and right side mirrors outside the passenger compartment of the host vehicle A shown in FIG. The rear side cameras 51 and 52 may be fixed to the left and right doors of the host vehicle A, or may be attached to each door so that the orientation of the imaging surface of the imaging element can be adjusted.

  The right rear side camera 51 captures a range of the right side and the rear side (hereinafter, right rear side) of the traveling direction at a predetermined frame rate (for example, 30 frames per second) around the host vehicle A. To do. The left rear side camera 52 captures a range of the left side and the rear side (hereinafter, left rear side) in the traveling direction at a predetermined frame rate around the host vehicle A. The rear side cameras 51 and 52 sequentially output image data of a series of captured images 41R and 41L (see FIGS. 5 and 7), which are continuously captured, to the image processing apparatus 20 as video signals.

  The rear side monitors 61 and 62 include a display device such as an organic EL (OLED (Organic Light-Emitting Diode)) display or a liquid crystal display, and a control unit that controls each pixel of the display device. The rear side monitors 61 and 62 are connected to the image processing apparatus 20. The rear side monitors 61 and 62 are arranged at the base of each pillar located on both sides of the windshield 18 in the vehicle interior of the host vehicle A shown in FIG. The rear side monitors 61 and 62 may be fixed to the instrument panel 19 or may be attached to the instrument panel 19 so that the orientation of the display surface can be adjusted.

  The right rear side monitor 61 displays an image of the right rear side of the host vehicle A taken by the right rear side camera 51 (see FIG. 1) based on the video signal output from the image processing device 20. The left rear side monitor 62 displays an image of the left rear side of the host vehicle A taken by the left rear side camera 52 (see FIG. 1) based on the video signal output from the image processing device 20.

  As shown in FIG. 3, the image processing apparatus 20 includes at least a main processor 21, a drawing processor 22, a rewritable nonvolatile memory 23, an input / output interface 24 for inputting / outputting information, and a bus connecting them. A control circuit 20a is provided. The image processing apparatus 20 acquires video signals from the rear side cameras 51 and 52 and outputs them to the rear side monitors 61 and 62. The image processing apparatus 20 can construct a plurality of functional blocks (31 to 38) shown in FIG. 4 by executing the program stored in the memory 23 by the processors 21 and 22.

  Next, details of the left and right rear side images 40R and 40L displayed on the rear side monitors 61 and 62 by the electronic mirror system 100 will be described based on FIGS. 5 to 8 with reference to FIG. .

  A captured image 41R by the right rear side camera 51 shown in FIG. 5 is one frame of the right rear side video 40R before being processed and output from the right rear side camera 51 to the image processing device 20. The captured image 41R is image-processed like an electronic mirror image 49R shown in FIG. 6, and is output from the image processing apparatus 20 to the right rear side as one frame of the right rear side video 40R displayed on the right rear side monitor 61. It is output to the monitor 61.

  A captured image 41L obtained by the left rear side camera 52 shown in FIG. 7 is one frame of the left rear side video 40L before being output from the left rear side camera 52 to the image processing device 20. The captured image 41L is image-processed like an electronic mirror image 49L shown in FIG. 8, and is processed from the image processing apparatus 20 as one frame of the left rear side image 40L displayed on the left rear side monitor 62. The data is output to the monitor 62.

  A complicated area 42, a low information content area 43, and a region of interest 44 can be set in each of the rear side images 40R and 40L before processing shown in FIGS.

  The complicated area 42 is an area in which a complicated image is projected by the movement of the object to be photographed in each of the rear side images 40R and 40L. The object to be photographed is not an important tangible object that is of interest to the driver, but is a standing object that stands continuously alongside a road such as a guardrail, a soundproof wall, and a tree. The landscape of the side including these standing objects flows at high speed toward the rear in the complicated area 42 as the host vehicle A travels. As a result, a drastic change occurs between the frames in each of the complicated areas 42 of the rear side images 40R and 40L. The intense display change in the complicated area 42 attracts the driver and makes the driver feel annoying.

  The low information amount area 43 is an area in which the road surface, the roadside zone, the sky, and the like of the road in the adjacent lane are shown in the rear side images 40R and 40L. In the low information amount area 43, as in the complicated area 42, there is substantially no important object to be captured by the driver, such as a parallel car, a two-wheeled vehicle, a bicycle, and a pedestrian. In each of the rear side images 40R and 40L, each low information amount region 43 is a region with very little change between frames. Note that a road surface on which a plurality of white lines extending in an inclination direction with respect to the traveling direction of the host vehicle A is continuously drawn may correspond to the complicated area 42 instead of the low information amount area 43.

  The region of interest 44 is a region in which important tangible objects of interest of the driver are displayed in the rear side images 40R and 40L. Examples of the object to be photographed in the region of interest 44 include a succeeding vehicle A1 that travels behind the host vehicle A, an emergency vehicle Q1 (see FIG. 9), and a roadway outer line 47a (see FIG. 15) on a narrow mountain slope. Is included. A plurality of regions of interest 44 can be set in each of the rear side images 40R and 40L. When a plurality of regions of interest 44 are set, each region of interest 44 is given a priority.

  As shown in FIGS. 5 to 8, the range included in the region of interest 44 is cut out from each of the captured images 41R and 41L as a cut-out image 44a. The cutout image 44a is enlarged in accordance with the size of the range that is the logical sum of the complicated area 42 and the low information amount area 43 while substantially maintaining the aspect ratio in the vertical and horizontal directions. The enlarged image 45 obtained by enlarging the cut-out image 44a is pasted on these areas so as to hide the complicated area 42 and the low information amount area 43. As a result, electronic mirror images 49R and 49L output to the rear side monitors 61 and 62 are generated.

  A frame image 44b and a leader line image 45a are drawn on the electronic mirror images 49R and 49L. The frame image 44b is displayed in a rectangular frame shape and indicates the outer edge of the region of interest 44 to be cut out of the cut image 44a. The leader line image 45a extends linearly from the corner of the frame image 44b toward the corner of the enlarged image 45. The leader line image 45 a shows the connection between the region of interest 44 and the enlarged image 45.

  In each of the above rear side images 40R and 40L, the subsequent vehicle A1 shown in the region of interest 44 is displayed in an enlarged manner. In the rear side images 40R and 40L, distant objects such as the following vehicle A1 appear very small. Therefore, it is difficult for the driver to determine the presence / absence of the following vehicle A1 and the relative speed, and it is difficult to recognize the following vehicle A1 that is overtaking at a high speed from a distance. Therefore, the enlarged display of the succeeding vehicle A1 can be useful information presentation for the driver.

  Here, in each of the rear side images 40R and 40L, the object to be imaged superimposed on the complicated area 42 and the low information amount area 43 may not be the succeeding vehicle A1. Hereinafter, other display modes of the rear side images 40R and 40L that display an enlarged view of other than the following vehicle A1 will be described with reference to FIGS. In addition, in the bird's-eye views such as FIGS. 9 and 11, a range indicated by a dot schematically indicates a range that can be captured by each camera.

  As shown in FIGS. 9 and 10, when there is an emergency vehicle Q1 (ambulance) as a following vehicle on the right rear side of the host vehicle A, an emergency is shown in the left rear side image 40L by the right rear side camera 51. A range including the automobile Q1 is the region of interest 44. As a result, the emergency vehicle Q1 is enlarged and displayed in the right rear side image 40R (see FIG. 6). On the other hand, the emergency vehicle Q1 is not shown in the left rear side image 40R by the left rear side camera 52. However, in the right rear side image 40R, a notification image 45e for notifying the emergency vehicle Q1 is displayed so as to overlap the complicated area 42 and the like. The notification image 45e includes an emergency vehicle icon 45b that notifies the emergency vehicle Q1, a warning icon 45c that alerts the driver, an arrow icon 45d that indicates the position of the emergency vehicle Q1, and the like. The left rear side image 40L can be notified of the position of the emergency vehicle Q1 without disturbing the driver by a display different from the right rear side image 40R.

  As a modification of the above display, the emergency vehicle Q1 is not limited to an ambulance, but may be a police vehicle, a fire engine, or the like. Further, when the bicycle approaches from the left rear while traveling in the city, the electronic mirror system 100 notifies the complicated area 42 of the right rear side image 40R (see FIG. 6) and the like to notify the existence of the bicycle. An image can be superimposed and displayed. Furthermore, when the following vehicle A1 or the like does not exist, the electronic mirror system 100 can superimpose and display a notification image including a warning icon for alerting lane departure warning and blind spot warning on each complicated area 42 and the like.

  As shown in FIG. 11, for example, when the succeeding vehicle A1 exists behind the host vehicle A in a curve curved in the right direction, the succeeding vehicle A1 is out of the imaging range of the left rear side camera 52. However, when the own vehicle A moves to the left lane so as to pass the preceding vehicle A2, the driver of the own vehicle A may move to the left lane in the same manner as the own vehicle A. Need to pay attention to. Under such circumstances, it is desirable that the warning of the following vehicle A1 is given by the left rear side image 40L shown in FIG.

  Therefore, a part of the surrounding image 46 picked up by the rear camera 92b (see FIG. 11) of the periphery monitoring camera 92 is pasted on the complicated area 42 of the left rear side video 40L. Specifically, the rear camera 92b can capture a range behind the host vehicle A as a range different from the rear side around the host vehicle A. Therefore, the rear camera 92b can capture the subsequent vehicle A1 traveling in the same lane as the host vehicle A even during curve traveling. Therefore, if the region of interest is set in the video imaged by the rear camera 92b, the cutout image 144a including the succeeding vehicle A1 is cut out. This cut-out image 144a is cut out from the surrounding image 46 so as to include a part of the object to be captured (for example, a lane marking 46a) shown in the captured image 41L. As described above, the enlarged image 145 obtained by enlarging the subsequent vehicle A1 is superimposed and displayed on the left rear side video 40L.

  As shown in FIGS. 13 and 14, for example, when there are a plurality of succeeding vehicles A1 behind the host vehicle A in a traffic jam, the left rear side image 40L includes a plurality of following vehicles A1 and a complicated area 42. It is possible to enlarge the display. The enlarged image 145 displayed in the left rear side video 40L is an image obtained by enlarging the cutout image 144a cut out from the surrounding image 46 so as to include a plurality of subsequent vehicles A1. In order to acquire such a cut-out image 144a, the region of interest is set in a range surrounding the plurality of subsequent vehicles A1 in the surrounding image 46 of the rear camera 92b. The enlarged image 145 is a combination of a cut image 144a cut out from the surrounding image 46 of the rear camera 92b and a cut image 44a cut out from the captured image 41L of the left rear side camera 52 (see FIG. 7). May be generated.

  As shown in FIGS. 15 and 16, for example, when the host vehicle A is traveling on a narrow mountain slope, the left rear side image 40 </ b> L can display an enlarged overhead image 47 in which the roadway outer line 47 a is enlarged. is there. The bird's-eye view image 47 is a part of an image obtained by converting the viewpoint of the surrounding image 46 captured by the left side camera 92c of the periphery monitoring camera 92 as if looking down from directly above the host vehicle A. The left side camera 92c is fixed to the left door or the like in a posture with the imaging surface facing downward. The left side camera 92c can capture a wider range of the left side of the host vehicle A than the left rear side camera 52. In the surrounding image 46 converted from the viewpoint, the vicinity of the left front wheel of the host vehicle A including the roadway outer line 47 a is set as the region of interest 44. The overhead image 47 facilitates confirmation of the position of the host vehicle A in a narrow lane. Note that the viewpoint conversion of the surrounding image 46 can be performed by either the periphery monitoring device 91 or the image processing device 20 shown in FIG.

  As shown in FIG. 17, the electronic mirror system 100 can display the right rear side video 40 </ b> R covering the complicated area 42 with a predetermined mask image 48 on the right rear side monitor 61. Of course, it is possible to cover the complicated area 42 of the left rear side image 40L (see FIG. 7) with the mask image 48.

  In order to display the various rear side images 40R and 40L described above on the rear side monitors 61 and 62, details of the functional blocks (31 to 36) constructed in the control circuit 20a of the image processing device 20 will be described. Based on FIG. 4, it demonstrates below, referring FIGS. 3, 5-5.

  The image acquisition unit 31 acquires rear side captured images 41R and 41L captured by the rear side cameras 51 and 52, respectively. In addition, the image acquisition unit 31 can acquire a surrounding image 46 (see FIG. 12 and the like) captured by the periphery monitoring camera 92 of the periphery monitoring device 91. Furthermore, the image acquisition unit 31 can acquire, through the in-vehicle communication device 93, an image captured by a camera attached to a roadside machine, a received image captured by an in-vehicle camera of another vehicle, and the like.

  The complicated area setting unit 32 performs image analysis on a series of captured images 41R and 41L acquired along the time series by the image acquisition unit 31, thereby setting a range that becomes the complicated area 42 in each of the rear side images 40R and 40L. Set. Specifically, the complicated area setting unit 32 calculates an optical flow in the continuous captured images 41R and 41L. The complicated area setting unit 32 defines the complicated area 42 in which the movement of the object to be photographed is fast from the gradient information in the time direction based on the optical flow. The complicated area setting unit 32 integrates measurement information such as traveling speed, monitoring information related to other vehicles traveling around, and the result of object recognition by image analysis, etc. The complicated area 42 can be set in an area that does not exist.

  The low information amount region setting unit 33 sets a range to be the low information amount region 43 in each of the rear side images 40R and 40L. Similar to the complicated area setting unit 32, the low information amount area setting unit 33 integrates a plurality of pieces of information acquired by the information acquisition unit 36, thereby reducing the tangible object that is an interest of the driver to an area where there is no tangible object. The information amount area 43 can be set.

  The region-of-interest setting unit 34 sets the region of interest 44 in a range where there is a tangible object that is an object of interest of the driver in each of the rear side images 40R and 40L. The region-of-interest setting unit 34 uses the own lane information and other vehicle position information included in the monitoring information, the result of object recognition such as the following vehicle A1 by image analysis, the road shape information by the navigation device 95, and the like. Can be set. Further, the region-of-interest setting unit 34 obtains traffic jam information and speed limit information included in the received information, travel speed included in the measurement information, travel locus information calculated from the steering angle and yaw rate included in the measurement information, and the like. 44 can be used for setting. For example, the region-of-interest setting unit 34 can identify an emergency vehicle Q1 (see FIG. 9), a general vehicle, a bicycle, and the like by object recognition using image analysis.

  When the region of interest 44 sets the region of interest 44 so as to include the following vehicle A1, the region of interest 44 can be adjusted. Specifically, the position of the subsequent vehicle A1 in each of the captured images 41R and 41L varies depending on the road shape. Therefore, the region-of-interest setting unit 34 adjusts to the road shape such as the uphill, the downhill, and the left and right curves based on the image analysis result of the captured images 41R and 41L, the road shape information, the travel locus information, and the like. Adjustment is performed to move the position of the region of interest 44 up, down, left and right. The region-of-interest setting unit 34 can cause the region of interest 44 to follow the succeeding vehicle A1.

  The region-of-interest setting unit 34 can set the region-of-interest 44 in the video or the like by the peripheral monitoring camera 92. In addition, the region-of-interest setting unit 34 can set a plurality of regions of interest 44 in each of the rear side images 40R and 40L and the image of the peripheral monitoring camera 92. The region-of-interest setting unit 34 assigns a priority to each region of interest 44 when a plurality of regions of interest 44 are set. The priority is the highest in the region of interest 44 in which the emergency vehicle Q1 (see FIG. 9) appears, and then the region of interest 44 in which the subsequent vehicle A1 with the higher relative speed with the host vehicle A (see FIG. 11) appears, It is set to be a region of interest 44 in which the following vehicle A1 appears. The priority of the roadway outer line 47a (see FIG. 15) is lower than that of the following vehicle A1.

  The image composition unit 35 cuts out the range set as the region of interest 44 as the cut-out images 44a and 144a (see FIG. 12) from the captured images 41R and 41L and the surrounding image 46 (see FIG. 12). When a plurality of regions of interest 44 are set, the image composition unit 35 cuts out the range of the region of interest 44 with the highest priority as the cut images 44a and 144a. The image composition unit 35 enlarges the cut-out images 44a and 144a according to the area of the logical sum of the complicated area 42 and the low information amount area 43, and generates enlarged images 45 and 145 (see FIG. 12). In addition to the enlarged images 45 and 145, the image composition unit 35 pastes the frame image 44b and the leader line image 45a on the complicated area 42 and the low information amount area 43, so that the electronic mirror images 49R and 49L (FIG. (See FIG. 8). The image composition unit 35 gives priority to the complicated area 42 among the complicated area 42 and the low information amount area 43 as an area to which the enlarged images 45 and 145 are pasted.

  The image composition unit 35 (see FIG. 17) pastes the notification image 45e (see FIG. 10) and a predetermined mask image 48 on the complicated area 42 and the low information amount area 43 instead of the enlarged images 45 and 145. Electronic mirror images 49R and 49L (see FIGS. 6 and 8) can be generated. Further, when the total area of the complicated area 42 and the low information amount area 43 is less than a predetermined value, the image composition unit 35 electronically captures the captured images 41R and 41L without attaching the enlarged image 45 and the notification image 45e. The mirror images 49R and 49L are used. The image composition unit 35 outputs the electronic mirror images 49R and 49L to the rear side monitors 61 and 62, respectively.

  The information acquisition unit 36 acquires measurement information from the in-vehicle sensor 83, monitoring information from the periphery monitoring device 91, reception information from the in-vehicle communication device 93, position information from the navigation device 95, road shape information, and the like from the communication bus 99. As described above, the information acquired by the information acquisition unit 36 is used for setting the regions 42 to 44 by the region setting units 32 to 34.

  Details of the image processing process realized by the above functional blocks will be described with reference to FIGS. 3 and 5 to 8 based on FIG. The process shown in the flowchart of FIG. 18 is started by the control circuit 20a when the power source of the electronic mirror system 100 is turned on and the power supply to the image processing apparatus 20 is started. The control circuit 20a repeats the process shown in FIG. 18 until the power of the electronic mirror system 100 is turned off.

  In S101, the captured images 41R and 41L are acquired, and the process proceeds to S102. In S101, when there is an acquirable surrounding image 46, the surrounding image 46 is acquired. In S102, measurement information, monitoring information, and the like are acquired from the communication bus 99 as information required for setting the areas 42 to 44, and the process proceeds to S103. In S103, the complicated area 42 and the low information amount area 43 are set based on the images and information acquired in S101 and S102, and the process proceeds to S104.

  In S104, it is determined whether the area of the logical sum of the complicated area 42 and the low information amount area 43 is equal to or larger than a predetermined value. If it is determined in S104 that the total area of the complicated area 42 and the low information amount area 43 is less than the predetermined value, the process proceeds to S105. In S105, the captured images 41R and 41L are set as the electronic mirror images 49R and 49L, respectively, and the process proceeds to S111.

  If it is determined in S104 that the total area of the complicated area 42 and the low information amount area 43 is equal to or greater than a predetermined value, the process proceeds to S106. In S106, the region of interest 44 is set from the images and information acquired in S101 and S102, and the process proceeds to S107. In S106, a plurality of regions of interest 44 given priority can be set.

  In S107, an image to be pasted in the complicated area 42 and the low information amount area 43 is selected, and the process proceeds to S108. In S107, based on the priority set in S106, the image to be pasted is selected from the enlarged images 45 and 145, the notification image 45e, the overhead image 47, and the like. In S108, it is determined whether there is an image selected in S107. If the region of interest 44 is not set in S106 and the notification image 45e and the overhead image 47 are not acquired, it is determined in S108 that there is no selected image, and the process proceeds to S109. In S109, each of the electronic mirror images 49R and 49L is generated by pasting the mask image 48 on the complicated area 42 or the like, and the process proceeds to S111.

  If it is determined in S108 that there is a selected image, the process proceeds to S110. In S110, the electronic mirror images 49R and 49L are generated by pasting any one of the enlarged images 45 and 145, the notification image 45e, the overhead image 47, and the like selected in S107 on the complicated area 42 and the like. Proceed to S111. In S111, the generated electronic mirror images 49R and 49L are output to the rear side monitors 61 and 62, and the process returns to S101.

  In the electronic mirror system 100 as in the first embodiment described so far, the rear side monitors 61 and 62 are installed in the vehicle interior. Therefore, the rear side images 40L and 40R reflected on the rear side monitors 61 and 62 are visually recognized by the driver at a position closer to the central vision than the reflected image reflected on the optical door mirror. Therefore, the fast movement of the object to be photographed in the rear side images 40L and 40R is easily felt troublesome by the driver. In addition, in the rear side images 40L and 40R photographed at a predetermined frame rate, for example, how the standing objects lined up continuously move in the direction opposite to the original moving direction due to the so-called strobe effect. Can be displayed. Such unnatural images are easily felt annoying by the driver.

  Therefore, in the first embodiment, the enlarged images 45 and 145 of the enlarged region of interest 44 are pasted on the complicated region 42 of the rear side images 40R and 40L. Therefore, the video of the complicated area 42 is obscured by the video of the region of interest 44. In addition, since the video of the region of interest 44 attached to the complicated region 42 is displayed in an enlarged manner, the succeeding vehicle A1 and the like that are important for the driver's interest can be shown in detail. Accordingly, it is possible to realize information presentation that can be easily recognized by a viewer such as a driver while reducing the annoyance of the rear side images 40R and 40L.

  In addition, according to the first embodiment, not only the complicated area 42 but also the low information amount area 43 is an area where the enlarged images 45 and 145 can be pasted. In this way, if a wide area that can be pasted is secured, the enlarged images 45 and 145 are further enlarged and pasted on the rear side images 40R and 40L. Therefore, the rear side images 40R and 40L can present information more easily recognized by the driver.

  Further, according to the first embodiment, the enlarged images 45, 145 and the like are preferentially pasted on the complicated area 42. Therefore, the complicated area 42 can be reliably covered with the enlarged images 45, 145 and the like. According to the above, even if the low information amount region 43 is added to the target region to which the enlarged images 45, 145 and the like are pasted, the effect of reducing the troublesomeness of the rear side images 40R, 40L is surely exhibited. .

  Furthermore, if the rear side images 40R and 40L are image-analyzed as in the first embodiment, the annoying display range in the currently acquired series of captured images 41R and 41L is an ugly and complicated area 42. Can be set. By affixing the enlarged images 45, 145 and the like to the complicated area 42 set in this way, the effect of reducing the troublesomeness of the rear side images 40R, 40L is reliably exhibited.

  In addition, the region of interest 44 of the first embodiment can be set to include the following vehicle A1. Therefore, the rear side images 40R and 40L can easily show the subsequent vehicle A1 approaching the host vehicle A from a distance by an enlarged display. As described above, the rear side images 40R and 40L can surely prompt the driver to pay attention to the following vehicle A1.

  Further, according to the first embodiment, the region of interest 44 including information that is useful and important to the driver is preferentially pasted to each of the rear side images 40R and 40L by setting the priority to the plurality of regions of interest 44. Attached. As described above, it is possible to avoid a situation in which the rear side images 40R and 40L are displayed in a complicated manner due to the arbitration processing for narrowing down the pasted image. In addition, the rear side images 40R and 40L can present only information that is useful and important to the driver in an easy-to-understand manner by reducing the number of images to be pasted.

  Furthermore, in the first embodiment, even when the region of interest 44 is not set, the complicated region 42 is obscured by the mask image 48. Therefore, the effect of reducing the troublesomeness of the rear side images 40R and 40L is surely exhibited. Instead of pasting the mask image 48, for example, a process of blurring the video of the complicated area 42 can be performed. Even by such processing, the effect of reducing the troublesomeness of the video in the complicated area 42 can be exhibited.

  In addition, in the first embodiment, the region of interest 44 can be set also in the surrounding image 46 acquired from the surrounding monitoring camera 92 and the like other than the rear side cameras 51 and 52. Therefore, the electronic mirror system 100 can also present useful and important information generated outside the photographing range of the rear side cameras 51 and 52 to the driver in an easy-to-understand manner.

  In the first embodiment, when the cutout image 144a is cut out from the surrounding image 46, a part of the lane marking 46a shown in the captured images 41R and 41L is also included in the enlarged image 145. Therefore, a viewer such as a driver can grasp a rough position where the enlarged image 145 is cut out. As described above, when the region of interest 44 is set in the surrounding image 46, it is desirable that at least a part of the object to be captured shown in the captured images 41R and 41L is also included in the cutout image 144a.

  In the first embodiment, the main processor 21 and the drawing processor 22 correspond to the “processor” described in the claims, and the enlarged image 45 corresponds to the “image of the enlarged region of interest” described in the claims. Equivalent to. The right rear side camera 51 and the left rear side camera 52 correspond to the “imaging device” recited in the claims, and the right rear side monitor 61 and the left rear side monitor 62 are within the scope of the claims. This corresponds to the “display” described. Further, the rear camera 92b and the left camera 92c correspond to “another imaging device” recited in the claims. S101 corresponds to the “image acquisition step” recited in the claims, and S103 corresponds to the “complex area setting step” recited in the claims. Further, S106 corresponds to the “region of interest setting step” recited in the claims, and S110 corresponds to the “image composition step” recited in the claims.

(Second embodiment)
The second embodiment of the present invention shown in FIGS. 19 to 22 is a modification of the first embodiment. In the second embodiment, processing different from that in the first embodiment is performed by the control circuit 20a. In addition to the image acquisition unit 31 and the information acquisition unit 36 that are substantially the same as those in the first embodiment, the control circuit 20a includes a scene estimation unit 237, an enlargement necessity determination unit 238, a complicated region setting unit 232, and a region of interest setting unit 234. The image composition unit 235 is constructed as a functional block. Hereinafter, the details of each functional block will be described based on FIGS. 19 to 21 with reference to FIG. Note that the following content of the left rear side image 40L is naturally applicable to the right rear side image 40R (see FIG. 3).

  The scene estimation unit 237 estimates a driving scene in which the driver is driving the vehicle A based on the information acquired by the information acquisition unit 36. Specifically, the scene estimation unit 237 determines whether the host vehicle A is traveling on a highway (including Expressway, Autobahn, Autoroute, etc.) based on road type information and travel speed measurement information by the navigation device 95. judge.

  The enlargement necessity determination unit 238 determines whether or not the region of interest 44 needs to be enlarged when the scene estimation unit 237 determines that the host vehicle A is traveling on a highway. The enlargement necessity determination unit 238 determines that the enlarged display is necessary in a situation in which there is a good prospect of the rear of the own vehicle A and there is likely to be a succeeding vehicle A1 that passes by at a higher speed than the own vehicle A. I do.

  Specifically, in order to determine whether or not enlargement display is necessary by the enlargement necessity determination unit 238, the information acquisition unit 36 receives the VICS information from the in-vehicle communication device 93, the speed limit information from the navigation device 95, and the in-vehicle sensor 83. Get travel speed measurement information. The enlargement necessity determination unit 238 estimates whether or not the traveling road is not congested based on the VICS information and the like. The enlargement necessity determination unit 238 compares the traveling speed of the host vehicle A with the minimum speed limit of the road being traveled, and if the travel speed is equal to or higher than the minimum speed limit, the traveling road is not congested. Presume that there is. The enlargement necessity determination unit 238 compares the traveling speed of the host vehicle A with the maximum speed limit of the road being traveled. If the traveling speed is less than the maximum speed limit, the succeeding vehicle that travels at a higher speed than the host vehicle A. Presume that A1 exists. The enlargement necessity determination unit 238 makes an enlargement display necessity determination when it is estimated that the road on which the vehicle is traveling is not congested or congested and there is a subsequent vehicle A1 that travels at a high speed.

  The complicated area setting unit 232 sets the complicated area 42 at a position associated with the driving scene estimated by the scene estimation unit 237. When it is estimated that the scene estimation unit 237 is traveling on the highway, the complicated area setting unit 232 sets a predetermined range in the rear side image 40L as the complicated area 42. When the host vehicle A is traveling on the highway, the complicated area setting unit 232 sets a range in which insignificant objects such as a median strip and a soundproof wall are reflected as the complicated area 42. Specifically, in the captured image 41 </ b> L, a predetermined range on the outer side and the upper side in the width direction of the host vehicle A is the complicated area 42.

  The region-of-interest setting unit 234 sets the region of interest 44 at a position associated with the driving scene estimated by the scene estimation unit 237. The region-of-interest setting unit 234 sets, as the region of interest 44, a range in which the following vehicle A1 can be included in the rear side image 40L when the scene estimation unit 237 estimates that the vehicle is traveling on the highway. The region of interest 44 is a rectangular region having a predetermined number of pixels vertically and horizontally, for example, 60 pixels vertically and 80 pixels horizontally. The aspect ratio of the rectangular area as the region of interest 44 is substantially the same as the aspect ratio of the complicated area 42. The region-of-interest setting unit 234 can move the position of the rectangular region as the region of interest 44 using, for example, the upper left corner of the rectangular region as a reference position. In addition, the region-of-interest setting unit 234 can enlarge or reduce the size of the region of interest 44.

Specifically, as shown in FIG. 21, the region-of-interest setting unit 234 expands the region of interest 44 so as to expand a lane located at a specific distance (hereinafter referred to as an enlarged rear distance) behind the host vehicle A. Set. The region-of-interest setting unit 234 can increase / decrease the enlarged backward distance based on Equation 1 below.
(Expression 1) Enlarged rear distance (m) = {maximum speed limit−travel speed} (m / s) × set value (s)
In the above formula 1, the difference between the maximum speed limit of the currently traveling road and the traveling speed of the own vehicle A is a value obtained by estimating the relative speed when the surrounding vehicle A1 passes the own vehicle A. . The set value is a value input to the operation device 81 by a user such as a driver, and is a value of time (seconds) acquired by the information acquisition unit 36 as input information. The driver can change the set value according to the preference of confirming the succeeding vehicle A1 before the passing by the succeeding vehicle A1.

  The initial enlarged rear distance is set so that, for example, the succeeding vehicle A1 that overtakes after 2 seconds is mainly included in the region of interest 44 (44i). When the traveling speed of the host vehicle A deviates from the speed limit to the lower speed side, the region-of-interest setting unit 234 extends the enlarged rear distance so that the subsequent vehicle A1 further away from the host vehicle A is included. Thereby, for example, the range that becomes the region of interest 44 (44s) is adjusted so that the following vehicle A1 that overtakes after 5 seconds is included.

  Similarly to the first embodiment, the region-of-interest setting unit 234 moves the position of the region of interest 44 up and down according to the road shape such as uphill, downhill, and left and right curves based on the road shape information and the travel locus information. Adjust to move left and right. Thereby, for example, even during curve driving, the region-of-interest setting unit 234 can adjust the position of the region of interest 44 (44c) so as to include the succeeding vehicle A1 that overtakes after 5 seconds, for example.

  When the enlargement display necessity determination unit 238 determines that enlargement display is necessary, the image composition unit 235 cuts out the range set as the region of interest 44 from the captured image 41L as a cut image 44a. The image composition unit 235 enlarges the cutout image 44a in accordance with the area of the predetermined complicated region 42. The image composition unit 235 creates an enlarged image 45 shown in FIG. 8 by enlarging the cut-out image 44a, and pastes it in the complicated area 42 together with the frame image 44b and the leader line image 45a, so that the electronic mirror image 49L (see FIG. 8). ) Is generated.

  Details of the video processing processing of the second embodiment realized by the above functional blocks will be described with reference to FIG. 19 based on FIG.

  In S201, each captured image 41L is acquired, and the process proceeds to S202. In S202, VICS information, measurement information, speed limit information, and the like are acquired from the communication bus 99 as information required for setting the complicated region 42 and the region of interest 44, and the process proceeds to S203. In S203, it is determined based on the information acquired in S202 whether or not the host vehicle A is traveling on a highway. If it is determined in S203 that the host vehicle A is traveling on a general road instead of an expressway, the process proceeds to S204. In S204, in order to display the rear side video 40L that is not enlarged, each captured image 41L is set as each electronic mirror image 49L (see FIG. 8), and the process proceeds to S210.

  If it is determined in S203 that the vehicle is traveling on a highway, the process proceeds to S205. In S205, it is determined whether enlargement display is necessary. In S205, it is determined that the enlarged display is necessary when the running road is not congested or congested and the own vehicle A is running at a slower speed than the following vehicle A1. If it is determined in S205 that enlarged display is unnecessary, the process proceeds to S204. On the other hand, if it is determined in S205 that enlarged display is necessary, the process proceeds to S206. In S206, the complicated area 42 is set at a predetermined position, and the process proceeds to S207. In S207, the range of the region of interest 44 is calculated and set from the speed limit information, the travel speed measurement information, the enlarged rear distance based on the driver's setting value, etc., and the process proceeds to S208.

  In S208, it is determined whether or not the region of interest 44 can be set by the calculation in S207. For example, when it is determined in S208 that the region of interest 44 is difficult to set, such as when the following vehicle A1 is out of the captured image 41L due to the road shape, the process proceeds to S204. On the other hand, if it is determined in S208 that the region of interest 44 can be set, the process proceeds to S209.

  In S209, the enlarged image 45 is generated by enlarging the cut image 44a obtained by cutting out the region of interest 44 from each captured image 41L while maintaining the aspect ratio according to the size of the complicated region 42. Then, by pasting the generated enlarged image 45 on the complicated area 42, each electronic mirror image 49L (see FIG. 8) is obtained, and the process proceeds to S210. In S210, each electronic mirror image 49L generated in S204 or S209 is output to each rear side monitor 62. As described above, the rear side image 40L is displayed on the rear side monitor 62.

  Even in the second embodiment described so far, the same effects as those in the first embodiment are achieved, so that the troublesomeness of the rear side images 40R and 40L is reduced, and it is easy for a viewer such as a driver to recognize. Information presentation is realized. In addition, the feasibility of the image processing device 20 is improved by specializing in the use of enlargement of a far rear area on a highway as in the second embodiment.

  Further, in the second embodiment, the troubled area 42 is set at a position associated with the driving scene by paying attention to the fact that the range where the troublesome display can be specified by the driving scene. According to such processing, the complicated area 42 can be set with high accuracy without performing advanced image processing. Therefore, it is possible to reduce the load for image processing while ensuring the effect of reducing the troublesomeness of the video on the rear side.

  Furthermore, in the second embodiment, not only the complicated area 42 but also the region of interest 44 is set at a position associated with the driving scene. Thereby, it is possible to set the region of interest 44 with high accuracy while eliminating the need for image processing for extracting the region of interest 44. As a result, it is possible to reduce the load for image processing after realizing information that can be easily recognized by the driver.

  In addition, according to the second embodiment, when the setting value is changed by an input to the operation device 81, the region-of-interest setting unit 234 changes the position and size of the range to be the region of interest 44 to display an enlarged display. The position of the succeeding vehicle A1 is adjusted. In addition, the region-of-interest setting unit 234 can change the range of the region of interest 44 according to the estimated relative speed between the following vehicle A1 and the host vehicle A. According to the position adjustment of the region of interest 44 as described above, the electronic mirror system 100 can present information about the subsequent vehicle A1 at a timing that is easy for the driver to recognize.

  In the second embodiment, S201 corresponds to the “image acquisition step” recited in the claims, and S206 corresponds to the “complex area setting step” recited in the claims. S207 corresponds to the “region of interest setting step” recited in the claims, and S209 corresponds to the “image composition step” recited in the claims.

(Other embodiments)
Although a plurality of embodiments according to the present invention have been described above, the present invention is not construed as being limited to the above embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present invention. can do.

  In the above embodiment, the electronic mirror system cuts out some images not only from the respective captured images of the respective rear side cameras but also from the peripheral images of the peripheral monitoring camera that captures “other video”, and each rear It was possible to paste on the side image. However, the electronic mirror system sets a region of interest in the image of the in-vehicle camera of another vehicle received by the in-vehicle communication device and the image of the camera set in the roadside machine, and one of the images of one frame constituting these images. The part can be enlarged and displayed.

  In the above embodiment, each rear side monitor is arranged in the vicinity of the base of the pillar. However, the arrangement of the rear side monitor can be changed as appropriate. For example, it is possible to arrange rear side monitors on the left and right sides of the steering. Furthermore, the electronic mirror system can be provided in the own vehicle together with a conventional optical door mirror.

  The “driver” in the above embodiment does not have to perform all driving operations. For example, during the automatic driving, the occupant of the own vehicle monitoring the driving operation by the in-vehicle system can be included in the “driver”.

  In the above embodiment, a large amount of information is used for setting the region of interest. However, information that can be used for setting the region of interest is not limited to the information exemplified in the above embodiment. The region of interest can be set using detection information of any sensor mounted on the vehicle and any reception information that can be acquired from outside the vehicle.

  In the above embodiment, the gradation data of each pixel in the complicated area is overwritten by the gradation data of the enlarged image. However, for example, by overlaying the enlarged image on the image of the complicated area, the image of the complicated area on the background side may be slightly seen through the enlarged image. The enlarged image may be an image obtained by upscaling the cut image.

  In the first embodiment, a plurality of succeeding vehicles are displayed in an enlarged manner during a traffic jam. Such a region of interest at the time of traffic jam can be set to include all “vehicles up to priority n” after giving priority to all detected vehicles, for example. Further, in a traffic jam, a warning icon (see FIG. 10) for alerting a subsequent vehicle such as “!” May be displayed instead of the enlarged image showing the plurality of subsequent vehicles A1.

  In the first embodiment, the left and right image processing processes have been described together. However, the image processing apparatus can individually execute the left and right rear side image processing processes. In the second embodiment, the highway is described as an example of the estimated driving scene. However, the estimated driving scene is not limited to the highway. For example, an urban area is estimated as a driving scene, and a bicycle, a pedestrian, and the like can be enlarged and displayed. Alternatively, a mountain slope is estimated as the driving scene, and the roadway outer line 47a (see FIG. 15) can be enlarged and displayed.

  In the first embodiment, both the frame image 44b and the leader line image 45a are displayed. However, these images may be switched between display and non-display by a user operation such as a driver.

  In the above embodiment, the functions provided by the processors 21 and 22 of the image processing apparatus 20 can be provided by hardware and software different from those described above, or a combination thereof. For example, a part or all of the video processing process to which the image processing method according to the present invention is applied can be executed by a combination meter mounted on a vehicle, a processor provided in a navigation device, or the like.

20 image processing device, 31 image acquisition unit, 32, 232 complicated region setting unit, 33 low information amount region setting unit, 34, 234 region of interest setting unit, 35, 235 image composition unit, 36 information acquisition unit, 237 scene estimation unit , 40R Right rear side image (rear side image), 40L Left rear side image (rear side image), 41R, 41L captured image, 42 Complex area, 43 Low information area, 44 Area of interest, 45 Enlarged image (image), 46 Ambient image, 48 Mask image, 51 Right rear side camera (imaging device), 52 Left rear side camera (imaging device), 61 Right rear side monitor (display), 62 Left rear Side monitor (display), 144a Cut image, A own vehicle, A1 following vehicle

Claims (15)

An image (40R, 40L) of the rear side of the vehicle photographed by the imaging device (51, 52) mounted on the vehicle (A) is displayed on the display (61, 62) provided in the vehicle interior of the vehicle. An image processing apparatus to be displayed on
An image acquisition unit (31) for acquiring rear side captured images (41R, 41L) imaged by the imaging device;
A complicated area setting unit (32, 232) for setting a complicated area (42) in which a complicated image is projected by the movement of the object to be photographed in the rear side image;
A region-of-interest setting unit (34, 234) for setting a region of interest (44) of interest of the driver of the vehicle in the rear side image;
An image processing apparatus comprising: an image composition unit (35) that pastes the enlarged image (45) of the region of interest to the complicated region of the captured image and outputs the pasted image (45) to the display device. .   The image composition unit pastes the enlarged image of the region of interest in the low information amount region (43) in which the object of interest of the driver does not substantially exist in addition to the complicated region. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, wherein the image composition unit prioritizes the complicated area among the complicated area and the low information amount area as an area to which the image of the region of interest is pasted. .   The said complicated area setting part (32) sets the said complicated area in the said image | video of the said rear side by image-analyzing a series of the said captured image acquired by the said image acquisition part. The image processing apparatus according to any one of 1 to 3. A scene estimation unit (237) for estimating a driving scene in which the driver is driving the vehicle;
The said complicated area setting part (232) sets the said complicated area to the position linked | related with the said driving scene estimated by the said scene estimation part, The Claim 1 characterized by the above-mentioned. The image processing apparatus described.   The image processing apparatus according to claim 5, wherein the region-of-interest setting unit (234) sets the region of interest at a position associated with the driving scene estimated by the scene estimation unit.   The image processing apparatus according to claim 1, wherein the region-of-interest setting unit sets the region of interest so that a subsequent vehicle (A1) traveling behind the vehicle is included. .   The region-of-interest setting unit sets a range as the region of interest so that the following vehicle separated from the vehicle is included as the traveling speed of the vehicle deviates from the maximum speed limit of the currently traveling road toward the low speed side. The image processing apparatus according to claim 7, wherein adjustment is performed. An information acquisition unit (36) that acquires input information of a setting value that increases or decreases the distance to the following vehicle reflected in the region of interest;
The image processing apparatus according to claim 7, wherein the region-of-interest setting unit adjusts a range to be the region of interest according to the setting value acquired by the information acquisition unit. The region-of-interest setting unit assigns a priority to each region of interest when a plurality of regions of interest are set,
The image processing apparatus according to claim 1, wherein the image composition unit pastes the region of interest having a high priority among the regions of interest to the complicated region. .   The said image synthetic | combination part pastes the mask image (48) which hides the said complicated area to the said complicated area, when the said interested area is not set by the said interested area setting part. The image processing apparatus according to any one of the above. The image acquisition unit acquires a surrounding image (46) from another imaging device (92b, 92c) that captures a range different from the rear side of the periphery of the vehicle,
The image processing apparatus according to claim 1, wherein the region-of-interest setting unit sets the region of interest in another video imaged by the other imaging device. .   The region-of-interest setting unit is included in a cut-out image (144a) in which a part of an object to be imaged in the captured image is cut out from the surrounding image when the region of interest is set in the other video. The image processing apparatus according to claim 12, wherein a range of the region of interest is set. The image processing apparatus according to any one of claims 1 to 13,
The imaging device for outputting the rear side image to the image processing device;
An electronic mirror system comprising: the display that displays the rear side image output from the image processing apparatus. An image (40R, 40L) of the rear side of the vehicle photographed by the imaging device (51, 52) mounted on the vehicle (A) is displayed on the display (61, 62) provided in the vehicle interior of the vehicle. An image processing method to be displayed on the screen,
As steps executed by at least one processor (21, 22),
An image acquisition step (S101, S201) for acquiring rear side captured images (41R, 41L) imaged by the imaging device;
A troublesome area setting step (S103, S206) for setting a troublesome area (42) in which a troublesome picture is displayed by the movement of the object to be photographed in the rear side image;
A region of interest setting step (S106, S207) for setting a region of interest (44) of interest of the driver of the vehicle in the rear side image;
An image composition step (S110, S209) for pasting the enlarged image (45) of the region of interest to the complicated region of the captured image and outputting the image to the display unit. Image processing method.

Images