JPWO2009107199A1 - Image processing apparatus, image processing method, image processing program, and in-vehicle terminal - Google Patents

Abstract

The image processing device (3) includes an image input unit (7) that acquires a captured image of the image pickup device (2) provided in the vehicle, a road branch point or a curve, and a distance between the vehicle and the navigation system (5). Vehicle information input unit (8) acquired from the above, a recording unit (15) in which imaging information and driver information are recorded in advance, and an enlargement target recognition unit for recognizing a predetermined enlargement object included in the input captured image (9) and a synthesizing unit (11) that generates a synthesized image obtained by synthesizing the enlarged image with the captured image when the enlarged object is recognized. The synthesizing unit (11) uses the distance acquired by the vehicle information input unit (8) and the imaging information and driver information recorded in the recording unit (15). A non-blind spot area that is not a blind spot image is calculated and synthesized so that the enlarged image overlaps the non-blind spot area. As a result, the driver can recognize the information on the blind spot area and the enlarged image with a single action.

Description

  The present invention relates to an image processing apparatus that processes, for example, an image captured by a camera or the like installed in a vehicle for display to a user.

  In recent years, a system using a camera installed in a vehicle (for example, a blind corner monitor (BCM)) has been introduced for the purpose of preventing a collision accident at an intersection with a poor view and a T-junction. is there.

  The BCM is a camera installed at the front end of the vehicle, for example, and images a blind spot area that the driver cannot see from the driver's seat and displays it on the in-vehicle monitor. BCM provides visual assistance to the driver.

  However, if the vehicle enters at an angle other than 90 degrees with respect to the road, for example, when entering the intersection where the vehicle does not go straight, the left and right road conditions of the vehicle are not shown on the BCM in-vehicle monitor at the same time. Sometimes. That is, there may be a blind spot that is not displayed on the BCM monitor. In addition, there are cases where a utility pole, a pedestrian, or the like becomes a shield, and the driver cannot grasp the left and right situations even when viewing the video on the BCM monitor. In such a case, the driver may not be aware of the approaching vehicle, and the BCM alone cannot sufficiently confirm the safety.

  On the other hand, as a means of preventing accidents, a reflecting mirror (corner mirror) that displays information that becomes a blind spot of the driver is installed at the intersection.

  A fault detection system using such a reflecting mirror provided on the road side has been proposed (see, for example, Japanese Patent Application Laid-Open No. 2007-69777 (Patent Document 1)). This system irradiates infrared light toward a reflecting member provided on the road side from the vehicle side, and determines whether or not a dangerous object exists from an image generated by the infrared light reflected by the reflecting member. . If a dangerous object is present, the system notifies the driver accordingly.

In addition, a system has been proposed in which a corner mirror is recognized from an image captured by a camera provided in front of the vehicle, enlarged and displayed on a HUD (Head-Up Display) (for example, Japanese Patent Application Laid-Open No. 2007-2007). No. -102691 (see Patent Document 2).
JP 2007-69777 A JP 2007-102691 A

  However, depending on conditions such as the size, shape, orientation, and road width of the corner mirror, there may be a region (dead angle) that cannot be captured by the corner mirror. Therefore, the driver cannot sufficiently confirm the safety only with the corner mirror and the enlarged image thereof.

  Thus, since the situation of the blind spot area changes every moment, the driver needs to constantly check both the corner mirror installed on the road or its image and the BCM image. For example, immediately before a vehicle enters an intersection, the driver performs an action such that the information on the blind spot area captured by the reflector is a reflector installed on the road and the information on the blind spot area captured by the BCM camera is confirmed on a monitor. Do. Such an action is a burden on the driving driver.

  That is, even if the environment for presenting the driver's blind spot area or enlarged image is prepared by BCM or a corner mirror, information on the blind spot and the enlarged image can be obtained by one action of the driver (an act of checking the monitor). If it cannot be confirmed, the usefulness of the information will be halved.

  Therefore, there is a need for a mechanism that allows the driver to recognize both the blind spot area and the enlarged image information with a single action, and to use each information effectively.

  Therefore, an object of the present invention is to provide an image processing device that enables a driver to recognize information on a blind spot area and an enlarged image by a single action.

  An image processing apparatus according to the present invention includes an image input unit that acquires a captured image captured by an imaging device provided in a vehicle, a road branch point or curve in the traveling direction of the vehicle, and a distance between the vehicle and A vehicle information input unit that is acquired based on information from an in-vehicle device provided in the vehicle, a recording unit that records imaging information indicating characteristics of the imaging device, and driver information relating to the field of view of the driver of the vehicle; An enlarged object recognition unit that recognizes a predetermined enlargement object included in the input captured image, and an enlarged image obtained by enlarging the enlargement object when the enlargement object is recognized by the enlargement object recognition unit. A combining unit that generates a combined image combined with the captured image, and the combining unit acquires the distance acquired by the vehicle information input unit, the imaging information recorded in the recording unit, and the driver information. Using, in the captured image, to identify the non-blind spot is not a driver's blind spot image is synthesized such that the enlarged image on the non-blind area overlap.

  In the above configuration, the combining unit uses the information about the distance between the road branching point or curve in the traveling direction of the vehicle and the vehicle, the characteristics of the imaging device, and the driver's field of view, thereby taking a captured image captured by the imaging device. An area that is not an image of the driver's blind spot (that is, a non-blind spot area) can be identified. Therefore, the synthesizing unit can generate a synthesized image in which the enlarged image of the enlargement target is superimposed on the non-dead angle area in the captured image. When the composite image is displayed, the driver can check the image of the blind spot included in the captured image and the enlarged image of the enlargement object at the same time only by looking at the composite image. That is, the driver can confirm both the blind spot captured by the imaging device and the object to be enlarged by a single action (an act of viewing the displayed composite image).

  According to the present invention, the driver can recognize the information on the blind spot area and the enlarged image with a single action.

Functional block diagram showing the overall configuration of the in-vehicle system including the image processing apparatus The figure which shows schematic structure at the time of seeing through the vehicle 10 by which the vehicle-mounted system was mounted. The figure which shows the range in the horizontal direction photographed with the camera of the vehicle The figure which shows the example of the image image | photographed with the camera of a vehicle The figure which shows the example in case a vehicle approachs at an angle other than 90 degree | times with respect to a road The figure which shows the example in case a vehicle approachs the intersection where the obstruction exists on the left side A diagram conceptually showing the area that a driver can check using a road reflector The figure which shows an example of the image input into an image processing apparatus from a camera The figure which shows an example of the synthesized image which an image processing apparatus produces | generates Flowchart showing an operation example of the image processing apparatus Shows when viewed intersection from above, the distance L, l _0, an example of such an intersection position K1 A flowchart showing an example of processing of the composition control unit 14 The figure which shows the example of an image pick-up field and a field of view when seeing the vehicle which is going to approach an intersection from the top The figure which showed the picked-up image image | photographed with the camera in the example of FIG. 11 so that it might correspond to the image | video display area of a monitor. The figure which shows an example in case the synthesized image containing the enlarged image of an enlarged size is displayed The figure which shows the example of an image pick-up field and a field of view when seeing the vehicle which is going to approach an intersection from the top The figure which shows the example at the time of seeing the condition where a vehicle approachs at an angle other than 90 degree | times with respect to a road from the top The figure which expands and shows the part of the camera 2 of a vehicle An example of the situation where the vehicle is heading for a curve seen from above

  In the image processing device according to the embodiment of the present invention, the combining unit is configured to determine an imaging region imaged by the imaging device around a position of the branch point or curve, a distance between the branch point or curve and the vehicle, and Calculate using imaging information, and calculate a visual field area that enters the driver's field of view around the position of the branch point or curve using the distance between the branch point or curve and the vehicle and the driver information, The position of the non-blind area in the captured image can be specified using the positional relationship of the visual field area with respect to the imaging area.

  As in the above configuration, the combining unit calculates the imaging region of the imaging device and the driver's visual field region at a position based on the position of the branch point or curve. These positional relationships correspond to the positional relationship between the captured image and the non-dead area. Therefore, the synthesizing unit can more accurately calculate the non-dead angle area in the captured image using the positional relationship between the imaging area and the visual field area.

  In the image processing apparatus according to the embodiment of the present invention, the composition unit may determine the degree of enlargement of the enlarged image using the distance acquired by the vehicle information input unit.

  This makes it possible to adjust the degree of enlargement of the enlarged image of the road reflector according to the distance between the vehicle and the branch point or curve. As a result, a composite image is generated that makes it easier for the driver to check the blind spot in the road reflector and the captured image.

  In the image processing device according to the embodiment of the present invention, the image input unit further inputs a horizontal steering angle of the imaging device at the time of capturing the captured image, and the combining unit uses the horizontal steering angle. The imaging area can be calculated.

  Here, the horizontal rudder angle of the imaging device is an amount represented by a rotation angle from a predetermined position when the optical axis of the imaging device provided in the vehicle rotates in a horizontal plane with the vertical direction as an axis from the predetermined position. It is. Thereby, the synthesizing unit can calculate an appropriate non-dead angle region according to the rotation amount even for an image captured by rotating the optical axis in the horizontal direction.

  In the image processing apparatus according to the embodiment of the present invention, the vehicle information input unit further acquires a value indicating a curvature of a curve in the traveling direction of the vehicle, and the combining unit uses the curvature to enlarge the enlarged image. It can be set as the aspect which determines the degree of expansion of.

  Accordingly, it is possible to adjust the degree of enlargement of the enlarged image of the road reflector according to the curvature of the curve in the traveling direction of the vehicle. As a result, a composite image including an enlarged image of the road reflector having an appropriate size according to the curvature of the curve is generated.

  In the image processing apparatus according to the embodiment of the present invention, the vehicle information input unit further acquires information indicating an accident occurrence frequency at the branch point or the curve, and the synthesis unit uses the occurrence frequency of the matter. Thus, the degree of enlargement of the enlarged image can be determined.

  Accordingly, it is possible to adjust the degree of enlargement of the enlarged image of the road reflector according to the frequency of occurrence of an accident at a branch point or a curve in the traveling direction of the vehicle. As a result, a composite image including an enlarged image of a road reflector having an appropriate size according to the risk level of a branch point or a curve is generated.

  An image processing apparatus according to an embodiment of the present invention can operate in cooperation with a navigation system provided in the vehicle, and a vehicle information input unit is configured to detect a road in a traveling direction of the vehicle from the navigation system provided in the vehicle. It can be set as the aspect which acquires the distance of a branch point or a curve, and the said vehicle. Thus, a process can be efficiently performed by setting it as the structure which can be cooperated with a navigation system.

  In the image processing device according to the embodiment of the present invention, the vehicle information input unit further inputs, from the car navigation system, information indicating whether or not the enlargement object is present at the branch point or the curve, The enlargement target recognizing unit may recognize the predetermined enlargement object included in the input captured image when the enlargement object is present at the branch point or the curve.

  Thus, the recognition process is performed more accurately and efficiently by determining whether to execute recognition based on the information indicating the presence or absence of the enlargement target object at the branch point or the curve.

  In the image processing apparatus according to the embodiment of the present invention, the enlargement target recognition unit recognizes a predetermined enlargement object included in the input captured image when the distance is equal to or less than a predetermined distance. It can be.

  An image processing method according to the present invention is an image processing method executed by a computer, and includes an image input step of inputting a captured image captured by an imaging device provided in a vehicle, and a navigation system provided in the vehicle. A vehicle information input step for inputting a distance between the vehicle from a road junction or curve in the traveling direction of the vehicle based on information from an in-vehicle device provided in the vehicle, and the computer accessible A recording information acquisition step of reading and inputting imaging information indicating characteristics of the imaging device and driver information relating to the field of view of the driver of the vehicle recorded in the recording unit; and a predetermined information included in the input captured image An enlargement object recognition step for recognizing an enlargement object, and when the enlargement object is recognized in the enlargement object recognition step, A combined step of generating a combined image obtained by combining an enlarged image obtained by enlarging a large object with the captured image, and in the combining step, the distance acquired in the vehicle information input step, and the recording information acquiring step Using the acquired imaging information and the driver information, a process for identifying a non-blind area that is not a driver's blind spot image in the captured image and combining the enlarged image so as to overlap the non-blind spot area Is executed.

  An image processing program according to the present invention includes an image input process for obtaining a captured image captured by an imaging device provided in a vehicle, a road branch point or curve in the traveling direction of the vehicle, and a distance between the vehicle and Vehicle information input processing acquired based on information from an in-vehicle device provided in the vehicle, imaging information indicating characteristics of the imaging device recorded in a computer-accessible recording unit, and the driver of the vehicle A record information acquisition process for acquiring driver information relating to the field of view, an enlargement object recognition process for recognizing a predetermined enlargement object included in the input captured image, and the enlargement object recognition process have been recognized. An image processing program for causing a computer to execute a synthesis process for generating a synthesized image obtained by synthesizing an enlarged image obtained by enlarging the object to be enlarged with the captured image. Thus, in the synthesis process, using the distance acquired in the vehicle information input process and the imaging information and the driver information acquired in the recorded information acquisition process, A non-blind area that is not a blind spot image is specified, and the computer is caused to perform a process of combining the enlarged image so as to overlap the non-blind spot area.

  An in-vehicle terminal according to the present invention is an in-vehicle terminal capable of cooperating with an imaging device provided in a vehicle, and a map in which road information including a function of specifying the position of the vehicle and a road branch point or a curve position is recorded. A navigation system unit including a data recording unit, an image input unit for acquiring a captured image captured by the imaging device, a road branch point or curve in the traveling direction of the vehicle, and a distance between the vehicle and the navigation A vehicle information input unit that is acquired using vehicle position and road information specified by the system unit, a recording unit that records imaging information indicating characteristics of the imaging device, and driver information related to the field of view of the driver of the vehicle And an enlargement object recognition unit that recognizes a predetermined enlargement object included in the input captured image, and an enlargement object recognized by the enlargement object recognition unit, Serial comprising a synthesizing unit for an enlarged image enlarged object was expanded to produce a composite image by combining with the captured image, and a display device for displaying a composite image in which the combining unit has generated.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Outline of configuration of in-vehicle system>
FIG. 1 is a functional block diagram showing a configuration of the entire in-vehicle system including the image processing apparatus according to the present embodiment. An in-vehicle system 1 shown in FIG. 1 is a system provided in a vehicle, and includes a camera (imaging device) 2, an image processing device 3, a GPS antenna 4, a navigation system (navigation system unit) 5, and a monitor (display unit) 6. . For example, the camera 2 is provided at a position where a front view of the vehicle can be photographed. The image processing device 3 receives and processes the image of the front field of view taken by the camera 2 and outputs it to the monitor 6.

  The GPS antenna 4 receives radio waves from a plurality of GPS artificial satellites (GPS satellites). The navigation system 5 measures the current position of the vehicle based on the radio wave received by the GPS antenna 4. The navigation system 5 generates navigation information using the current location and map data recorded in advance in the map data recording unit 17 and displays the navigation information on the monitor 6. The map data includes road maps (including information on road widths, branch point positions, and curve positions), various facility data, landmarks, and the like. The map data is used for processing such as display of the current location, route search, route guidance, etc. by the navigation system 5, for example.

  The image processing device 3 receives information on intersections or curves in the traveling direction, the current position of the vehicle, and the like from the navigation system 5 and uses them for image processing. The image processing apparatus 3 includes an image input unit 7, a vehicle information input unit 8, an enlargement target recognition unit 9, a synthesis unit 11, an output unit 15, and a recording unit 16. The composition unit 11 includes an enlargement unit 12, a superposition unit 13, and a composition control unit 14. Details of the inside of the image processing apparatus 3 will be described later.

<Vehicle mounting example>
2A and 2B are diagrams illustrating an example of a configuration when the in-vehicle system 1 is mounted on a vehicle. FIG. 2A is a diagram illustrating a schematic configuration when the vehicle 10 on which the in-vehicle system 1 is mounted is seen through. A camera 2 is provided at the front end of the vehicle 10 and is connected to a housing (on-vehicle terminal) 18. The housing 18 is formed by integrating the navigation system 5, the image processing device 3, and the monitor 6. The monitor 6 is formed at a position that can be seen by the driver H1 who has boarded the vehicle 10.

  The housing 18 incorporates a computer including, for example, a CPU, a recording medium (RAM, ROM, HDD, etc.), a display, a power supply circuit, and a bus line connecting these. The functions of the navigation system 5 and the image input unit 7, the vehicle information input unit 8, the enlargement target recognition unit 9, the synthesis unit 11, and the output unit 15 of the image processing device 3 are realized by the CPU executing predetermined programs. The A program for realizing the above functions and a recording medium on which the program is recorded are also an example of an embodiment of the present invention. The recording unit 16 and the map data recording unit 17 are embodied by a recording medium included in the computer.

  The mounting form of the in-vehicle system 1 is not limited to the example shown in FIG. 2A. For example, the navigation system 5 and the monitor 6 may be formed in a single casing, and the image processing apparatus 3 may be provided as the camera 2 and an ECU (Electronic Control Unit) connected to the casing. The image processing apparatus 3 can be configured by a chip such as a 1394 controller LSI, for example. The monitor 6 may be formed by an instrumental panel, an HUD, or the like. In FIG. 2A, a region sandwiched between two straight lines m1 and n1 indicates a range in the vertical direction taken by the camera 2.

  FIG. 2B is a diagram illustrating a range in the horizontal direction that is captured by the camera 2 of the vehicle 10 about to enter the intersection. In FIG. 2B, a region sandwiched between two straight lines p2 and n2 indicates a range in the horizontal direction taken by the camera 2. In the example shown in FIG. 2B, the angle α between the two straight lines p2 and n2 is a value less than 180 °. (A range that is captured by the camera and that is displayed on the monitor 6 like this angle α is referred to as a monitorable angle.) Here, the monitorable angle α of the camera 2 is 180. When the front end of the vehicle 10 enters the intersection, the left and right images of the road perpendicular to the traveling direction of the vehicle 10 are captured by the camera 2 and displayed on the monitor 6. In addition, since the camera is provided at the position of the front end of the vehicle 10, when the vehicle 10 moves forward, the driver H1 can confirm the blind spot ahead immediately.

  FIG. 3 is a diagram illustrating an example of a captured image captured by the camera 2 of the vehicle 10 about to enter the intersection. In the image shown in FIG. 3, the right end portion AR and the left end portion AL show an area that becomes a blind spot from the driver H1.

  In the example shown in FIG. 2B, the left and right sides of the vehicle 10 are photographed by one camera having a horizontal monitoring angle close to 180 °. For example, two cameras having a monitoring angle of approximately 90 ° are used. The vehicle 10 may be installed in the right direction and the left direction, respectively. In addition, if three cameras in the front, right, and left directions are installed at the front end of the vehicle, the monitorable range can be set to 180 ° or more. The number of cameras installed is not particularly limited, and can be determined as appropriate in consideration of the cost and the ease of viewing images.

  Moreover, the installation position of the camera is not limited to the positions shown in FIGS. 2A and 2B. For example, a camera can be provided on a vehicle hood, rearview mirror, door mirror, or the like. Further, a camera may be provided behind the vehicle 10 in order to capture a rear image when the vehicle 10 moves backward. In the embodiment of the present application, a case where the front is photographed by one camera will be described as an example.

<Example of output image of image processing apparatus 3>
When the vehicle 10 enters an intersection or a T-junction, the left and right situations may not be captured by the camera 2. For example, as shown in FIG. 4, when the vehicle 10 enters at an angle other than 90 degrees with respect to the road, and there is a ridge 21 on the right side of the vehicle 10, the right end of the image captured by the camera 2. Shows the ridge 21 instead of the situation in the right direction of the road. That is, the vehicle 22a coming from the right side becomes a shadow of the bag 21 by the camera 2 and does not appear in the photographed image. Further, the vehicle 22b coming from the left side does not appear in the image of the camera 2 until it enters within the monitoring possible angle of the camera 2.

  As another example, as shown in FIG. 5, when the vehicle 10 enters an intersection where a shield 23 (for example, a utility pole or a pedestrian) is present on the left side, the vehicle 22c approaching from the left Since it is hidden by the shadow of the shield 23, it is not displayed in the image taken by the camera 2. In this way, the camera 2 may not be able to capture the situation of the left and right roads.

  On the other hand, the driver H1 can confirm the situation of the road that cannot be directly seen by a road reflector (commonly called a curve mirror) provided at an intersection, a T-junction, or a curve. FIG. 6 is a diagram conceptually illustrating an area that the driver H1 can confirm using the road reflector when the vehicle 10 is about to enter the T-junction. In the example shown in FIG. 6, two road reflecting mirrors 24a and 24b are provided on the T-junction. The range that the driver H1 of the vehicle can directly check and confirm is the range sandwiched between the straight line p3 and the straight line n3. The driver H1 can check the range between the straight line p4 and the straight line n4 by looking at the road reflecting mirror 24a, and can check the range between the straight line p5 and the straight line n5 by looking at the road reflecting mirror 24b. it can.

  In the example shown in FIG. 6, a region S <b> 1 that becomes the left and right blind spots of the driver H <b> 1 is captured by the camera 2 and displayed on the monitor 6. In this case, the driver H <b> 1 can confirm the left and right situations that cannot be seen directly by looking at the road reflector and the monitor 6. For example, in the situation shown in FIG. 6, the image processing apparatus 3 according to the present embodiment has both the blind spot area S1 and the areas reflected in the road reflectors 24a and 24b only when the driver H1 looks at the monitor 6. It is possible to provide an image that makes it possible to confirm.

  FIG. 7A is a diagram illustrating an example of an image input from the camera 2 to the image processing device 3. The image processing device 3 generates a composite image by superimposing an enlarged image obtained by enlarging the portion A1 of the road reflector in the image shown in FIG. 7A on the original image. FIG. 7B is a diagram illustrating an example of a composite image generated by the image processing device 3. As illustrated in FIG. 7B, the image processing device 3 can generate an image in which the enlarged image is superimposed on a portion where a non-blind area that is not the blind area S <b> 1 appears in the original image (FIG. 7A).

  As described above, the enlarged image of the road reflector is superimposed and displayed on the non-blind area of the driver H1 in the original captured image, so that the entire blind area of the driver H1 can be displayed on one screen. A detailed configuration and operation example of the image processing apparatus 3 that enables such image processing will be described below.

<Configuration of Image Processing Device 3>
In the image processing apparatus 3 illustrated in FIG. 1, the image input unit 7 acquires image data of a captured image from the camera 2 and makes the enlargement target recognition unit 9 and the synthesis unit 11 accessible. For example, the image input unit 7 performs A / D conversion and other necessary conversion on the image signal sent from the camera 2 and records the image signal for each frame on a recording medium accessible by the enlargement target recognition unit 9 and the synthesis unit 11. . The image input unit 7 may receive image data that has undergone necessary conversion such as A / D conversion.

  The enlargement target recognition unit 9 reads the image data acquired by the image input unit 7 in units of frames, and whether there is an area that can be recognized as an enlargement target (here, a road reflector) in the image of each frame. Judge whether or not. If there is a region that can be recognized as a road reflector, the enlargement target recognition unit 9 extracts the data of the region and passes it to the enlargement unit 12. The enlargement unit 12 enlarges the image in the area and generates an enlarged image.

  The enlargement target recognition unit 9 can recognize the portion of the road reflector in the image using a known image recognition technique. As an example, the enlargement target recognition unit 9 first extracts an edge portion in an image using a Laplacian filter. Then, the enlargement target recognition unit 9 matches the image data composed of the edge portions with the pre-recorded feature data of the road reflector (for example, a standard road reflector template) to obtain a correlation value. The area where the correlation value is larger than the threshold value can be determined as the area of the road reflector.

  Further, as feature amount data, a template that is easily misrecognized as a road reflector, such as a road sign, may be recorded in advance. In this case, the enlargement target recognition unit 9 may not recognize the object as a road reflector when the correlation value between the template and the image data including the edge portion is larger than the threshold value. Thereby, recognition accuracy improves. The road reflector recognition process is not limited to the above example.

  Thus, the enlargement object recognized by the enlargement object recognition unit 9 is an object that is required to be enlarged and displayed to the driver. That is, an object that provides information useful for driving by a driver, such as a road reflector, is defined in advance as an object to be enlarged. In the above example, the road reflector that is the enlargement target portion is defined by pre-recorded feature value data. The enlargement object is not limited to a road reflector, and for example, a road sign, a guide board, a road sign (characters (eg, “to rare”) or arrows that are desired to be drawn on the road surface) may be defined as an enlargement object. Good.

  The enlargement unit 12 enlarges the image of the area of the road reflector according to a value indicating the degree of enlargement notified from the composition control unit 14. The enlargement unit 12 passes the enlarged image to the superposition unit 13.

  The superimposing unit 13 superimposes the received enlarged image on a corresponding frame (hereinafter referred to as an original frame) of the image data of the captured image acquired by the image input unit 7 to generate a composite image. When superimposing, the superimposing unit 13 acquires information indicating the overlapping position in the original frame of the enlarged image from the synthesis control unit 14, and superimposes the enlarged image on the original frame based on the acquired information.

  In this way, the composite image generated by the superimposing unit 13 is output to the monitor 6 via the output unit 15. It should be noted that the frame whose reflection mirror portion is not recognized by the enlargement target recognition unit 9 is sent to the output unit 15 without being processed by the superposition unit 13 and displayed on the monitor 6.

  In the recording unit 16, imaging information indicating the characteristics of the camera 2 included in the vehicle 10 and driver information regarding the field of view of the driver H <b> 1 of the vehicle 10 are recorded in advance. The imaging information includes information for obtaining a range captured by the camera 2. For example, the monitoring information of the camera 2, the angle of view, the lens characteristics, the installation position of the camera 2 in the vehicle 10, and the like are included in the imaging information.

  Further, the driver information includes information that makes it possible to infer the visual field of the driver who has arrived at the driver's seat of the vehicle 10. For example, the position of the driver's eyes in the vehicle 10, the driver's visual field characteristics (for example, effective visual field), and the like are included in the driver information. The operator information is not limited to a fixed value recorded in advance. For example, the vehicle information input unit 8 may receive the driver's visual field information from an in-vehicle device (not shown) that monitors the driver's eyes and record it in the recording unit 16 as driver information.

  The vehicle information input unit 8 acquires the current position of the vehicle 10 and the position of the nearest branch point in the traveling direction of the vehicle 10 from the navigation system 5, and the distance from the vehicle 10 to the branch point (hereinafter referred to as distance L). ) Is calculated and notified to the composition control unit 14.

  In addition, the acquisition method of the distance L from the vehicle 10 to a branch point is not restricted to this. The vehicle information input unit 8 can acquire the distance L based on data from an in-vehicle device having a function of collecting information for specifying the current position, but the in-vehicle device is not particularly limited. For example, the vehicle information input unit 8 may receive data indicating the distance L from the vehicle 10 to the branch point from the navigation system 5, or the radio wave received by the GPS antenna 4 and the map data of the map data recording unit 17. It may be used to calculate the distance L. Further, the vehicle information input unit 8 may acquire the distance to the curve in the traveling direction in addition to the branch point or instead of the branch point. Further, the vehicle information input unit 8 may specify the current position of the vehicle 10 by further using information from a vehicle speed sensor, a vehicle body direction sensor, or the like (not shown), for example.

  The composition control unit 14 uses the distance L from the vehicle 10 to the branch point and the imaging information and driver information recorded in the recording unit 16 to superimpose the degree of enlargement of the region of the road reflector and the enlarged image in the image. The position is calculated and notified to the enlargement unit 12 and the superposition unit 13, respectively. Here, the composition control unit 14 calculates an area of the image that can be directly viewed by the driver H1 in the original frame of the captured image (= a non-blind area that is not an image of the blind spot of the driver H1). The superposition position is calculated so that the enlarged image is superposed.

  For example, the composition control unit 14 uses the distance L and imaging information to calculate a range (imaging area) that is captured by the camera 2 around the position of the distance L from the vehicle 10. Further, the composition control unit 14 uses the distance L and the driver information to calculate a range (view field region) that enters the driver's field of view around the position of the distance L from the vehicle 10. Then, the composition control unit 14 calculates a non-dead angle region in the original frame of the captured image using the positional relationship between the captured region and the visual field region. The composition control unit 14 can calculate the non-blind area so that the positional relationship of the visual field area with respect to the imaging area corresponds to the positional relationship of the non-blind area with respect to the original frame of the captured image. In addition, the calculation method of a non-dead angle area | region is not restricted to this. The composition control unit 14 can determine the non-dead angle area of the driver H1 using the characteristics of the camera 2, the road shape, the intersection position, the own vehicle position, and the like.

  The composition control unit 14 can also control the operation of the image input unit 7 or the enlargement target recognition unit 9 using information input by the vehicle information input unit 8. For example, the composition control unit 14 may control the image input unit 7 and the enlargement target recognition unit 9 to operate only when the distance L is smaller than a predetermined distance. Thereby, whenever the vehicle 10 by the expansion object recognition part 9 approaches a branch point, the recognition process of a road reflector is performed.

  Further, the recognition process of the road reflector is not necessarily performed every time the intersection is approached. For example, the map data of the navigation system 5 stores in advance information indicating whether or not there is an object to be enlarged (here, a road reflector) at each branch point, and the vehicle information input unit 8 can acquire the information. Can be configured. In this case, the composition control unit 14 can control the enlargement target recognition unit 9 to perform recognition processing when the vehicle 10 approaches a branch point where a road reflector is located. Note that the enlargement target recognition unit 9 may receive the information directly from the vehicle information input unit 8 and determine whether the recognition process needs to be executed.

<Operation Example of Image Processing Device 3>
Next, an operation example of the image processing apparatus 3 will be described. FIG. 8 is a flowchart illustrating an operation example of the image processing apparatus 3. In the operation example shown in FIG. 8, first, the image processing apparatus 3 initializes (sets to 0) a flag “Flag” indicating whether to output a composite image (Op 1).

  Then, the vehicle information input unit 8 obtains the nearest intersection position K1 in the current position of the vehicle 10 and the traveling direction of the vehicle 10 from the navigation system 5 (Op2). The current position information and the intersection position K1 are represented by, for example, latitude and longitude. The vehicle information input unit 8 calculates a distance L from the current position of the vehicle 10 to the intersection position K1 (Op3). The distance L is notified to the composition control unit 14.

The composition control unit 14 determines that the distance L is smaller than the threshold value l ₀ , that is, the vehicle 10 is in the vicinity of the intersection (L <l ₀ : Yes at Op4) and before passing the intersection (L ≧ 0: Yes at Op5). In this case, the image input unit 7 is caused to acquire a captured image of the camera 2 (Op9).

FIG. 9 is a diagram showing an example of the positional relationship between the distances L, l ₀ , the intersection position K1, and the like when the intersection is viewed from above. In the example shown in FIG. 9, the intersection position K1 is the position of the center of the intersection (the intersection of the lines (center lines) passing through the centers of the intersecting roads). The vehicle 10 is traveling toward the intersection position K1. Here, the distance l ₀ is the sum of the distance l ₁ determined by the road width A of the road on which the vehicle 10 is approaching and a predetermined fixed value l ₂ . The distance l ₁ is a distance from the intersection position to the roadside, and is obtained by, for example, l ₁ = A / 2. In this way, the threshold value l ₀ for determining the image acquisition period by the camera 2 can be calculated by the composition control unit 14 adding a value based on the road width A and a fixed value.

  In Op9 of FIG. 8, the image input unit 7 acquires, for example, one frame of the captured image. Hereinafter, an example in which one frame is acquired will be described. However, the image input unit 7 may acquire a plurality of frames in Op9, and the following Op10 process may be executed for each frame.

  If the vehicle 10 is not near the intersection (No at Op4) or after passing the intersection (at Op8), the composition control unit 14 does not acquire the captured image of the camera 2 and sets the flag A determination is made (Op5). When the flag is “Flag = 0” (No in Op5), the processing in Op2 is executed again. When “Flag = 1” (Yes in Op5), the composition control unit 14 instructs the enlargement unit 12 and the superposition unit 13 to end the video superimposition processing (Op6), and sets the flag to “Flag”. = 0 "(Op7). Thereafter, the processing of Op2 is executed again.

By the process of Op4～Op9, when the vehicle 10 reaches the intersection position at a point a predetermined distance l _0, acquires a captured image of the camera 2 by the image input unit 7 is started and the vehicle 10 passes the intersection position, Acquisition of the captured image of the camera 2 is completed.

  When the image input unit 7 acquires a frame (original frame) of the captured image of the camera 2 at Op9, the enlargement target recognition unit 9 extracts an area (road reflector area) that can be recognized as a road reflector in the original frame. (Op10). When the enlargement target recognition unit 9 extracts a road reflector region (Yes in Op11), the composition control unit 14 calculates the degree of enlargement of the road reflector region and the position where the enlarged image is superimposed on the original frame. (Op13). In Op13, the composition control unit 14 calculates the enlargement degree and the superimposed position using the distance L calculated in Op3 and the imaging information and driver information recorded in the recording unit 16. At this time, the superimposed position is calculated so that the enlarged image is superimposed on the area (non-dead angle area) that can be directly seen by the driver H1 in the image of the original frame. Details of the Op13 process will be described later.

  The enlargement unit 12 generates an enlarged image obtained by enlarging the area of the road reflector extracted at Op10 based on the enlargement degree calculated at Op13 (Op14). The superposition unit 13 generates a composite image in which the original frame and the enlarged image are superposed based on the superposition position calculated in Op13 (Op15). And the output part 15 produces | generates the display data processed so that a synthesized image can be displayed on the monitor 6 (Op16), and outputs it to the monitor 6 (Op17). As a result, the monitor 6 displays an image in which the road reflector is enlarged and superimposed on the non-dead angle region in the captured image of the camera 2. When the composite image is displayed on the monitor 6, the flag is set to “Flag = 1” (Op18).

Thereafter, Op2 is executed again. With the processing shown in FIG. 8 described above, the image captured by the camera 2 is displayed on the monitor 6 until the vehicle 10 reaches the point at the distance l ₀ from the intersection position and passes through the intersection position. At this time, when the road reflector is moved to the captured image of the camera 2, the road reflector is enlarged and displayed in a non-dead angle region in the image. The driver H1 can confirm both the condition of the blind spot area moved to the road reflector and the situation of the blind spot area captured by the camera 2 by just looking at the image on the monitor 6.

<Example of calculation of enlargement level and superimposed position>
Here, an example of the calculation of the enlargement degree and the overlapping position in Op13 of FIG. 8 will be described. FIG. 10 is a flowchart illustrating an example of processing of the synthesis control unit 14 in Op13. In FIG. 10, the composition control unit 14 extracts the camera characteristics of the camera 2 as imaging information from the recording unit 16 (Op21). The camera characteristics include, for example, the mounting position of the camera 2, the angle of view, and the lens characteristics.

  Further, the composition control unit 14 acquires the driver's viewpoint position and the driver's effective visual field as the driver information from the recording unit 16 (Op22). Normally, in the vehicle 10, the viewpoint position of the driver can be determined from the position of the driver's seat. Therefore, the driver's seat and position can be recorded in the recording unit 16 in advance as data indicating the driver's viewpoint position.

  The composition control unit 14 uses the camera characteristics and the distance L acquired at Op21 to calculate an imaging area photographed by the camera 2 near the intersection position (Op22). Further, the composition control unit 14 calculates a visual field area that can be directly seen by the driver near the intersection using the driver information and the distance L acquired in Op22 (Op23).

  Below, with reference to FIG. 11, the calculation example of Op22 and Op23 is demonstrated. Here, a case will be described in which the imaging region and the visual field region on the straight line on the horizontal plane perpendicular to the traveling direction of the vehicle including the intersection position K1 are calculated as the imaging region and the visual field region near the intersection position. Note that the imaging region and the visual field region near the intersection are not limited to this example. For example, an imaging region and a visual field region on a straight line or a plane passing through a position advanced by a half of the road width A from the intersection position K1 may be calculated.

  FIG. 11 is a diagram illustrating an example of an imaging region and a visual field region when the vehicle 10 about to enter an intersection is viewed from above. The example shown in FIG. 11 is an example where the camera 2 position of the vehicle 10 is the current position of the vehicle 10 and the intersection position K1 is on a straight line extending from the current position of the vehicle 10 in the traveling direction.

  In FIG. 11, a region sandwiched between the straight line p <b> 6 and the straight line n <b> 6 indicates a range in the horizontal direction taken by the camera 2. An angle “α” between the straight line p6 and the straight line n6 is a monitorable angle of the camera 2. For example, the value of the angle of view of the camera 2 included in the camera characteristics can be used as the value of “α”. The composition control unit 14 can also calculate the value of “α” based on the angle of view of the camera 2 and the lens characteristics. Alternatively, the value of “α” may be recorded in the recording unit 16 as a fixed value in advance.

  A region sandwiched between the straight line p7 and the straight line n7 indicates a range that can be directly seen by the driver H1. An angle “β” between the straight line p7 and the straight line n7 is an effective visual field of the driver H1. The value of “β” may be recorded in the recording unit 16 as a fixed value in advance, for example. It should be noted that the effective visual field of human beings is a visual field that can capture a target only by eye movement and can accept a target target from noise. Since the standard human effective visual field is about 15 degrees to the left and right, this value may be recorded as the value of the angle “β”.

The distance in the traveling direction from the installation position of the camera 2 to the driver H1 is represented by l _m , and the distance in the direction perpendicular to the traveling direction is represented by n. The distance l _m and the distance n can be obtained from the installation position of the camera 2 acquired at Op21 and the driver's viewpoint position acquired at Op22. Also, these distances l _m and the distance n, can also be recorded in advance in the recording unit 16.

  On the straight line q on the horizontal plane and including the intersection position K1 and perpendicular to the traveling direction of the vehicle, the range photographed by the camera 2 is from the intersection of the straight line q and the straight line p6 to the straight line q and the straight line n6. Until the intersection of The range that the driver H1 can directly view is from the intersection of the straight line q and the straight line p7 to the intersection of the straight line q and the straight line n7.

  In Op22, the composition control unit 14 calculates one half (= m1) of the length of the range photographed by the camera 2 on the straight line q as the imaging region. For this calculation, the monitorable angle α and the distance L between the vehicle 10 and the intersection position are used. Specifically, as shown in the following formulas (1) and (2), the value of m1 can be calculated.

tan (α / 2) = m1 / L ―――― (1)
m1 = L × tan (α / 2) ―――― (2)
In Op23, the composition control unit 14 calculates, on the straight line q, one half (= m2) of the length directly visible to the driver H1 as the visual field region. The calculation distance L, the angle β and the distance l _m in the traveling direction from the camera 2 to the driver H1 is used. Specifically, the value of m2 can be calculated as shown in the following formulas (3) and (4).

tan (β / 2) = m2 / (L + l m) ---- (3)
m2 = (L + l _m ) × tan (β / 2) ―――― (4)
The composition control unit 14 calculates the positional relationship between the imaging region and the visual field region using the above m1 and m2. Specifically, in the example illustrated in FIG. 11, the composition control unit 14 calculates the distance X from the intersection of the straight line q and the straight line p6 to the intersection of the straight line q and the straight line p7. The distance X is an example of a value representing the positional relationship between the imaging region and the visual field region, and is a distance from the left end of the imaging region to the left end of the visual field region. The value of X is, for example, the following formula (5) when l ₀ <L, the following formula (6) when l ₁ <L ≦ l ₀ , and the following formula (7) when 0 <L ≦ l _1. Can be calculated. Thus, by changing the calculation method depending on the value of the distance L between the vehicle 10 and the intersection position K1, it is possible to calculate an appropriate superposition position according to the distance L.

If l ₀ <L:
X = 0 ―――― (5)
If l ₁ <L ≦ l ₀ :
X = m1- (m2-n)
= {L × tan (α / 2)} − {(L + l _m ) × tan (β / 2) −n} (6)
If 0 <L ≦ l ₁ :
X = {l ₁ × tan (α / 2)} − {(l ₁ + l _m ) × tan (β / 2) −n} (7)
Using m1, m2, and X calculated in this way, the composition control unit 14 calculates a value indicating a non-dead angle area in the captured image captured by the camera 2 (Op25 in FIG. 10). For example, the composition control unit 14 calculates the position of the left end of the portion (non-blind area) where the field of view of the driver H1 is shown in the captured image.

  Specifically, the composition control unit 14 uses the value (m1) indicating the imaging region, the value (m2) indicating the visual field region, and the distance X from the left end of the imaging region to the left end of the visual field region. The position of the left end of the non-blind area in the image is calculated. An example of this calculation will be described with reference to FIG.

  FIG. 12 is a diagram illustrating a captured image captured by the camera 2 in the example of FIG. 11 so as to correspond to the display area of the monitor. In the example shown in FIG. 12, the width (2 × m1) of the imaging region on the straight line q corresponds to the lateral width W1 of the captured image G1. Here, the horizontal width W1 of the captured image G1 is the horizontal width of the video display area in the monitor 6. The horizontal width W1 is recorded in the recording unit 16 in advance as a fixed value, for example.

As shown in FIG. 12, the positional relationship between the imaging region and the visual field region on the straight line q can be regarded as corresponding to the positional relationship between the entire image and the non-dead angle region in the captured image G1. That is, the relationship between the width W1 of the captured image G1 and the length X _PIX from the left end of the captured image G1 to the left end of the non-dead angle region is considered to correspond to the relationship between the width (2 × m1) of the imaged region and the distance X. be able to. Then, the following formula (8) is established.

(2 × m1): W1 = X: X _PIX ―――― (8)
X _PIX can be calculated by the following equation (9).

X _PIX = (X × W1) / (2 × m1) ―――― (9)
The composition control unit 14 calculates the value of X _{PIX as} a value indicating a non-blind area (Op25 in FIG. 10), and further notifies X _PIX as it is as a value indicating the superimposition position (Op26). . Note that the value indicating the superimposed position is not limited to X _PIX . For example, the composition control unit 14 may obtain the positions of the left end and the right end of the visual field area on the straight line q, and may set the position in the captured image corresponding to the intermediate point as the superimposed position.

  The calculation method of the non-dead angle area is based on the premise that the monitorable angle α of the camera 2 is smaller than 180 °. When the angle of view of the camera 2 is 180 ° or more, for example, the left and right angles are reduced from 180 ° by 1 degree, α is set to 178 °, and the non-blind area can be calculated by the above calculation method.

Note that the larger the value of α, the larger the range that the camera 2 captures, and the image of the road reflector portion with respect to the captured image becomes smaller and the visibility decreases. Therefore, the maximum monitorable angle α _max (= threshold) of the camera 2 that can recognize the road reflector is recorded in the recording unit 16 in advance, and the angle of view of the camera 2 exceeds the maximum monitorable angle α _max. In addition, an image in a region within the maximum monitorable angle α _max can be used as a captured image. Thereby, even if the angle of view of the camera 2 is 180 ° or more, the non-blind area can be calculated using the above calculation method.

Next, in Op27, the composition control unit 14 calculates the enlargement size (size after enlargement) of the enlargement image. Here, as shown in FIGS. 7A and 7B, the horizontal width of the video display area of the monitor 6, that is, the horizontal width of the captured image is W1, the vertical width is H1, the horizontal width of the enlarged image is W2, and the vertical width is H2. The values of W1 and H1 are recorded in advance in the recording unit 16, for example. The value of W2 and H2, for example, the following formula in the case of l ₀ <L (10) and (11), the following equation (12) in the case of l ₁ <L ≦ l ₀ and (13), 0 <L ≦ In the case of l ₁ , it can be calculated by the following formulas (14) and (15). Thus, by changing the calculation method of the enlargement size (W2, H1) according to the value of the distance L between the vehicle 10 and the intersection position K1, it is possible to calculate an appropriate enlargement size according to the distance L. .

If l ₀ <L:
W2 = 0 ―――― （10）
H2 = 0 ―――― (11)
If l ₁ <L ≦ l ₀ :
W2 = W1- (a × L) ―――― (12)
H2 = H1− (b × L) ―――― (13)
If 0 <L ≦ l ₁ :
W2 = W1- (a × l ₁ ) ―――― (14)
H2 = H1- (b × l ₁ ) ―――― (15)
In the above formulas (12) to (15), the coefficients a and b are constants, and appropriate values of the coefficients a and b are determined based on the camera characteristics and the road width B, for example. Specifically, the coefficient a is determined using a table in which a combination of the horizontal view angle (horizontal view angle VX) value and the road width B value of the camera 2 and the coefficient a corresponding to the combination is recorded in advance. can do. Table 1 below is an example of a table in which a coefficient a corresponding to a combination of values of the horizontal angle of view VX and the road width B is recorded.

  Similarly, the coefficient b is determined using a table in which a combination of a vertical field angle (vertical field angle VZ) value and a road width B value of the camera 2 and a coefficient b corresponding to the combination is recorded in advance. be able to. Table 2 below is an example of a table in which the coefficient b corresponding to the combination of the values of the vertical angle of view VZ and the road width B is recorded.

  The method for determining the coefficients a and b is not limited to the above example. For example, in addition to the road width, horizontal angle of view, and vertical angle of view, the vertical size and horizontal size of the image sent by the camera may be added to the conditions for determining the coefficients a and b. Further, the road width B is not necessarily added to the conditions. Further, the enlargement size can be determined using the value indicating the non-dead angle area calculated in Op25. For example, the size of the enlarged image may be a size that fits within the non-dead angle area.

  In this way, the enlargement size (W2, H2) calculated in Op27 is notified to the enlargement unit 12. The enlargement unit 12 enlarges the area of the road reflector in the original frame of the captured image so that the enlargement size (W2, H2) is obtained.

  As described above with reference to FIGS. 10 to 12, the composition control unit 14 calculates the imaging region using the imaging information of the camera 2 and the distance L, and determines the visual field region using the driver information and the distance L. calculate. Then, the composition control unit 14 obtains the positional relationship between the imaging region and the visual field region using a value indicating the relationship between the installation position of the camera 2 and the position of the driver H1. The composition control unit 14 can calculate a value indicating a non-blind area using this positional relationship.

<Example of display image>
FIG. 13 is a diagram illustrating an example in which a composite image including an enlarged image having an enlarged size is displayed on the monitor 6. The example shown in FIG. 13 is an example of the display screen of the monitor 6 at each position of the vehicle 10 when the vehicle 10 enters and passes through the intersection.

When the vehicle 10 has not reached the threshold value l ₀ from the intersection position K1 (when l ₀ <L), the screen D1 is displayed on the monitor 6. When the vehicle 10 enters a distance within l ₀ from the intersection position K1, the screen D2 is displayed. On the screen D2, the road reflector portion is enlarged and displayed superimposed on the original captured image. Thereafter, in a period until the vehicle 10 enters the road at the intersection (period where l ₁ <L ≦ l ₀ ), the size of the enlarged image of the road reflector increases as the vehicle 10 approaches the intersection. Is displayed (screen D3). That is, an enlarged image having a horizontal width W2 and a vertical width H2 calculated by the above equations (12) and (13) is displayed. Then, the period until the vehicle 10 enters the intersection road and reaches the intersection position K1 (period of 0 <L ≦ l ₁ ), the enlarged image has a fixed size (W2, H1 in the above formulas (14) and (15)). Is displayed (screen D4). When the intersection position K1 is passed (0> L), the enlarged image is not displayed as in the screen D5.

As described above, the screen transition shown in FIG. 13 displays the enlarged image of the road reflector superimposed on the original captured image in a size and position that is easy for the driver H1 to confirm in accordance with the situation of the vehicle 10. The In particular, immediately after the vehicle enters the intersection (immediately after L = l ₁ ), the left and right road conditions that the driver H1 cannot directly see are displayed on one screen. For this reason, the driver H1 can confirm the condition of the blind spot reflected on the road reflector and the condition of the blind spot captured by the camera 2 only by looking at the monitor 6.

The screen transition described above is an example and is not limited to this. For example, the image by the camera 2 may not be displayed when L <0 and when l ₀ <L.

  In this way, the driver H1 can check the status of the blind spot area captured by the camera 2 and the road reflector with a single action (the act of checking the status of the blind spot area on the monitor 6). As a result, the driver H1 can reliably recognize a situation in which the burden is reduced and changes every moment.

  In addition, the image processing device 3 can detect the area of the road reflector from the captured image of the camera 2, enlarge the area, and display it on one screen superimposed on the non-dead angle area of the original captured image. become. As a result, it is possible to present to the driver H1 the state of the bicycle, pedestrian, or vehicle that exists in an area that cannot be captured by the camera 2 or the road reflector alone. As a result, we can expect a reduction in encounter accidents, which will account for most traffic accidents.

<Variation of superimposed position calculation>
Here, a modified example of the calculation of the imaging region (m1), the visual field region (m2), and the value (X) indicating the positional relationship described with reference to FIG. 11 will be described. In the example shown in FIG. 11, the optical axis of the camera 2 and the direction seen by the driver H <b> 1 (the direction of the line of sight) are substantially parallel to each other and are also parallel to the traveling direction of the vehicle 10. On the other hand, in this modification, a case where the optical axis of the camera 2 and the direction viewed by the driver H1 are not parallel will be described. FIG. 14 is a diagram illustrating an example of an imaging region and a visual field region in the present modification when the vehicle 10 about to enter the intersection is viewed from above. The variables L, m1, lm, n, α, and β shown in FIG. 14 are the same as the variables shown in FIG. In the example shown in FIG. 14, the direction of the line of sight of the driver H1 has moved to the left by an angle γ from the traveling direction on the horizontal plane. Here, a region sandwiched between the straight line p8 and the straight line n8 is a region where the driver H1 can see the straight line. For example, the value of the angle γ may be recorded in the recording unit 16 in advance, or may be set by the driver H1 via the navigation system 5. Moreover, the information regarding the angle γ may be acquired via the vehicle information input unit 8 from an in-vehicle device (not shown) that monitors the driver's line of sight.

  In the example shown in FIG. 14, on the straight line q, half the length of the range photographed by the camera 2 (= m1) is calculated by the above equation (2) as in the case shown in FIG. can do.

m1 = L × tan (α / 2) ―――― (2)
In this modification, the distance between the intersection F1 between the straight line q and a straight line extending in the traveling direction from the driver H1 and the leftmost point (intersection of the straight line p8 and the straight line q) in the range where the driver H1 can be directly seen is Let m3. The composition control unit 14 calculates the value of m3 as the value of the visual field area of the driver H1. Since the angle between the straight line connecting the point F1 and the driver H1 and the straight line p8 is (γ + β / 2), the following equation (16) is established. Therefore, m3 can be calculated by the following formula (17).

tan (γ + β / 2) = m3 / (L + l m) ---- (16)
m3 = (L + l _m ) × tan (γ + β / 2) ―――― (17)
The composition control unit 14 calculates the positional relationship between the imaging region and the visual field region using the above m1 and m3. Specifically, the composition control unit 14 calculates a distance X2 from the intersection of the straight line q and the straight line p6 to the intersection of the straight line q and the straight line p8. This is the distance from the left end of the imaging area to the left end of the visual field area. The value of X2 can be calculated by, for example, the following formulas (18) to (20).

If l ₀ <L:
X2 = 0 ―――― (18)
If l ₁ <L ≦ l ₀ :
X2 = m1- (m3-n)
= {L × tan (α / 2)} − {(L + l _m ) × tan (γ + β / 2) −n}
―――――― (19)
If 0 <L ≦ l ₁ :
X2 = {l ₁ × tan (α / 2)} − {(l ₁ + l _m ) × tan (γ + β / 2) −n}
―――――― (20)
Using m1, m3, and X2 calculated in this way, the composition control unit 14 calculates a value indicating a non-dead angle region in the captured image captured by the camera 2. For example, the relationship between the width W1 of the captured image and the length X _PIX-2 from the left end of the captured image to the left end of the non-dead angle region is considered to correspond to the relationship between the width (2 × m1) of the imaging region and the distance X2. And (21) below holds. Thereby, X _PIX-1 can be calculated by the following equation (22).

(2 x m1): W1 = X2: X _PIX-2 ―――― (21)
X _PIX-2 = (X2 × W1) / (2 × m1) ―――― (22)
Composition control unit 14, the value of X _PIX-2, is calculated as a value indicating a non-blind area, the value of the X _PIX-2, it is possible to notify the superimposition unit 13 as a value indicating a superimposed position. Accordingly, the output unit 15 can generate a composite image in which the enlarged image is superimposed on the non-dead angle region in the captured image and output the composite image to the monitor 6.

  In addition, in this modification, the case where driver | operator H1's eyes | visual_axis shifted | deviated to the left from the advancing direction was demonstrated. By using the calculation method of this modification, the non-dead angle region can be calculated even when the optical axis of the camera 2 is rotated by a predetermined angle (horizontal rudder angle) in the horizontal plane.

FIGS. 15A and 15B show an example in which the optical axis of the camera 2 rotates by a predetermined horizontal steering angle. FIG. 15A is a diagram illustrating an example of a situation in which the vehicle 10 enters the road at an angle other than 90 degrees from above. At the branch point (T-junction) shown in FIG. 15A, two roads merge at an angle λ. Here, as an example, the intersection of the center lines of each of the two roads that merge is set as a branch point position K2. The road width of the road on which the vehicle is passing is B, and the road width of the road on which the vehicle 10 is to enter is A2. The distance from the vehicle 10 to the branch point position K2 is L, the distance from the branch point position K2 to the roadside is l ₁ , the threshold value (fixed value) for determining the display start point of the enlarged image is l ₂ , and from the branch point position K2 The distance to the display start position of the enlarged image is indicated by l ₀ . The distance l ₁ is calculated by, for example, l ₁ = (A2 / 2) cosλ + (B / 2) tanλ. The road information such as the road widths A2 and B, the angle λ of the two roads to be joined, and the branch point position K2 can be acquired from the navigation system 5 via the vehicle information input unit 8.

  FIG. 15B is an enlarged view showing a part of the camera 2 of the vehicle 10. 15B, the Y axis indicates the traveling direction of the vehicle 10, and the line J indicates the direction of the optical axis of the camera 2 in the horizontal plane. The optical axis J of the camera 2 rotates counterclockwise by an angle ε from the Y axis in the horizontal plane. That is, it can be said that the horizontal steering angle of the camera 2 is ε.

  Here, the steering angle control of the camera 2 can be performed by another ECU (for example, a camera control ECU (not shown)) mounted on the vehicle. As an example, the ECU can control the horizontal rudder angle ε of the camera 2 in accordance with the angle λ between the road on which the vehicle 10 is traveling and the road to be approached. For example, the horizontal rudder angle ε of the camera 2 is controlled so that the optical axis is perpendicular to the road to be entered.

  The information indicating the horizontal steering angle ε of the camera 2 can be acquired by the image input unit 7 as camera characteristics from the ECU, for example. The composition control unit 14 can receive the horizontal steering angle ε of the camera 2 from the image input unit 7 and use it for calculation of the non-dead angle region.

  The composition control unit 14 calculates an imaging region when the camera 2 is rotated by the horizontal rudder angle ε, and calculates a non-dead angle region by obtaining a positional relationship between the calculated imaging region and the visual field region. The composition control unit 14 can calculate the imaging region when the camera 2 is rotated by the horizontal rudder angle ε in the same manner as the calculation method of the visual field region when the line of sight of the driver H1 is rotated by the angle γ described above. it can. Further, the composition control unit 14 can calculate an appropriate size (W2, H2) of the enlarged image corresponding to the distance L using the above formulas (10) to (15).

  In this way, even when the optical axis of the camera 2 is rotated by a predetermined horizontal rudder angle in the horizontal plane, the non-dead angle region can be calculated and the superimposed position and the enlarged size of the enlarged image can be determined.

<Calculation of superposition position and enlargement degree when vehicle 10 heads for a curve>
Another modification of the operation by the composition control unit 14 will be described. Here, a calculation example of the synthesis control unit 14 when the vehicle 10 heads for a curve will be described. FIG. 16 is a diagram of an example of a situation in which the vehicle 10 heads for a curve as seen from above.

FIG. 16 shows a situation immediately before the vehicle 10 reaches the curve of the radius of curvature R. As an example, the curve position K3 is an intersection of a road center line and a line that bisects the rotation angle (θ) of the curve. The width of the road on which the vehicle is passing is A, the distance from the vehicle 10 to the curve position K3 is L, and the distance from the curve position K3 to the point where the curve starts is l ₁ (l ₁ = (R + A / 2) sin (θ / 2)), the threshold (fixed value) for determining the display start point of the enlarged image is indicated by l ₂ , and the distance from the curve position K3 to the display start position of the enlarged image is indicated by l ₀ . The road information such as the road width A, the curve rotation angle θ, the curve position K3, and the curve curvature radius R can be acquired from the navigation system 5 via the vehicle information input unit 8.

  Similar to the calculation of the non-blind area described with reference to FIGS. 11 and 12, the composition control unit 14 can calculate the non-blind area of the captured image by obtaining the positional relationship between the imaged area and the visual field area. it can. Further, the composition control unit 14 can calculate an appropriate size (W2, H2) of the enlarged image corresponding to the distance L using, for example, the following formulas (23) to (28).

If l ₀ <L:
W2 = 0 ―――― (23)
H2 = 0 ―――― (24)
If l ₁ <L ≦ l ₀ :
W2 = W1− (a × L) + (e × L) ―――― (25)
H2 = H1− (b × L) + (f × L) ―――― (26)
If 0 <L ≦ l ₁ :
W2 = W1− (a × l ₁ ) + (e × l ₁ ) ―――― (27)
H2 = H1− (b × l ₁ ) + (f × l ₁ ) (28)
The coefficients a and b in the equations (25) to (28) can be determined by the camera characteristics and the road width A in the same manner as the coefficients a and b in the equations (12) to (15). The coefficients e and f are coefficients determined by the curve R. For example, the values of the coefficients e and f that increase as R decreases are defined in advance and recorded in the recording unit 16. Specifically, the values of the coefficients e and f are determined by a function that defines the values of the coefficients e and f that change depending on R, a table that records the values of the coefficients e and f corresponding to each value of R, and the like. be able to.

  In this way, even when the vehicle 10 approaches the curve and passes through, the non-dead angle area can be calculated and the superimposed position and enlarged size of the enlarged image of the road reflector can be determined. Further, since the size of the enlarged image is adjusted according to the curvature of the curve, for example, a road reflector can be displayed in a large size on a curve having a large curvature, that is, a large degree of bending.

<Calculation to determine the degree of enlargement of the enlarged image according to the accident frequency>
When determining the enlargement size of the enlarged image, the composition control unit 14 can take into account various factors in addition to the road information and the vehicle position information. For example, the composition control unit 14 may determine the enlarged size according to the frequency of accident occurrence at an intersection where the vehicle 10 is about to enter. Here, as shown in FIG. 9, an example of calculation when the vehicle 10 is about to enter an intersection will be described. The composition control unit 14 can obtain a value indicating the frequency of occurrence of an accident at an intersection where the vehicle 10 is about to enter from the navigation system 5 via the vehicle information input unit 8, for example.

  The value indicating the accident occurrence frequency is, for example, the number of accidents near the intersection in the past 10 years. Alternatively, the value indicating the accident occurrence frequency may be a value indicating the accident occurrence frequency for each season or the difference in the accident occurrence frequency depending on the time zone. Such values reflect, for example, the situation where there are few accidents in summer but snowy roads in winter and the incidence of accidents increases, or there are few accidents during the day but there are more accidents at night and early morning. Value.

  The composition control unit 14 can calculate an appropriate size (W2, H2) of the enlarged image corresponding to the distance L using, for example, the following formulas (29) to (34).

If l ₀ <L:
W2 = 0 ―――― (29)
H2 = 0 ―――― (30)
If l ₁ <L ≦ l ₀ :
W2 = W1− (a × L) + (c × L) ―――― (31)
H2 = H1− (b × L) + (d × L) ―――― (32)
If 0 <L ≦ l ₁ :
W2 = W1− (a × l ₁ ) + (c × l ₁ ) ―――― (33)
H2 = H1− (b × l ₁ ) + (d × l ₁ ) (34)
The coefficients a and b in the equations (29) to (34) can be determined by the camera characteristics and the road width A in the same manner as the coefficients a and b in the equations (12) to (15). The coefficients c and d are coefficients determined by the frequency of accident occurrence at the intersection where the vehicle 10 is about to enter. For example, the values of the coefficients c and d that increase as the accident occurrence frequency increases are defined in advance and recorded in the recording unit 16. Specifically, the coefficients c and d are determined by a function that defines the values of the coefficients c and d that change according to the accident occurrence frequency, a table that records the values of the coefficients c and d corresponding to each value of the accident occurrence frequency, and the like. The value can be determined.

  In this way, the enlarged size of the enlarged image of the road reflector can be determined according to the frequency of accidents at the intersection where the vehicle 10 enters. Thereby, for example, at the intersection where the accident occurrence frequency is high, the size of the enlarged image of the road reflector can be increased, and an image that the driver H1 can easily confirm can be displayed on the monitor 6.

Claims (12)

An image input unit for acquiring a captured image captured by an imaging device provided in the vehicle;
A vehicle information input unit that acquires a distance between the vehicle and a road branch point or curve in the traveling direction of the vehicle based on information from an in-vehicle device provided in the vehicle;
A recording unit that records imaging information indicating characteristics of the imaging device and driver information relating to the field of view of the driver of the vehicle;
An enlargement target recognition unit for recognizing a predetermined enlargement object included in the input captured image;
A synthesis unit that generates a synthesized image obtained by synthesizing an enlarged image obtained by enlarging the enlargement object with the captured image when an enlargement object is recognized by the enlargement target recognition unit;
The synthesizing unit is not a driver's blind spot image in the captured image using the distance acquired by the vehicle information input unit and the imaging information and the driver information recorded in the recording unit. An image processing apparatus that identifies a non-blind area and synthesizes the enlarged image so as to overlap the non-blind area. The combining unit calculates an imaging region imaged by the imaging device around the branch point or curve using the distance between the branch point or curve and the vehicle and the imaging information,
A visual field area that falls within the driver's field of view around the branch point or curve is calculated using the distance between the branch point or curve and the vehicle and the driver information,
The image processing apparatus according to claim 1, wherein a position of the non-dead angle area in the captured image is specified using a positional relationship of the visual field area with respect to the imaging area.   The image processing apparatus according to claim 1, wherein the synthesis unit determines the degree of enlargement of the enlarged image using the distance acquired by the vehicle information input unit. The image input unit further inputs a horizontal steering angle of the imaging device at the time of imaging the captured image,
The image processing apparatus according to claim 2, wherein the combining unit calculates the imaging region using the horizontal steering angle. The vehicle information input unit further acquires a value indicating the curvature of a curve in the traveling direction of the vehicle,
The image processing apparatus according to claim 1, wherein the synthesis unit determines a degree of enlargement of the enlarged image using the curvature. The vehicle information input unit further acquires information representing the frequency of occurrence of an accident at the branch point or the curve,
The image processing apparatus according to claim 1, wherein the synthesis unit determines a degree of enlargement of the enlarged image using the frequency of occurrence of the matter. The navigation system provided in the vehicle is operable in cooperation,
7. The vehicle information input unit according to claim 1, wherein the vehicle information input unit obtains a distance between a road branch point or a curve in a traveling direction of the vehicle and the vehicle from a navigation system provided in the vehicle. Image processing device. The vehicle information input unit further acquires, from the car navigation system, information indicating whether or not the enlargement object is present at the branch point or the curve,
8. The enlargement object recognition unit recognizes a predetermined enlargement object included in the input captured image when the enlargement object exists at the branch point or the curve. The image processing apparatus described.   The expansion object recognition unit recognizes a predetermined expansion object included in the input captured image when the distance is equal to or less than a predetermined distance. The image processing apparatus according to claim 1. An image processing method executed by a computer,
An image input step for inputting a captured image captured by an imaging device provided in the vehicle;
A vehicle information input step of inputting a distance between the vehicle and a branch point or curve of a road in the traveling direction of the vehicle based on information from an in-vehicle device provided in the vehicle;
A recording information acquisition step of reading and inputting imaging information indicating characteristics of the imaging device and driver information relating to the visual field of the driver of the vehicle, recorded in a recording unit accessible by the computer;
An enlargement object recognition step for recognizing a predetermined enlargement object included in the input captured image;
A synthesis step of generating a synthesized image obtained by synthesizing an enlarged image obtained by enlarging the enlarged object with the captured image when an enlarged object is recognized in the enlargement target recognition step;
In the synthesizing step, an image of a driver's blind spot in the captured image using the distance acquired in the vehicle information input step and the imaging information and the driver information acquired in the recorded information acquisition step. An image processing method in which a non-blind area that is not a non-blind area is identified and combined so that the enlarged image overlaps the non-blind area. An image input process for acquiring a captured image captured by an imaging device provided in the vehicle;
A vehicle information input process for obtaining a distance between a branch point or a curve of a road in the traveling direction of the vehicle and the vehicle based on information from an in-vehicle device provided in the vehicle;
Recording information acquisition processing for acquiring imaging information indicating characteristics of the imaging device and driver information relating to the field of view of the driver of the vehicle, recorded in a recording unit accessible by a computer;
An enlargement object recognition process for recognizing a predetermined enlargement object included in the input captured image;
An image processing program for causing a computer to execute a synthesis process for generating a synthesized image obtained by synthesizing an enlarged image obtained by enlarging the enlarged object with the captured image when an enlarged object is recognized by the enlargement target recognition process. ,
In the synthesis process, using the distance acquired in the vehicle information input process, the imaging information and the driver information acquired in the recorded information acquisition process, an image of a driver's blind spot in the captured image An image processing program for causing a computer to execute a process of specifying a non-blind spot area that is not and combining the enlarged image so as to overlap the non-blind spot area. An in-vehicle terminal capable of cooperating with an imaging device provided in a vehicle,
A navigation system unit comprising a function for identifying the position of the vehicle, and a map data recording unit for recording road information including the position of a road junction or curve;
An image input unit for acquiring a captured image captured by the imaging device;
A vehicle information input unit that obtains the distance between the vehicle branch point or curve in the traveling direction of the vehicle and the vehicle using the vehicle position and road information specified by the navigation system unit;
A recording unit that records imaging information indicating characteristics of the imaging device and driver information relating to the field of view of the driver of the vehicle;
An enlargement target recognition unit for recognizing a predetermined enlargement object included in the input captured image;
A synthesis unit that generates a synthesized image obtained by synthesizing an enlarged image obtained by enlarging the enlargement object with the captured image when an enlargement object is recognized by the enlargement target recognition unit;
A vehicle-mounted terminal comprising: a display unit that displays a synthesized image generated by the synthesizing unit.

Images