JP5893054B2 - Image processing apparatus, image processing server, image processing method, image processing program, and recording medium - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing server, an image processing method, an image processing program, and a recording medium that superimpose an image indicating a position on a predetermined display object in a front image of a moving object.

  In recent years, a camera mounted on a moving body is used to capture a surrounding image, and among many display objects displayed in the image, a figure or the like indicating an object to which the user should particularly be alerted is superimposed on the image. Techniques for displaying and supporting the attention have been proposed.

  For example, in the technique described in Patent Document 1, when a predetermined feature portion is detected by image recognition or the like in image data, a frame-shaped marker surrounding the feature portion is superimposed on the image.

JP 2004-320286 A

  However, when a surrounding image is captured in a moving image format with a camera mounted on a moving moving body, the moving object moves so that a display object in the image flows. When a frame-shaped marker is superimposed and displayed on a display object that is a characteristic part in the image according to the above-described conventional technique with respect to such a moving image image, processing for recognizing the display object and a frame marker to be superimposed and displayed Since it takes time to calculate the position and size of the image and to draw the frame marker, the frame marker is shifted from the target display object and is displayed in a superimposed manner, which makes it difficult for the user to recognize the attention object in the image. . Therefore, there has been a demand for a superimposing display technique of a support image that can appropriately support the user's recognition of a predetermined attention object in an image captured from a moving body.

  The problem to be solved by the present invention includes the above-described problem as an example.

In order to solve the above-mentioned problem, the invention according to claim 1 is an image processing apparatus, wherein a vanishing point of a forward image taken from a moving moving body or a point near the vanishing point is used as a reference point, and the reference generating means for generating a display object indicating a relative positional relationship between the point and the feature of the in the front image, and output means for outputting the forward side image including the display object generated by the generating means And the generation unit generates the display object so as to include a line segment located on a half line passing through the feature starting from the reference point.

In order to solve the above-mentioned problem, an invention according to claim 7 is an image processing server, wherein a reception unit that receives a front image captured from a moving moving body from an external image processing device, a point in the vicinity of the vanishing point or the vanishing point as a reference point, and generating means for generating display object indicating a relative positional relationship between the reference point and the feature thereof, the display object generated by the generating means Transmitting means to transmit to the external image processing apparatus, and the generating means generates the display object so as to include a line segment located on a half line passing through the feature starting from the reference point To do.

In order to solve the above-mentioned problems, an invention according to claim 8 is an image processing apparatus, comprising: an imaging unit mounted on a traveling moving body for capturing a front image; and an image processing server according to claim 7. Communication means for transmitting and receiving the front image, and display means for displaying the front image including the display object received by the communication means from the image processing server.

In order to solve the above-described problem, an invention according to claim 9 is an image processing method executed by an image processing device, wherein a vanishing point of a forward image taken from a moving moving object or a point in the vicinity of the vanishing point is obtained. a reference point, a generation step of generating a display object indicating a relative positional relationship between the feature of the in the front image and the reference point, and outputs the front image including the display object generated by the generating step An output step, and in the generation step, the display object is generated so as to include a line segment located on a half line passing through the feature starting from the reference point.

In order to solve the above-described problem, the invention described in claim 10 is an image processing program, and causes the image processing apparatus to execute the image processing method described in claim 9 .

In order to solve the above-mentioned problems, the invention described in claim 11 is a recording medium, and the image processing program according to claim 10 is stored so as to be readable by the image processing apparatus.

It is a perspective view showing an example of composition of vehicles carrying a driving support device containing an embodiment of an image processing device of the present invention. It is a block diagram which shows the hardware structural example of a driving assistance device. It is a figure which shows an example of the display of the vehicle window scenery seen from the cab, and the display corresponding to this. It is a figure showing the example of a display at the time of displaying the front image which the front camera of the attitude | position state shown in FIG. 3 imaged on the display. It is a figure showing the example of a display at the time of producing | generating and displaying the frame shape marker as an attention assistance image of a comparative prior art example. It is a figure showing the example of a display at the time of producing | generating and displaying the attention assistance image by embodiment. 4 illustrates a display example when a front image captured by the front camera 1 in the posture state illustrated in FIG. It is a figure showing the example of a display at the time of displaying a front picture which imprinted a lane outline, a lane vanishing point, and an average vanishing point on a display. It is a flowchart showing the control content which CPU of an imaging unit performs. It is a figure showing the example of a display at the time of producing | generating and displaying a caution assistance image by a half-line image. It is a figure showing the example of a display when a display mode is changed before and behind the center position among attention support images of a half line image. It is a figure showing the example of a display at the time of producing | generating and displaying an attention assistance image so that it may overlap with the line segment image a little longer than the display area of an attention object. It is a figure showing the example of a display at the time of producing | generating and displaying a caution assistance image with the two line segment images which envelop the display area | region of the attention object. It is a figure showing the example of a display when changing the display mode before and after the center position of the field between two line segment images which are attention support images. It is a figure showing the example of a display at the time of producing | generating and displaying an attention assistance image with the moving object which moves in a radial direction from a movement display reference point to an attention object. It is a figure showing the example of a display at the time of producing | generating and displaying an attention assistance image with the ellipse which flattened the frame shape marker radially. It is a figure showing the example of a display at the time of producing | generating and displaying a caution assistance image with the line segment image which connects two center positions each detected in the past and the newest image frame. It is a figure showing the example of a display at the time of producing | generating and displaying an attention assistance image combining a substantially radial figure and a frame shape marker. It is an example of the perspective view which shows the structural example of the vehicle carrying a server type driving assistance device, and a network. It is an example of the perspective view which shows the structural example of the vehicle carrying the image processing apparatus of the form using an electronic portable terminal. It is a block diagram which shows an example of the system configuration | structure of a driving assistance device at the time of providing a head-up display projector and a combiner. It is a figure which shows an example of the display of the vehicle window scenery seen from the vehicle inside of a vehicle, and the head-up display corresponding to this. It is the schematic diagram which looked at the aspect of the projection display to the combiner by a head-up display from the side. It is the figure which showed the example of the combination of the sharing of the hardware component between each component apparatus in each form. It is the figure which showed the example of the combination of sharing of the software processing content between each component apparatus in each form.

  Hereinafter, one of the embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a perspective view showing a configuration example of a vehicle equipped with a driving support apparatus including an embodiment of an image processing apparatus of the present invention. In FIG. 1, the driving support device S is provided with a single front camera 1 at a front side position of a room mirror 101 installed on the ceiling of a vehicle V that is a moving body. The imaging direction Dc of the front camera 1 is attached in a posture capable of imaging the front in the traveling direction of the vehicle V. In this example, the front camera 1 is attached to the vehicle V by an attachment device via a universal joint, for example, and the imaging direction Dc is a predetermined range in each of the yaw direction (horizontal direction) and the pitch direction (vertical direction). It is movable manually. The front camera 1 does not deviate from the imaging range the direction Dv of the vehicle axis passing through the center front-rear direction of the vehicle V within the movable range of the posture.

  FIG. 2 is a block diagram illustrating a hardware configuration example of the driving support device S. As illustrated in FIG. In FIG. 2, the driving support device S includes a front camera 1, an imaging unit 2, a display 3, a GPS 13, and a vehicle speed sensor 14.

  The front camera 1 corresponds to an image pickup unit, and uses, for example, a CCD image pickup device or the like to pick up a front image in the traveling direction of the vehicle and outputs a corresponding signal to a CPU (described later) of the imaging unit 2. Have In the example of the present embodiment, the front camera 1 continuously captures a plurality of image frames in a time series with a sufficiently short time period, thereby capturing a front image in the form of a moving image.

  The display 3 corresponds to a display means, and is composed of, for example, an LCD panel, and a caution described later on a front image captured by the front camera 1 based on an image signal input from a graphic controller (described later) of the imaging unit 2. It has a function of displaying a composite image in which a support image is superimposed.

  The GPS 13 has a function of measuring the current location of the vehicle V to acquire current position information and detecting the position of a predetermined attention object (described in detail later) based on map information stored in advance.

  The vehicle speed sensor 14 has a function of detecting the traveling speed of the vehicle V. A later-described CPU provided in the imaging unit 2 can identify whether the vehicle V is in a stopped state or at what traveling speed based on the detection signal of the vehicle speed sensor 14.

  The imaging unit 2 corresponds to an image processing apparatus, and includes a CPU 11, a storage device 12, and a graphic controller 15.

  The CPU 11 has a function of controlling the entire driving support device S by performing various calculations according to the operation of a predetermined program and exchanging information with other units and outputting various control instructions.

  The storage device 12 includes a ROM 12a, a RAM 12b, and a storage medium 12c. The ROM 12a is an information storage medium in which various processing programs and other necessary information are written in advance. The RAM 12b is an information storage medium on which information necessary for executing the various programs is written and read. The storage medium 12c is a non-volatile information storage medium such as a flash memory or a hard disk.

  The graphic controller 15 has a function of acquiring image data from a video RAM (not shown) and the GPS 13 under the control of the CPU 11 and displaying an image signal based on the image data on the display 3. Note that, as described above, the graphic controller 15 in the example of the present embodiment displays on the display 3 an image signal based on composite image data obtained by superimposing a later-described attention support image on the front image captured by the front camera 1. Output. In the example of the present embodiment, the graphic controller 15 causes the display 3 to display an image signal at an arbitrary timing regardless of the time period when the front camera 1 captures a front image in a moving image format.

  FIG. 3 is a diagram illustrating an example of a vehicle window landscape viewed from the cab and a display on the display 3 corresponding thereto. In the illustrated example, the vehicle V travels in a substantially straight traveling lane, and shows a state in which the vehicle V follows a front vehicle Vf that is in front of the traveling direction. The front camera 1 provided on the back side (traveling direction side) of the room mirror 101 captures a forward image including the straight traveling lane and the forward vehicle Vf. At this time, as described above, the imaging range of the front camera 1 includes the vehicle axis direction Dv of the vehicle V without removing it. The display 3 displays a front image captured by the front camera 1.

  Here, since the mounting posture of the front camera 1 is movable as described above, the imaging direction Dc may not be in a posture relationship parallel to the vehicle axial direction Dv as in the example illustrated. Such a shift can also occur when the user attaches the front camera 1 at an arbitrary position or when it is attached manually. Accordingly, in the present embodiment, recognition of a predetermined attention object that is moved and displayed in a moving image front image is taken into consideration while taking into account such a deviation between the imaging direction Dc of the front camera 1 and the vehicle axis direction Dv. Generate and display attention support images that can be supported appropriately. Hereinafter, a method and a display example of such an attention support image will be sequentially described.

  First, FIG. 4 shows a display example when a front image captured by the front camera 1 in the posture state shown in FIG. 3 is displayed on the display 3. In the illustrated example, the front image is displayed with the entire display coordinates XY set in the display area of the display 3, that is, the display coordinates XY coincide with the image coordinates of the front image. In FIG. 4, in addition to the preceding vehicle Vf, the post box 51, the two road signs 52 and 53, the two traffic lights 54 and 55, the sign 56, and the pedestrian 57 are displayed as display objects in the front image. ing.

  Further, as described above, the front camera 1 does not deviate the vehicle axis direction Dv of the vehicle V from the imaging range within the movable range of the posture, so that the moving display reference is always included in the display coordinates XY of the front image. Point P2 exists. This moving display reference point P2 is the infinitely distant moving position of the vehicle V in the current traveling direction in the forward image (the vehicle V is displayed in the display coordinates XY in the forward image displayed at that time. This is a point corresponding to a planned position that is reached when the vehicle moves in the traveling direction as it is. This point is always an approximate horizon (substantially horizontal line: not particularly shown in FIG. 4) at that time and an abbreviation of the vehicle axial direction Dv It is located on the intersection with the extension line, and this is called the vanishing point. The movement display reference point P2 corresponds to a reference point described in each claim.

  As the vehicle V moves in the vehicle axial direction Dv, all the display objects in the above-described front image move radially with the movement display reference point P2 as the center. That is, all the display objects in the front image are located in the vicinity of the movement display reference point P2 when they are far away from the vehicle V. Then, as the vehicle V moves toward the moving display reference point P2, each display object is on a half line (in the drawing) arranged substantially radially around the moving display reference point P2 in the moving image front image. (Refer to the broken arrow in FIG. 4).

  In the following, the case where the above-described postbox 51 is set as the attention object F will be considered. The attention object F is a display object that should alert the user who is a driver or a passenger of the vehicle V so as not to particularly overlook the many display objects displayed in the front image. is there. Here, as an example, the post box 51 is the attention object F, but the present invention is not limited to this. In another example, the vehicle may move at a sufficiently low speed compared with the speed of a vehicle such as a road sign, a traffic light, a signboard, or a vehicle such as a pedestrian. The attention object F corresponds to a predetermined feature described in each claim. The feature object includes a feature portion that can be detected by image recognition. The imaging unit 2 recognizes this characteristic part and detects the attention object F from the front image. As a method for setting the attention object F, a predetermined type of characteristic object such as a post box or a road sign may be set in advance, or it will be passed based on the map information stored in the GPS 13 in the future. A feature set near the road may be designated and set in advance, or the user directly touches on a touch panel (not shown in particular) provided with the feature in the front image on the display 3 A feature object of the same type as the specified feature object may be set as the attention object F by directly designating with an index such as a cursor.

  For the post box 51 set as the attention object F in this way, a frame-shaped marker 61 as shown in FIG. 5 is generated as the attention support image in order to call attention in the front image, and the post box 51 It is conceivable to superimpose and display 51 display areas. However, in this case, the CPU 11 of the imaging unit 2 detects the display area of the postbox 51 and its position by image recognition at the display coordinates XY of the front image, and has a frame shape according to the size of the display area. Processing to calculate and draw the size of the marker 61 must be performed. All of these processes are computationally intensive, and in particular, image recognition takes a long processing time. For this reason, when the front image is displayed in a moving image format, the corresponding postbox 51 is far from the frame shape even if the frame-shaped marker 61 is superimposed and displayed as shown in FIG. The attention from the user becomes difficult.

  Therefore, in the present embodiment, the attention support image is generated and displayed so as to geometrically indicate the relative positional relationship between the movement display reference point P2 and the position where the attention object F exists. Specifically, in the example of the present embodiment, as shown in FIG. 6, the center position P3 of the display area of the postbox 51 detected by image recognition in the front image is set as an existing position, and the moving display reference point P2 and the center are displayed. An attention support image is generated and displayed by a line segment G1 connecting the position P3. As a result, the line segment G1 of the attention support image generated and displayed in this way is formed as a substantially radial figure centered on the moving display reference point P2. This attention support image corresponds to the display object described in each claim.

  As described above, any display object displayed in the front image in the moving image format is moved and displayed radially with the movement display reference point P2 as the center as the vehicle V moves. That is, the line segment image G1 generated and displayed as the above-described attention support image indicates a movement locus in the front image of the post box 51 that is the attention object F. For this reason, even when the center position P3 of the post box 51 detected by the image recognition is slightly shifted from the center position in the latest image frame of the front image, the post box 51 is positioned on the extension of the line segment G1 of the attention support image. Therefore, it is easy to visually interpolate the deviation, and the user can recognize the post box 51 naturally and intuitively. The same effect can be obtained even when a pedestrian or the like sufficiently slow compared with the moving speed of the vehicle V is used as the attention object F.

  Here, the position of the moving display reference point P2 in the display coordinates XY of the front image often deviates from the center position of the display coordinates XY, and in which direction the imaging direction Dc of the front camera 1 at that time is. It depends on what you are looking for. In the present embodiment, the position of the moving display reference point P2 in the display coordinates XY is acquired by calculating the average vanishing point described below.

  First, FIG. 7 shows a display example when a front image captured by the front camera 1 in the posture state shown in FIG. 3 is displayed on the display 3. In order to avoid the complexity of illustration, only the forward vehicle Vf is displayed as a display object. In FIG. 7, when there is a deviation between the imaging direction Dc of the front camera 1 and the vehicle axis direction Dv as described above, the center position of the display coordinates XY of the display 3, that is, the vertical direction in the figure. The intersection positions of the center lines Lv and Ll in the lateral direction are separated from the vehicle axial direction Dv. In this case, the front image loses left-right symmetry with respect to the center line Ll. For example, even when the vehicle V is actually traveling in a straight traveling lane, the left and right sides of the traveling lane are sandwiched. The display objects displayed on the screen are asymmetric with respect to the travel lane in the direction and speed of movement in the front image.

  On the other hand, in the forward image captured when the vehicle is traveling in a straight traveling lane as in the illustrated example, the lane vanishing point P1 corresponding to the infinitely far point of the traveling lane is always an abbreviation of the time. It is located on the intersection of the horizon (substantially horizontal line) and the substantially extended line in the vehicle axial direction Dv. That is, the position of the lane vanishing point P1 when traveling in the straight traveling lane in this way matches the position of the movement display reference point P2.

  As a method for detecting the lane vanishing point P1, first, as shown in the figure, a white line applied to both sides of an actual traveling lane, a boundary line between a road surface and a curb, or an edge part of an asphalt road is displayed as a front image. And is recognized as two lane contour lines Lr forming the contour of the travel lane. Then, the intersection of the two lane outlines Lr on the extension line in the vehicle traveling direction is detected as the lane vanishing point P1 on the front image.

  Here, for example, as shown in FIG. 8, when the traveling lane is curved, the two lane contour lines Lr are also recognized in accordance with the direction and curvature of the curve on the front image. Although not particularly illustrated, when the curvature of this curve changes midway, the two lane contours Lr are recognized with priority given to the curvature on the near side of the vehicle V. Then, an extension line of the curve is calculated with an approximate expression based on the curvature on the near side, and the position of the intersection is detected as the lane vanishing point P1.

  When the vehicle V has traveled for a long time in various lanes and detected and recorded many lane vanishing points P1, the average vanishing point P2 located at the average position of the lane vanishing points P1 is statistically Close to the vehicle axis direction Dv and on the horizon. That is, the position of the average vanishing point P2 on the display coordinates XY can be regarded as the fixed position of the moving display reference point P2 corresponding to the imaging direction Dc of the front camera 1 at that time. As described above, in the present embodiment, as a preparatory work before generating an appropriate attention support image, the vehicle V travels in various lanes, detects a sufficiently large number of lane vanishing points P1, and based on them. A reliable average vanishing point P2 is calculated. The calculated average vanishing point P2 is acquired as the moving display reference point P2 on the display coordinates XY corresponding to the imaging direction Dc of the front camera 1 at that time.

  FIG. 9 is a flowchart showing the control contents executed by the CPU 11 of the imaging unit 2 in order to realize the operation mode described above. This flow is called and executed at an appropriate time interval, for example, while the front camera 1 captures a front image in the form of a moving image.

  In FIG. 9, first, in step S5, based on the detection signal of the vehicle speed sensor 14, it is determined whether or not the vehicle V is traveling at a predetermined speed or higher. If the vehicle V is traveling at a predetermined speed or higher, the determination is satisfied, and the routine goes to Step S10.

  In step S10, a front image (still image frame) captured by the front camera 1 at that time is acquired.

  The process proceeds to step S15, and it is determined whether or not the average vanishing point P2 on the display coordinates XY corresponding to the imaging direction Dc of the front camera 1 has already been acquired since it was previously set. If the average vanishing point P2 corresponding to the imaging direction Dc at that time has not yet been acquired, the determination is not satisfied, and the routine goes to Step S20.

  In step S20, it is determined whether or not a predetermined time has elapsed since the previous detection of the lane vanishing point P1. If the predetermined time has not elapsed, the determination is not satisfied, and the process returns to step S5 and the same procedure is repeated. Thus, by providing a time interval for detecting the lane vanishing point P1, the diversity of the detected lane vanishing point P1 can be ensured and the reliability of the detected average vanishing point P2 can be improved.

  On the other hand, if the predetermined time has elapsed, the determination is satisfied, and the routine goes to Step S25.

  In step S25, two lane contour lines Lr in the current travel lane are recognized in the front image captured in step S10.

  Moving to step S30, the lane vanishing point P1 in the current traveling lane is detected based on the two lane contours Lr recognized in step S25.

  Moving to step S35, the lane vanishing point P1 detected in step S30 is additionally recorded so as to be accumulated in the storage medium 12c.

  The process proceeds to step S40, and it is determined whether or not the lane vanishing point P1 is detected and stored with a sufficient number that can detect the average vanishing point P2 that is already highly reliable at that time. If the number of detections is not yet sufficient at that time, the determination is not satisfied and the routine returns to step S5 and the same procedure is repeated.

  On the other hand, when the lane vanishing point P1 is detected and stored with a sufficient number that can detect the average vanishing point P2 with high reliability already at that time, the determination is satisfied, and the routine goes to Step S45.

  In step S45, all lane vanishing points P1 recorded in the storage medium 12c are read out.

  Moving to step S50, the average vanishing point P2 located at the average position of all the lane vanishing points P1 read out in step S45 is calculated. And it returns to step S5 and repeats the same procedure.

  On the other hand, if the average vanishing point P2 corresponding to the imaging direction Dc at that time has already been acquired in the determination in step S15, the determination is satisfied, and the process proceeds to step S55.

  In step S55, it is determined whether or not the attention object F is set. Note that the attention object F may be set in advance or set by the user. Moreover, when there is a setting of the attention object F based on the map information, it may also be determined whether or not the attention object F is close enough to be displayed in the front image. If no attention object F has been set, or if it has been set based on map information but is not yet close enough to be displayed in the front image, the determination is not satisfied and the process returns to step S5. Repeat the same procedure.

  On the other hand, if the attention object F has been set and it is necessary to display the attention support image on the front image, the determination is satisfied, and the routine goes to Step S60.

  In step S60, the attention object F is recognized by image recognition in the front image acquired in step S10, and the center position P3 of the display area is detected as the presence position of the attention object F. In addition, the procedure of this step S60 corresponds to the generating means and the generating step described in each claim.

  Proceeding to step S65, an image of a line segment G1 connecting the average vanishing point P2 to the center position P3 of the attention object F on the display coordinates XY is generated as an attention support image. In addition, the procedure of this step S65 also corresponds to the generation means and generation process described in each claim.

  The process proceeds to step S70, where the line segment G1 of the attention support image generated in step S65 is output to the display 3 so as to be superimposed on the front image. And it returns to step S5 and repeats the same procedure. The procedure of step S70 corresponds to the output means and the output process described in each claim.

  As described above, in the imaging unit 2 of the above embodiment, the moving display located at or near the average vanishing point P2 (corresponding to the vanishing point) of the forward image taken from the traveling vehicle V (corresponding to the moving object). As an attention support image (corresponding to a display object) indicating a relative positional relationship between the reference point P2 (corresponding to a reference point) and a predetermined attention object F (corresponding to a predetermined feature) in the front image. Steps S60 and S65 for generating the line segment image G1 (corresponding to generating means), and a procedure for outputting the forward image including the attention support image generated by the steps S60 and S65 (output means) Equivalent).

  In the image processing method executed by the imaging unit 2 of the above embodiment, the average vanishing point P2 (corresponding to the vanishing point) of the forward image taken from the traveling vehicle V (corresponding to the moving body) is located in the vicinity thereof. Attention support image (corresponding to a display object) showing a relative positional relationship between the moving display reference point P2 (corresponding to a reference point) and a predetermined attention object F (corresponding to a predetermined feature) in the front image. Steps S60 and S65 for generating the line segment image G1 (corresponding to the generation step), and an output step for outputting the front image including the attention support image generated by the steps S60 and S65. Run.

  If it does in this way, with the movement to the advancing direction at that time of the vehicle V, all the display objects in a front image will be moved and displayed from the movement display reference point P2 or its vicinity to the outer peripheral side of the said front image. . The attention support image G1 showing the relative positional relationship between the movement display reference point P2 and the center position P3 of the attention object F is an image schematically showing the movement direction of the attention object F in the front image. Become. For this reason, even when the center position P3 of the attention object F detected by the image recognition that takes a relatively long processing time is slightly deviated from the center position of the same attention object F in the latest image frame, the user can support the attention. By referring to the image G1, the position of the attention object F in the front image can be intuitively grasped. As a result, it is possible to superimpose and display an attention support image that can appropriately support attention to a predetermined attention object F in an image captured from the vehicle V.

  In addition to the configuration described above, in the procedure of step S65, the attention support image G1 may be generated with a substantially radial figure centered on the moving display reference point P2 in the front image.

  In this way, the attention support image G1 generated with a substantially radial graphic almost overlaps the movement trajectory in the front image of the attention object F. Therefore, even when the center position P3 of the attention object F detected by the image recognition is slightly deviated from the center position of the same attention object F in the latest image frame of the front image, the attention object F is displayed in the attention support image G1. Therefore, the user can visually grasp the target object F naturally and intuitively.

  In addition to the configuration described above, in step S25, in the front image, two lane contour lines Lr that form the contours of the travel lane that are located on both the left and right sides of the travel lane on which the vehicle V travels are recognized. The procedure, the procedure of step S30 for detecting the intersection of the two lane outlines Lr on the extension line in the traveling direction as a lane vanishing point P1, and the plurality of lanes detected in the past by the procedure of step S30 An average vanishing point P2 located at each average position of the vanishing point P1 is calculated, and the procedure of step S50 is executed as the moving display reference point P2 used in the procedure of step S65.

If the average vanishing point P2 is calculated by the method as described above, even when the imaging direction Dc of the front camera 1 is in an arbitrary direction, the vanishing point can be accurately identified and set as the moving display reference point P2. . For this reason, as shown in FIG. 6, the direction of the line segment G1 connecting the center position P3 of the display area of the post box 51 and the moving display reference point P2 remains unchanged, and the post box 51 is an extension of the line segment G1 of the attention support image. Since it is accurately positioned above, the user can obtain the maximum effect of being able to recognize the postbox 51 naturally and intuitively.
In the configuration described above, the average vanishing point P2 obtained from the front image is the moving display reference point P2, but the moving display reference point P2 may be set by another method. For example, the vanishing point may be calculated by another method, and when the imaging direction Dc of the front camera 1 is installed almost accurately, the center point of the front image may be used as the movement display reference point P2. In any method, if the vicinity of the accurate vanishing point can be set as the movement display reference point P2, the direction of the line segment G1 connecting the center position P3 of the display area of the postbox 51 and the movement display reference point P2 becomes almost unchanged. Since the post box 51 is positioned on an extension of the line segment G1 of the attention support image, the user can recognize the post box 51 naturally and intuitively.

  In addition to the configuration described above, the predetermined attention object F in the front image is a feature or a pedestrian.

  In this case, the feature is stationary with respect to the ground, and the pedestrian moves at a sufficiently low speed as compared with the speed of the vehicle V, so that the relative speed between them and the vehicle V is almost the same. It can be regarded as equivalent to the vehicle speed of the vehicle V. Thereby, the center position of the same attention object F in the latest image frame does not greatly deviate from the moving direction in the front image of the attention object F indicated by the attention support image G1. As a result, the reliability of the attention support image G1 for intuitively grasping the position of the attention object F in the front image can be ensured.

  In addition to the above-described configuration, the procedure of step S65 generates the attention support image so as to include a line segment that is located on a half line that passes through the center position P3 of the attention object F starting from the average vanishing point P2. May be.

  In this manner, the line segment image G1 generated as the attention support image substantially coincides with the past movement locus or the future movement locus in the forward image of the attention object F. Therefore, even when the center position P3 of the attention object F detected by the image recognition is slightly shifted from the center position of the same attention object F in the latest image frame of the front image, the line segment image G1 of the attention support image is There is a high possibility of overlapping with the attention object F in the latest image frame. Further, even when there is no overlap, the attention object F is located on the extension of the line segment image, so that the deviation can be easily visually interpolated, and the user can grasp the attention object F naturally and intuitively. . Further, since attention is supported by the line segment image G1, it is possible to clearly specify and indicate the direction or area to be noted in a relatively narrow range.

  Further, in the frame shape marker 61 shown in FIG. 5 as the attention support image of the comparative conventional example, the image frame of the front image is switched and the attention object F moves and displays on the display coordinates XY to follow it. Therefore, it is necessary to redraw the entire frame shape marker 61 by changing the position. However, when the attention support image is generated and displayed by the line segment image G1 connecting the moving display reference point P2 and the center position P3 of the attention object F as shown in FIG. Even if F moves on the display coordinates XY, it may be additionally drawn so as to extend the distal end portion in the radial direction of the line segment image G1. In this case, since it is not necessary to redraw the entire line segment image G1, not only the burden of the drawing process is reduced, but the center of the attention object F by image recognition can be extended if the tip portion can be extended with an appropriate length. Since it is not necessary to repeat the detection of the position P3, the burden of the processing for generating the attention support image G1 can be greatly reduced, and the overall processing speed can be increased.

  In addition to the above-described configuration, the front camera 1 (corresponding to the imaging unit) that supplies the front image mounted on the vehicle V and captured in the steps S60 and S65, and the step S70. And a display 3 (corresponding to display means) for displaying the front image including the attention support image G1 output in step S1.

  In this way, the driving support device S can integrally and independently perform the imaging of the front image and the display of the front image on which the attention support image G1 is superimposed.

  In addition to the above-described configuration, the steps S60 and S65 may be set as the attention object F that is arbitrarily designated by the user among the display objects existing in the front image.

  In this way, since the display object directly designated on the front image by the user can be set as the attention object by the direct contact on the touch panel (not shown) provided on the display 3 or the index such as the cursor, the attention object It is possible to set the object F in real time, and convenience is improved.

  In the above embodiment, the attention support image is generated and displayed with the line segment image G1 connecting the moving display reference point P2 to the center position P3 of the display area of the attention object F as shown in FIG. The present invention is not limited to this. For example, as shown in FIG. 10, the attention support image is represented by a half-line image G2 that passes through the center position P3 of the display area of the attention object F (the mail post 51 in the illustrated example) starting from the moving display reference point P2. It may be generated and displayed. Even in this case, the half-line image G2 of the attention support image results in a substantially radial figure centered on the movement display reference point P2, and the whole half-line image G2 is the past and future movement of the attention object F. The entire locus is shown and always overlaps the display area of the attention object F. For this reason, it is possible to prompt the user to pay attention to the attention object F more reliably than the line segment image G1 in the above embodiment.

  Further, the half-line image G2 is generated and displayed so that the display mode is different between the section from the moving display reference point P2 to the center position P3 of the attention object F and the section on the outer peripheral side from the center position P3. Also good. For example, as shown in a half-line image G2A in FIG. 11, a solid line is displayed in the section from the movement display reference point P2 to the center position P3 of the attention object F, and a broken line is displayed in the section on the outer peripheral side from the center position P3. You may display in the aspect of. In this way, the entire attention support image G2A can indicate the entire past and future movement trajectories of the attention object F, and the approximate position of the attention object F in the entire movement trajectory is at that time. You can show if there is. For this reason, it is possible to prompt the user to pay attention to the attention object F more clearly than the simple half-line image G2.

  Even when the attention support image is generated as a line segment image, for example, as shown in FIG. 12, a line segment image G3 having a length slightly longer than the size of the display area of the attention object F is used. You may produce | generate and display so that it may overlap with the center position P3 of F. In this case, while the line segment image G3 simply indicates the moving direction of the attention object F, the center position P3 can be particularly noted. Further, by adding an arrowhead to the outer peripheral side of the line segment image G3, the entire line segment image G3 can be seen as a movement vector of the attention object F, and the movement of the attention object F in the front image is performed. The expression corresponding to is possible.

  In the above description, the attention support image is generated and displayed so as to include a line segment located on the half line passing through the center position P3 of the display area of the attention object F starting from the moving display reference point P2. Is not limited to this. For example, as shown in FIG. 13, the attention support image may be generated and displayed as an image including two line segments G4 that envelop the display area of the attention object F, each starting from the moving display reference point P2. Even in this case, the two line segment images G4 result in a substantially radial figure centered on the moving display reference point P2, and the entire area between the two line segment images G4 is the object of attention. The entire display range reflecting the scale of the display area of F in the past and the future perspective is shown and always overlaps the display area of the attention object F. For this reason, it is possible for the user to grasp the existence range of the attention object F more reliably than the line segment image G1 in the above embodiment.

  Further, the display mode is different between the section from the moving display reference point P2 to the center position P3 of the attention object F and the section on the outer peripheral side from the center position P3 in the two line segment images G4 and the area between them. May be generated and displayed. For example, as indicated by G4A in FIG. 14, in the region between the two line segment images G4, a section from the moving display reference point P2 to the center position P3 of the attention object F, and the outer peripheral side from the center position P3. Alternatively, the color (or density) may be changed for each section and displayed by semi-transparent paint processing. In this way, the entire attention support image G4A can indicate the entire passing range reflecting the scale of the display area of the attention object F in the past and the future, and the attention object within the entire passing range. It can indicate where F is approximately at that time. For this reason, it is possible to prompt the user to pay attention to the attention object F more clearly than the above-described two simple line segment images G4.

  In the above description, the attention support image is generated and displayed fixedly arranged in a radial position around the moving display reference point P2, but the present invention is not limited to this. For example, as shown in FIG. 15, a moving object G5 that moves in a radial direction from the moving display reference point P2 on a half line passing through the center position P3 of the display area of the attention object F starting from the moving display reference point P2. An attention support image may be generated and displayed with an image including. In the example shown in FIG. 15, the center position P3 of the attention object F from the movement display reference point P2 on the line segment connecting the movement display reference point P2 and the center position P3 of the attention object F (see the dotted line in the figure). A caution support image is generated and displayed by a moving object G5 in the form of a meteor that dynamically moves in the direction toward. The meteor moving object G5 shown in this example expresses the moving direction with the tip and tail fins, and moves and displays the meteor at a speed faster than the movement of the attention object F in the front image. In this case, the movement trajectory and movement direction of the attention object F can be expressed by dynamic animation. Note that a line segment (see the dotted line in the figure) connecting the movement display reference point P2 and the center position P3 of the attention object F may be displayed or may be omitted. Further, the meteor moving object G5 may be stopped at the center position P3 of the attention object F, or may be moved to the outer peripheral side of the forward image by overtaking the attention object F as it is.

  Also for the circular frame shape marker 61 shown in FIG. 5 as the attention support image of the conventional comparative example, as shown in FIG. 16, the center of the display area of the attention object F starts from the moving display reference point P2. The attention support image G6 may be generated and displayed by being flattened along a half line (see the broken line in the drawing) passing through the position P3. Even in this case, the elliptical attention support image G6 results in a substantially radial figure centered on the constantly moving reference point P2, and the entire elliptical image G6 is the entire past and future movement locus of the attention object F. Is always shown in the display area of the attention object F. For this reason, it is possible for the user to grasp the existence range of the attention object F more reliably than the line segment image G1 in the above embodiment.

  In the above description, the relative position between the center position P3 of the attention object F detected in the image frame of one still image and the moving display reference point P2 that is fixedly acquired on the display coordinates XY. However, the present invention is not limited to this. For example, two center positions P3 respectively detected in the past image frame and the latest image frame with respect to the same attention object F among a plurality of image frames captured in time series with a moving image front image. An attention support image may be generated and displayed so as to include a connecting line segment. In this case, for example, a line segment image G7 as shown in FIG. 17 becomes a caution support image, and this line segment image G7 is also a substantially radial figure centered on the movement display reference point P2, as a result. 17 shows geometrically the relative positional relationship between the point P2 (not shown in FIG. 17) and the center position P3 of the attention object F (the center position P4 in the figure indicates a past image frame). Center position detected in step 1). For this reason, the center position P3 can be particularly noted while the moving direction of the attention object F is simply indicated by the line segment image G7. In addition, since the line display image G7 is not required to set the movement display reference point P2, it is useful as a caution support image to be displayed when a sufficient lane vanishing point P1 has not yet been obtained.

  Also, for example, as shown in FIG. 18, the attention support image (line segment image G1 in the illustrated example) of the substantially radial figure centering on the moving display reference point P2 as described above, and the attention support image of the conventional comparative example As an alternative, the frame shape marker 61 shown in FIG. 5 may be combined and displayed. When the imaging period of the image frame of the front image is sufficiently long or the processing speed of the CPU 11 of the imaging unit 2 is sufficiently high, as shown in the figure, any attention support image G1, 61 is generated and displayed with an accurate arrangement. These combinations can most clearly prompt the user to pay attention to the attention object F.

  Further, in the driving support device S of the present embodiment, the front image on which the attention support images G1 to G7 are superimposed may be directly recorded on the storage medium 12c or the like, and applied to a drive recorder that is played back later. Alternatively, only the front image is recorded in the storage medium 12c, and the attention support images G1 to G7 are generated and superimposed for the first time at the time of later reproduction or editing, and the superimposed image is further recorded. Also good.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and technical idea of the present invention.

(1) First modification: When a front image is stored in an external database In the above embodiment, the driving support device S uses the front image captured by itself, and generates and displays the image recognition process and the attention support image. Or a so-called stand-alone type in which the front image is recorded on the storage medium 12c provided therein, but the present invention is not limited to this. For example, as shown in FIG. 19, the driving support device S2 includes a wireless communication device 21 (corresponding to communication means) and an antenna 22, and communicates via wireless communication with a relay station 23 (corresponding to receiving means and transmitting means). A forward image is transmitted to a database server 25 (corresponding to an image processing server) provided in an external data center 24 on the network, and image recognition of the forward image, generation of an attention support image, and recording thereof are performed on the database server 25. It may be a so-called server type. Then, the front image on which the attention support image is superimposed may be returned from the database server 25 to the driving support device S2 and displayed on the display 3. Alternatively, only the attention support image may be returned from the database server 25 to the driving support device S2, and the driving support device S2 may superimpose the attention support image on the front image and display it on the display 3.

  As a result, even if the past forward image is accidentally deleted on the storage medium 12c of the driving assistance device S2, the forward image and the attention assistance image recorded in the database server 25 of the data center 24 are used as backups. it can.

(2) Second modification: When configured using an electronic portable terminal In the above embodiment and the first modification, the device configuration mounted on the vehicle V is only the driving support devices S and S2. The present invention is not limited to this. In recent years, there are many models in which an electronic portable terminal 31 such as a commercially available smartphone or an electronic book terminal is equipped with a camera, a display, or a GPS, which is detachably attached to, for example, a cradle 32 connected to the vehicle speed sensor 14. It may be mounted on the vehicle V (see FIG. 20). Then, the electronic portable terminal 31 is caused to capture a front image, and a CPU included in the same electronic portable terminal 31 is caused to function as an image processing device by causing the CPU to perform predetermined image recognition processing and attention support image generation. Also in the image processing apparatus having such a configuration, the attention support image exemplified in the above embodiment can be generated and displayed.

  In this case, the electronic portable terminal 31 corresponds to the image processing device described in each claim, the camera equipped in the electronic portable terminal 31 corresponds to the imaging means, and among the control procedures processed by the CPU of the electronic portable terminal 31 The procedure for generating the attention support image corresponds to the generation means, the procedure for outputting the front image including the attention support image corresponds to the output means, and the display equipped in the electronic portable terminal 31 corresponds to the display means.

  Although not particularly illustrated, the above-described server type can be used even in a configuration using the electronic portable terminal 31. In this case, the electronic portable terminal 31 includes a communication unit, and the database server 25 of the data center 24 corresponds to an image processing server that recognizes a front image, generates a attention support image, and records them.

  The above-described embodiment, first modified example, and second modified example all have the same basic hardware components and software processing contents, and their sharing among the constituent devices is different. Only. Major examples of these assignments are shown in FIGS.

  First, FIG. 24 shows a combination example of sharing of hardware components among the component devices. 24 corresponds to the storage medium 12c or the database server 25 that records the front image on which the attention support image is superimposed, and the communication unit corresponds to the wireless communication device 21. Is. In-vehicle devices (driving support devices, cradle) in the component equipment may be equipped with a vehicle speed sensor, and the data center may be equipped with a storage unit. In addition, GPS may be installed in either an in-vehicle device or a portable terminal, but cannot be installed in a data center. In addition, in the case of the server type, it is essential to equip either the mobile terminal or the in-vehicle device and a communication unit in the data center.

  In addition to the sharing example shown in the drawing, the camera and the display may be equipped either with a portable terminal or an in-vehicle device, and the storage unit may be equipped with any of the portable terminal, the in-vehicle device, or the data center. Each hardware component only needs to be provided in any one of the components that can be equipped, but for example, a configuration may be adopted in which both the mobile terminal and the in-vehicle device are provided with a display. Moreover, in a portable terminal usage form, when a vehicle-mounted apparatus is provided with any of a camera, a display, GPS, a memory | storage part, and a communication part, when using it, a portable terminal and vehicle-mounted apparatus are a connector, short-range wireless, It is necessary to be able to attach and receive information signals via communication or the like. Of the hardware components, the camera, the display, and the communication unit (in the case of the server type) must be installed in any of the corresponding components, but other GPS, storage units, and vehicle speed sensors The equipment is optional.

  Next, FIG. 25 shows an example of a combination of sharing of software processing contents among the constituent devices. The software processing contents shown in FIG. 25 include a calculation process for calculating the average vanishing point P2 (moving display reference point P2), a detection process for detecting the center position P3 of the attention object F in the front image, and an attention support image. There are four generation processes: a generation process to generate and a recording process to record a front image on which the attention support image is superimposed. Of these software processing contents, the detection process and the generation process must be performed by any of the constituent devices, but the other calculation processes and storage processes are optional. As for the sharing of software processing contents, basically, if a CPU with sufficient processing capability is installed in each component device and can send and receive information to and from each other, any processing content is basically shared by any component device. You can do it. The sharing example shown in the figure is a main example to the last, and various other combinations are possible.

(3) Third modification: When displaying the attention support image on the head-up display In the above embodiment, the composite image obtained by superimposing the attention support image on the front image captured by the front camera 1 is displayed on a single display. Although displayed, the present invention is not limited to this. For example, the display coordinates XY may be arranged so as to coincide with the driver's field of view through the window glass of the vehicle V, and a head-up display that displays a caution support image on the window glass or in front of it may be provided. .

  FIG. 21 is a block diagram illustrating a hardware configuration example of the driving support device S4 of the present modification example corresponding to FIG. 2 of the above embodiment. In FIG. 21, the driving support device S4 of the present modification has a head-up display projector 4 and a combiner 5 instead of the display 3, and the other hardware configuration is the same as that of the above embodiment. However, the graphic controller 15 outputs an image signal including only the image data of the attention support image without including the front image.

  FIG. 22 corresponding to FIG. 3 of the above embodiment is a diagram showing an example of a vehicle window scenery viewed from the inside of the vehicle V and a display of a head-up display corresponding to this. In the example shown in FIG. 22, the head-up display projector 4 is fixed to the ceiling roof above the driver's seat located on the right side in the figure. A combiner 5 is provided so as to cover the projection range of the head-up display projector 4 on the front side of the windshield 104. The head-up display projector 4 and the combiner 5 correspond to a head-up display. Based on the image signal output from the graphic controller 15, an attention support image (illustrated) is displayed on the front surface of the combiner 5 having a function as a half mirror as appropriate. In this example, the image of the line segment image G1) is projected and displayed.

  FIG. 23 is a schematic view of the aspect in which the head-up display projects and displays on the front side of the windshield 104 as seen from the side. The head-up display projector 4 sets display coordinates XY on a combiner 5 having a function as a half mirror, and projects and displays images of the attention support images G1 to G7. Here, the display coordinates XY are set to be arranged so as to coincide with the driver's field of view through the windshield 104 of the vehicle V, so that the driver is in the front field of view through the windshield 104. The attention support images G1 to G7 that geometrically indicate the relative positional relationship between the movement display reference point P2 and the position where the attention object F exists can be visually recognized.

  In this case, in order to achieve alignment between the arrangement of the attention support images G1 to G7 on the display coordinates XY and the arrangement in the driver's front view, the height of the driver's seat (the height position of the eyeball) It is necessary to appropriately adjust the arrangement of the display coordinates XY of the head-up display projector 4 according to the deviation of the imaging direction Dc of the front camera 1.

  As described above, in the driving support device S4 of the third modified example, the front camera 1 that is mounted on the traveling vehicle V and obtains the front image of the vehicle V, and the average vanishing point P2 in the front image. Alternatively, it is generated by the steps S60 and S65 for generating the attention support image indicating the relative positional relationship between the moving display reference point P2 located in the vicinity thereof and the predetermined attention object F, and the steps S60 and S65. The step S70 for outputting the attention support image that has been output, and the head-up display projector 4 and the combiner 5 (head-up display) for projecting the attention support image output in the step S70 within the field of view of the driver during driving. Equivalent).

  In this way, since the video of the attention support images G1 to G7 whose positions are aligned with the front view itself that the driver visually recognizes through the windshield 104 can be projected and displayed, the driver can support the attention without moving the line of sight greatly. Images G1 to G7 can be visually recognized and can be concentrated on driving.

  In the third modification, the head-up display is configured by projecting and displaying the images of the attention support images G1 to G7 from the head-up display projector 4 on the combiner 5 provided on the front side of the windshield 104. However, the present invention is not limited to this. In addition, for example, a transmissive liquid crystal panel or the like may be provided on the windshield itself, and the attention support image may be displayed by the self-light emission.

  In addition to those already described above, the methods according to the above-described embodiments and modifications may be used in appropriate combination.

1 Front camera (equivalent to imaging means)
2 Imaging unit (equivalent to image processing device)
3 Display (equivalent to a display device)
11 CPU
12 storage unit 13 GPS
14 Vehicle speed sensor 15 Graphic controller 21 Wireless communication device (corresponding to communication means)
22 Relay station (equivalent to receiving means, transmitting means)
24 data center 25 database server (equivalent to image processing server)
31 Electronic portable terminal (equivalent to a portable terminal device)
32 Cradle 51 Postbox 61 Frame-shaped marker Dc Imaging direction Dv Vehicle axial direction F Attention object (corresponding to a predetermined feature)
G1-G7 Attention support image (equivalent to display object)
Lr Lane outline P1 Lane vanishing point P2 Average vanishing point, moving display reference point (corresponding to the reference point)
P3 Center position of the display area of the target object S, S2, S3 Driving support device, S4
V vehicle

Claims (11)

Using a vanishing point of a forward image taken from a moving moving object or a point near the vanishing point as a reference point, a display object indicating a relative positional relationship between the reference point and a feature in the front image is generated. Generating means;
Output means for outputting a front image including the display object generated by the generation means;
With
The image processing apparatus according to claim 1, wherein the generation unit generates the display object so as to include a line segment located on a half line passing through the characteristic object with the reference point as a starting point.   2. The display unit according to claim 1, wherein the generation unit generates the display object so as to include a moving object that moves in a radial direction from the reference point on a half line that passes through the feature starting from the reference point. The image processing apparatus described.   3. The generation unit according to claim 1, wherein the generation unit generates the display object so that a display mode of the section from the reference point to the feature is different from that of the section from the feature to the outer peripheral side. Image processing apparatus. The front image is a plurality of image frames captured in time series as a moving image format,
The generating means generates the display object so as to include a line segment connecting the positions of the two feature objects detected in the past image frame and the latest image frame with respect to the same feature object. The image processing apparatus according to any one of claims 1 to 3. Imaging means for supplying the generating means with the forward image mounted on the moving body;
Display means for displaying a front image including the display object output by the output means;
The image processing apparatus according to claim 1, further comprising: The image processing apparatus according to claim 5 , wherein the display unit is a head-up display that displays the display object output by the output unit in a field of view when the driver is driving. Receiving means for receiving, from an external image processing device, a front image captured from a moving moving body;
Generating means for generating a display object indicating a relative positional relationship between the reference point and the feature object, with a vanishing point in the front image or a point near the vanishing point as a reference point;
Transmitting means for transmitting the display object generated by the generating means to the external image processing device;
With
The image processing server characterized in that the generation means generates the display object so as to include a line segment located on a half line passing through the feature starting from the reference point. An imaging means mounted on a traveling moving body for capturing a front image;
Communication means for transmitting and receiving the front image to the image processing server according to claim 7;
An image processing apparatus comprising: a display unit configured to display a front image including the display object received from the image processing server by the communication unit. An image processing method executed by an image processing apparatus,
Using a vanishing point of a forward image taken from a moving moving object or a point near the vanishing point as a reference point, a display object indicating a relative positional relationship between the reference point and a feature in the front image is generated. Generation process;
An output step of outputting a front image including the display object generated by the generation step ;
Run
In the generation step, the display object is generated so as to include a line segment located on a half line passing through the feature object with the reference point as a starting point.   An image processing program for causing an image processing apparatus to execute the image processing method according to claim 9. 11. A recording medium in which the image processing program according to claim 10 is stored so as to be readable by the image processing apparatus.

Images