JP6022907B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

  2. Description of the Related Art Conventionally, there is known a parking frame detection device that captures the surroundings of a vehicle with a camera attached to the vehicle, extracts an edge from a captured image of the camera, and detects a line of the parking frame using the extracted edge (patent) Reference 1 and Patent Document 2).

JP 2012-118656 A JP 2012-035723 A

  In the parking frame detection devices disclosed in Patent Literature 1 and Patent Literature 2, since the edge is directly extracted from the captured image of the camera, the edge can be extracted with the resolution of the captured image of the camera, but there is a problem that the processing load increases. there were.

(1) The image processing apparatus according to the first aspect of the present invention is generated by an overhead image generation unit that generates an overhead image around the vehicle and an overhead image generation unit based on a captured image obtained by capturing the vehicle periphery. Based on the first straight line extraction unit that extracts a straight line from the overhead image and the straight line extracted by the first straight line extraction unit, a search area for searching for a straight line in a captured image obtained by photographing the periphery of the vehicle is set. A search area setting unit, and a second straight line extraction unit that extracts a straight line from a captured image obtained by photographing the periphery of the vehicle using the search area set by the search area setting unit.
(2) In the image processing method according to the eighth aspect of the present invention, an overhead image around the vehicle is generated based on a photographed image obtained by photographing the surroundings of the vehicle, a straight line is extracted from the overhead image, and the extracted straight line is used. A search area for searching for a straight line is set in a photographed image obtained by photographing the periphery of the vehicle, and a straight line is extracted from the photographed image obtained by photographing the periphery of the vehicle using the search area.
(3) The image processing program according to the ninth aspect of the invention is generated by an overhead image generation process for generating an overhead image around the vehicle and an overhead image generation process based on a captured image obtained by capturing the vehicle periphery. Based on the first straight line extraction process for extracting a straight line from the overhead image and the straight line extracted by the first straight line extraction process, a search area for searching for a straight line in a captured image obtained by photographing the periphery of the vehicle is set. A search area setting process and a second straight line extraction process for extracting a straight line from a captured image obtained by photographing the periphery of the vehicle using the search area set by the search area setting process, To do.

  According to the present invention, it is possible to extract a straight line drawn on a traveling surface of a vehicle while suppressing a processing load.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. 4 is a flowchart for explaining a processing flow of the image processing apparatus according to the first embodiment. It is a figure explaining the production | generation process of a bird's-eye view image. (A) shows an example of a bird's-eye view image, and (b) shows an example of a photographed image of the camera. (A) shows an example of the search line set in the coordinate system of the overhead image in the first embodiment, and (b) shows an example in which the search line of (a) is projected on the captured image of the camera. (A) shows an example of a luminance profile, and (b) shows a differential result of the luminance profile of (a). An example of the edge detection result is shown. 10 is a flowchart illustrating a processing flow of the image processing apparatus according to the second embodiment. (A) shows an example of the search line set in the coordinate system of the bird's-eye view image in the second embodiment, and (b) shows an example in which the search line of (a) is projected on the captured image of the camera. 10 is a flowchart illustrating a processing flow of an image processing apparatus according to a third embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of an image processing apparatus 10 according to the first embodiment. The image processing apparatus 10 is used by being mounted on a vehicle, and includes an overhead image generation unit 11, a first straight line extraction unit 12, a search area setting unit 13, a second straight line extraction unit 14, And a straight line state recognition unit 15. The image processing apparatus 10 realizes the functions of these units by causing a computer (not shown) to execute a predetermined image processing program. The contents of the function of each part will be described in detail later.

  Four cameras 20 a to 20 d are connected to the image processing apparatus 10. The cameras 20a to 20d are electronic cameras that shoot around the vehicle in different shooting ranges, and are installed in various parts such as a vehicle body, a bumper, and a door mirror. The shooting range of each of these cameras is determined so as to cover the entire periphery of the vehicle. In this embodiment, the camera 20a shoots the shooting range in front of the vehicle, the camera 20b shoots the shooting range on the left side of the vehicle, the camera 20c shoots the shooting range on the right side of the vehicle, and the camera 20d shoots the shooting range on the rear side of the vehicle. Will be described. Captured images respectively acquired at predetermined frame rate intervals by the cameras 20 a to 20 d are output to the image processing apparatus 10.

  FIG. 2 is a flowchart showing the processing contents of the entire image processing apparatus 10. Here, an outline of each step will be described, and details of each step will be described in detail later. The processing shown in FIG. 2 is repeatedly executed at predetermined processing cycles, for example, when the vehicle speed is 0 to 40 km / h.

  In step S <b> 1, the bird's-eye view image generation unit 11 generates a bird's-eye view image (around view image) that shows a bird's-eye view of the entire periphery of the vehicle based on the captured images input from the cameras 20 a to 20 d. The overhead image generated by the overhead image generation unit 11 is output to the first straight line extraction unit 12.

  In the next step S2, the first straight line extraction unit 12 extracts a straight line drawn on the traveling surface of the vehicle from the overhead view image generated in step S1. A straight line drawn on the running surface of a vehicle means various road marking lines drawn on the road (roadway center line, lane boundary line, roadway outside line, pedestrian crossing, etc.) and parking frames drawn on the parking lot Such as a line. The straight line extraction result by the first straight line extraction unit 12 is output to the search region setting unit 13.

  In the next step S3, the search area setting unit 13 sets a search area for searching for a straight line drawn on the traveling surface of the vehicle in the captured image in front of the vehicle by the camera 20a. The search area is set based on a straight line extraction result by the first straight line extraction unit 12. In the present embodiment, the search area is set so that a straight line (hereinafter referred to as a vertical line) extending in the front-rear direction of the vehicle is extracted. Note that the vertical line includes not only a line completely parallel to the front-rear direction of the vehicle but also a line oblique to the front-rear direction of the vehicle. The search area setting result by the search area setting unit 13 is output to the second straight line extraction unit 14.

  In the next step S4, the second straight line extraction unit 14 extracts an edge from the search area set in step S3 in the captured image in front of the vehicle by the camera 20a, and displays a vertical line drawn on the traveling surface of the vehicle. Extract. The result of vertical line extraction by the second straight line extraction unit 14 is output to the straight line state recognition unit 15.

  In the next step S5, the straight line state recognition unit 15 recognizes the state of the vertical line (length, width, and multiplicity) extracted in step S4. The multiplicity indicates how many lines the straight line is (for example, a single line, a double line, a triple line,...). The recognition result of the state of the vertical line by the straight line state recognition unit 15 is output to, for example, a vehicle control device (not shown) connected to the image processing device 10 and used for vehicle parking assistance, travel control, and the like.

  Next, the content of the process in step S1 described above will be described with a specific example. FIG. 3 is a diagram illustrating an example of the captured images 21a to 21d captured by the cameras 20a to 20d, and an example of the overhead image 23 generated by the captured images 21a to 21d. The overhead image generation unit 11 creates four image regions 22a to 22d corresponding to the front, left side, right side, and rear of the vehicle 24 based on the captured images 21a to 21d respectively captured by the cameras 20a to 20d. To do. Then, the overhead image generation unit 11 generates the overhead image 23 by connecting the four image regions 22a to 22d.

  Moreover, the content of the processing of steps S2 to S5 described above will be described with another specific example. FIG. 4A shows an example of the bird's-eye view image 30, and FIG. 4B shows an example of an image 31 taken in front of the vehicle by the camera 20 a used for generating the bird's-eye view image 30. In the captured image 31 of the camera 20a shown in FIG. 4B, two vertical lines 32 and 33 are shown, and in the overhead image 30 shown in FIG. 4A, the two vertical lines 32, A part of 33 is included.

  In step S <b> 2 described above, the first straight line extraction unit 12 performs an edge extraction process for extracting an edge from the overhead image 30. An edge is a point where the luminance value changes abruptly in an image. In this edge extraction process, the bird's-eye view image 30 is searched in the horizontal direction (the left-right direction of the vehicle), and a point whose luminance value change is equal to or greater than a predetermined threshold is extracted as an edge. The first straight line extraction unit 12 extracts vertical lines 32 and 33 from the overhead image 30 based on the extracted positional relationship of the edges.

  In step S3 described above, the search area setting unit 13 is a virtual image obtained by extending the vertical lines 32 and 33 extracted in step S2 to the front of the vehicle in the coordinate system of the overhead image 30 as shown in FIG. Vertical lines 36 and 37 are set. The virtual vertical lines 36 and 37 are set to have a predetermined length from the start points (end points on the vehicle side) of the vertical lines 32 and 33 extracted in step S2 and extend beyond the shooting range of the overhead image 30. . Then, the search area setting unit 13 sets search lines 38a to 38f and 39a to 39f having predetermined lengths in the virtual vertical lines 36 and 37, which are orthogonal to the virtual vertical lines 36 and 37, respectively. The search lines 38a to 38f and 39a to 39f are set at predetermined intervals in the direction of the virtual vertical lines 36 and 37, and the virtual vertical lines 36 and 37 pass through the centers of the search lines 38a to 38f and 39a to 39f. Is set as follows.

  Then, as shown in FIG. 5B, the search area setting unit 13 uses the coordinate values of the search lines 38a to 38f and 39a to 39f set in the coordinate system of the overhead image 30 as the coordinates of the captured image 31 of the camera 20a. Convert to coordinate values in the system. As a result, the search lines 38a to 38f and 39a to 39f are projected on the captured image 31 of the camera 20a. That is, search lines 38a to 38f and 39a to 39f are set in the captured image 31 of the camera 20a. In this description, the search lines 38a to 38f and 39a to 39f set in the captured image of the camera 20a are collectively referred to as a search area.

  In step S4 mentioned above, the 2nd straight line extraction part 14 produces | generates the brightness | luminance profile showing the change of the brightness | luminance value on each search line set in step S3 in the picked-up image of the camera 20a. This luminance profile is generated by the Bresenham algorithm. FIG. 6A is a diagram showing an example of this luminance profile. The second straight line extraction unit 14 differentiates the generated luminance profile, and extracts an edge from the captured image of the camera 20a based on the differentiation result. FIG. 6B is a diagram showing a differentiation result of the luminance profile shown in FIG. The differential result shown in FIG. 6 (b) comes from the point where the luminance value suddenly changes from dark to light, that is, the peak 40 corresponding to the edge entering the straight line, and the point where the luminance value changes suddenly from light to dark, that is, from the straight line. A peak 41 corresponding to the edge is included. The interval from the edge corresponding to the peak 40 to the edge corresponding to the peak 41 corresponds to the width of the straight line drawn on the traveling surface of the vehicle. The second straight line extraction unit 14 finds the peak 40 and the peak 41, thereby extracting a straight edge drawn on the traveling surface of the vehicle.

  And the 2nd straight line extraction part 14 extracts a straight line from the picked-up image of the camera 20a based on the positional relationship of the extracted edge. FIG. 7 is a diagram illustrating an edge extraction result. The second straight line extraction unit 14 uses the combination of the line connecting the edges 42a to 42e and the line connecting the edges 43a to 43e to the vertical line 32 shown in the photographed image of the camera 20a shown in FIG. To extract. Further, the second straight line extraction unit 14 uses the combination of the line connecting the edges 44a to 44f and the line connecting the edges 45a to 45f to the vertical image captured in the photographed image of the camera 20a shown in FIG. Line 33 is extracted. Note that the second straight line extraction unit 14 can extract a vertical line by connecting surrounding edges even if a part of the edge constituting the vertical line is not extracted. For example, even when the edge 44c is not extracted, the second straight line extraction unit 14 can extract the vertical line by connecting the edges 44a, 44b, 44d, 44e, and 44f.

  In step S5 described above, the straight line state recognition unit 15 recognizes the length, width, and multiplicity of each vertical line 32, 33 extracted by the second straight line extraction unit 14. The length of the vertical line 32 is obtained using, for example, the distance from the edge 42a to the edge 42e. Further, the width of the vertical line 32 is obtained by using, for example, the distance from the edge 42a to the edge 43a. Further, the multiplicity of the vertical line 32 is obtained by using, for example, the number of edges detected in one search line. For example, if the number of edges detected in one search line is two, it is determined as a single line, and if it is four, it is determined as a double line.

  The recognition result of the state of the vertical line by the straight line state recognition unit 15 is output to, for example, a vehicle control device (not shown). In the vehicle control device, for example, the type of the vertical line (for example, whether it is a road marking line or a parking frame line) can be determined based on the recognition information of the vertical line width and multiplicity. Moreover, when the road marking line extended ahead of a vehicle is interrupted on the way, it is assumed that the stop line exists ahead. Therefore, the vehicle control device can determine that there is a stop line ahead if the length of the vertical line is shorter than a predetermined value based on the recognition information of the length of the vertical line. And the vehicle control apparatus can perform parking assistance of a vehicle, traveling control, etc. using such determination information of a vertical line. For example, when it is determined that the vertical line is a parking frame line, the vehicle control device prohibits rapid acceleration of the vehicle even if a sudden accelerator operation is performed due to a mistake in stepping on the brake pedal and the accelerator pedal during parking. Such traveling control is performed.

According to the first embodiment described above, the following operational effects can be obtained.
(1) In the image processing apparatus 10, the overhead image generation unit 11 generates an overhead image around the vehicle based on the captured images obtained by capturing the surroundings of the vehicle with the cameras 20 a to 20 d. The first straight line extraction unit 12 extracts a straight line from the overhead image generated by the overhead image generation unit 11. The search area setting unit 13 sets a search area (search line) for searching for a straight line in a photographed image obtained by photographing the front of the vehicle with the camera 20a based on the straight line extracted by the first straight line extraction unit 12. . The second straight line extraction unit 14 extracts a straight line from the photographed image obtained by photographing the front of the vehicle with the camera 20a using the search area set by the search area setting unit 13. As described above, the image processing apparatus 10 specifies an area for searching for a straight line in the captured image of the camera 20a based on the straight line extracted from the overhead image, and performs image processing for extracting the straight line in this search area. In addition, image processing for extracting a straight line from the entire region of the captured image of the camera 20a is not necessary. Therefore, since the image processing apparatus 10 can extract a straight line with the resolution of the captured image of the camera 20a while suppressing the processing load, the straight line drawn on the traveling surface of the vehicle can be accurately extracted.

(2) In the image processing apparatus 10, the straight line state recognition unit 15 recognizes the straight line state (length, width, and multiplicity) extracted by the second straight line extraction unit 14. Thereby, the image processing apparatus 10 can use the recognition information of the straight line state for vehicle parking assistance, travel control, and the like.

(3) In the image processing apparatus 10, the search area for searching for a straight line in the captured image of the camera 20a includes an area outside the range of the overhead image. The overhead image includes only a range in the vicinity of the vehicle (for example, about 2 to 3 m from the vehicle), but the captured image of the camera 20a includes a range farther than the overhead image. For this reason, the image processing apparatus 10 can extract a distant straight line that is not included in the overhead image from the captured image of the camera 20a.

(4) In the image processing apparatus 10, the first straight line extraction unit 12 extracts a vertical line from the overhead image generated by the overhead image generation unit 11. The search area setting unit 13 sets the search area so as to be orthogonal to the vertical line extracted by the first straight line extraction unit 12. The second straight line extraction unit 14 extracts an edge from the search region set by the search region setting unit 13 in the captured image obtained by photographing the front of the vehicle with the camera 20a, and extracts a straight line based on the edge extraction result. . Thereby, the image processing apparatus 10 can extract a vertical line (a line extending in the front-rear direction of the vehicle) from the captured image of the camera 20a.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the second embodiment, the configuration of the image processing apparatus 10 is the same as the configuration of the first embodiment (FIG. 1), and the description thereof is omitted.

  FIG. 8 is a flowchart showing the processing contents of the entire image processing apparatus 10 according to the second embodiment. The processing of steps S11 to S12 is the same as the processing of steps S1 to S2 of the first embodiment. That is, the overhead image generation unit 11 generates an overhead image based on each captured image input from the cameras 20a to 20d. The first straight line extraction unit 12 extracts a straight line drawn on the traveling surface of the vehicle from the generated overhead image.

  In the next step S13, the search area setting unit 13 sets a search area for searching for a straight line drawn on the traveling surface of the vehicle in the captured image in front of the vehicle by the camera 20a. In the second embodiment, unlike the first embodiment, the search area is set so that a straight line (hereinafter referred to as a horizontal line) extending in the left-right direction of the vehicle is extracted. The horizontal line includes not only a line completely parallel to the left-right direction of the vehicle but also a line oblique to the left-right direction of the vehicle.

  In the process of the next step S14, the second straight line extraction unit 14 extracts an edge from the search area set in step S13 in the captured image ahead of the vehicle by the camera 20a, and the horizontal line drawn on the traveling surface of the vehicle To extract.

  In the next step S15, the straight line state recognition unit 15 recognizes the horizontal line state (length and width) extracted in step S14. The recognition result of the horizontal line state by the straight line state recognition unit 15 is output to, for example, a vehicle control device (not shown) connected to the image processing device 10 and is used for vehicle parking assistance, travel control, and the like.

  Next, the content of the processing described above will be described with a specific example. FIG. 9A shows an example of the bird's-eye view image 50, and FIG. 9B shows an example of a photographed image 51 in front of the vehicle by the camera 20a used for generating the bird's-eye view image 50. In the captured image 51 of the camera 20a shown in FIG. 9B, three vertical lines 52 to 54 and one horizontal line 55 are shown, and these straight lines 52 to 55 are parking frame lines. And The bird's-eye view image 50 shown in FIG. 9A includes a part of the vertical lines 52 and 53 shown in the captured image 51 of the camera 20a, but does not include the vertical line 54 and the horizontal line 55.

  In step S12 mentioned above, the 1st straight line extraction part 12 searches the bird's-eye view image 50 to a horizontal direction (left-right direction of a vehicle) similarly to step S2 of 1st Embodiment, and extracts an edge. And the 1st straight line extraction part 12 extracts the vertical lines 52 and 53 from the bird's-eye view image 30 based on the positional relationship of the extracted edge.

  In step S13 described above, the search area setting unit 13 sets search lines 58a to 58o (search areas) in the coordinate system of the overhead image 50 as shown in FIG. The search lines 58a to 58o are set so that when the vertical lines 52 and 53 extracted from the bird's-eye view image 50 are the vertical lines of the parking frame, the horizontal line of the parking frame that is assumed to be ahead can be searched. The

  The parking frame is generally a frame of 5 m in length and 2.5 m in width for one vehicle. In the present embodiment, a parking frame model 59 is assumed that three parking frames are connected horizontally. Here, the end points of the vertical lines 52 and 53 extracted from the overhead image 50 and the end points on the near side of the two middle vertical lines of the parking frame model 59 are combined to form the vertical lines 52 and 53 and the parking frame model 59. Assuming a state where In this state, search lines 58a to 58o are set so that the horizontal line of the parking frame model 59 is extracted. The search lines 58a to 58o are set with a certain width so that a parking frame having a size different from that of the parking frame model 59 can be handled.

  Based on the above, in the present embodiment, the search lines 58a to 58o are specifically set as follows. The numerical value described in the description here is not the length of the overhead image 50 in the coordinate system, but the actual length. The search lines 58a to 58o are set to a length of 4 m in parallel with the vertical lines 52 and 53 extracted from the overhead image 50. Further, in the search lines 58a to 58o, a position 2m away from the vehicle-side end point of the vertical lines 52, 53 is a starting point, and a position 6m away from the vehicle-side end point of the vertical lines 52, 53 is 6m away from the vehicle side. Is the end point. The search lines 58a to 58o are within a range of 2.5 m to the left from the left vertical line 52 and within a range of 2.5 m to the right from the right vertical line 53 in the direction orthogonal to the vertical lines 52 and 53. And within the range from the left vertical line 56 to the right vertical line 57, it is set at predetermined intervals. Since the shooting range of the bird's-eye view image 50 is in the vicinity of the vehicle (for example, 2 to 3 m), the search lines 58a to 58o are set so as to protrude outside the shooting range of the bird's-eye view image 50.

  In this manner, when the search area setting unit 13 sets the search lines 58a to 58o in the coordinate system of the overhead image 50, as shown in FIG. 9B, the search area setting unit 13 sets the coordinate values of the search lines 58a to 58o to the camera 20a. Are converted into coordinate values in the coordinate system of the captured image 51. Thereby, the search lines 58f to 58o are projected on the captured image 51 of the camera 20a. In FIG. 9B, the search lines 58a to 58e are not projected because they are in a range not included in the captured image 51 of the camera 20a. In this way, the search lines 58f to 58o are set in the captured image 51 of the camera 20a.

  In step S14 mentioned above, the 2nd straight line extraction part 14 extracts an edge from each search line 58f-58o set in step S13 in the picked-up image 51 of the camera 20a similarly to 1st Embodiment. To do. Then, the second straight line extraction unit 14 extracts the horizontal line 55 from the photographed image of the camera 20a based on the extracted positional relationship of the edges. As described above, the image processing apparatus 10 according to the second embodiment can extract the horizontal line 55 that is not included in the overhead image 50.

  In step S15 described above, the straight line state recognition unit 15 recognizes the state (length / width) of the horizontal line extracted in step S14, and outputs the recognition result to, for example, a vehicle control device (not shown). In the vehicle control device, for example, the type of the horizontal line (for example, whether it is a road marking line or a parking frame line) can be determined based on the recognition information of the width of the horizontal line.

According to the second embodiment described above, the following operational effects can be obtained.
In the image processing apparatus 10, the first straight line extraction unit 12 extracts a vertical line from the overhead image generated by the overhead image generation unit 11. The search area setting unit 13 sets the search area so as to be parallel to the vertical line extracted by the first straight line extraction unit 12. The second straight line extraction unit 14 extracts an edge from the search region set by the search region setting unit 13 in the captured image obtained by photographing the front of the vehicle with the camera 20a, and extracts a straight line based on the edge extraction result. . Thereby, the image processing apparatus 10 can extract a horizontal line (a line extending in the left-right direction of the vehicle) from the captured image of the camera 20a while suppressing a processing load.

(Third embodiment)
Next, a third embodiment of the present invention will be described. Also in the third embodiment, the configuration of the image processing apparatus 10 is the same as the configuration of the first embodiment (FIG. 1), and thus the description thereof is omitted.

  FIG. 10 is a flowchart showing the processing contents of the entire image processing apparatus 10 according to the third embodiment. The processing of steps S21 to S22 is the same as the processing of steps S1 to S2 of the first embodiment. That is, the overhead image generation unit 11 generates an overhead image based on each captured image input from the cameras 20a to 20d. The first straight line extraction unit 12 extracts a straight line drawn on the traveling surface of the vehicle from the generated overhead image.

  The processes of the next steps S23 to S25 and the processes of steps S26 to S28 are executed in parallel, for example. The processes in steps S23 to S25 are the same as the processes in steps S3 to S5 of the first embodiment. That is, the search area setting unit 13 sets a search area (search line) for extracting a vertical line drawn on the traveling surface of the vehicle in the captured image of the camera 20a. The second straight line extraction unit 14 extracts an edge on the set search line and extracts a vertical line in the captured image of the camera 20a. The straight line state recognition unit 15 recognizes the state (length, width, and multiplicity) of the extracted vertical line.

  Moreover, the process of step S26-S28 is the same as the process of step S13-S15 of 2nd Embodiment. That is, the search area setting unit 13 sets a search area (search line) for extracting a horizontal line drawn on the traveling surface of the vehicle in the captured image of the camera 20a. The second straight line extraction unit 14 extracts an edge on the set search line and extracts a horizontal line in the captured image of the camera 20a. The straight line state recognition unit 15 recognizes the state (length and width) of the extracted vertical line.

  Thus, in the third embodiment, both vertical lines and horizontal lines are extracted from the captured image of the camera 20a. Although FIG. 10 shows an example in which the process of extracting a vertical line and the process of extracting a horizontal line are performed in parallel, either one may be performed first.

  In step S29, the straight line state recognition unit 15 recognizes the intersection angle between the vertical line and the horizontal line extracted from the captured image of the camera 20a. The intersection angle between the vertical line and the horizontal line is calculated, for example, by converting the vertical line and the horizontal line extracted from the captured image of the camera 20a into coordinate values in the coordinate system of the overhead image.

  The recognition result of the intersection angle between the vertical line and the horizontal line by the straight line state recognition unit 15 is output to, for example, a vehicle control device (not shown). For example, when the intersection angle between the vertical line and the horizontal line is approximately 90 degrees, it is assumed that the extracted vertical line and horizontal line are road marking lines or parking frame lines. However, when the crossing angle between the vertical line and the horizontal line is not approximately 90 degrees, it is assumed that the extracted vertical line and horizontal line are not road marking lines or parking frame lines. Therefore, the vehicle control device can determine whether or not the extracted vertical and horizontal lines are road marking lines or parking frame lines based on information on the intersection angle between the vertical and horizontal lines.

According to the third embodiment described above, the following operational effects can be obtained.
In the image processing apparatus 10, the first straight line extraction unit 12 extracts a vertical line from the overhead image generated by the overhead image generation unit 11. The search area setting unit 13 includes a search area (first search area) orthogonal to the vertical line extracted by the first straight line extraction unit 12 and a search area (second search area) parallel to the vertical line. Search area). The second straight line extraction unit 14 extracts an edge from the first search region in the captured image obtained by photographing the front of the vehicle with the camera 20a, and extracts a vertical line based on the edge extraction result. Further, the second straight line extraction unit 14 extracts an edge from the second search region in the captured image obtained by photographing the front of the vehicle with the camera 20a, and extracts a horizontal line based on the edge extraction result. Thereby, the image processing apparatus 10 can extract both the vertical line and the horizontal line from the captured image of the camera 20a while suppressing the processing load.

(Modification 1)
In the above-described embodiment, the example in which the four cameras 20a to 20d capture each of the above-described shooting ranges has been described. However, the number of cameras mounted on the vehicle and the shooting range are not limited thereto. In addition, the shooting range including the cameras may not necessarily cover the entire periphery of the vehicle. If the surroundings of the vehicle can be photographed within an appropriate range, a photographed image can be acquired for an arbitrary photographing range using an arbitrary number of cameras.

(Modification 2)
In the above-described embodiment, the example in which the captured images from the four cameras 20a to 20d are input to the image processing apparatus 10 has been described. However, the captured images of all the cameras 20a to 20d may not be input. An image captured by any one of the cameras may be input to the image processing apparatus 10. In this case, the overhead image generation unit 11 generates an overhead image only with the input captured image of the camera.

(Modification 3)
In the above-described embodiment, the example in which the straight line drawn on the traveling surface of the vehicle is extracted from the captured image in front of the vehicle by the camera 20a has been described. However, the captured image from which the straight line is extracted is not limited thereto. A straight line may be extracted from the captured images of the left side, the right side, and the rear of the vehicle by the cameras 20b to 20d in the same manner as in the above-described embodiment.

  As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiments, and other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. .

  DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus, 11 ... Overhead image production | generation part, 12 ... 1st straight line extraction part, 13 ... Search area setting part, 14 ... 2nd straight line extraction part, 15 ... Straight line state recognition part, 20a-20d ... Camera

Claims (9)

An overhead image generation unit that generates an overhead image around the vehicle based on a captured image of the surroundings of the vehicle;
A first straight line extraction unit that extracts a straight line from the overhead image generated by the overhead image generation unit;
Based on the straight line extracted by the first straight line extraction unit, a search region setting unit that sets a search region for searching for a straight line in a captured image obtained by photographing the periphery of the vehicle;
A second straight line extraction unit that extracts a straight line from a photographed image obtained by photographing the periphery of the vehicle using a search region set by the search region setting unit;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
An image processing apparatus, further comprising: a straight line state recognition unit for recognizing a straight line state extracted by the second straight line extraction unit. The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the search area includes an area outside the shooting range of the overhead image. In the image processing device according to any one of claims 1 to 3,
The search area setting unit sets the search area to be orthogonal to the straight line extracted by the first straight line extraction unit;
The second straight line extraction unit extracts an edge from a search region set by the search region setting unit in a captured image obtained by photographing the periphery of the vehicle, and extracts a straight line based on the extraction result of the edge. An image processing apparatus. In the image processing device according to any one of claims 1 to 3,
The search area setting unit sets the search area to be parallel to the straight line extracted by the first straight line extraction unit,
The second straight line extraction unit extracts an edge from a search region set by the search region setting unit in a captured image obtained by photographing the periphery of the vehicle, and extracts a straight line based on the extraction result of the edge. An image processing apparatus. In the image processing device according to any one of claims 1 to 3,
The search area setting unit is parallel to the first search area orthogonal to the straight line extracted by the first straight line extraction unit and the straight line extracted by the first straight line extraction unit as the search area. And a second search area that is
The second straight line extraction unit extracts an edge from the first search region of a captured image obtained by photographing the vehicle periphery, extracts a first straight line based on the extraction result of the edge, and An image processing apparatus, wherein an edge is extracted from the second search region of a photographed image obtained by photographing and a second straight line is extracted based on the extraction result of the edge. The image processing apparatus according to claim 6.
An image processing apparatus, further comprising: a crossing angle recognition unit that recognizes a crossing angle between the first straight line and the second straight line extracted by the second straight line extraction unit.   A bird's-eye view image around the vehicle is generated based on a photographed image of the vehicle periphery, a straight line is extracted from the bird's-eye view image, and a straight line is searched for in the photographed image of the vehicle periphery based on the extracted straight line. An image processing method characterized in that a search area for setting is set and a straight line is extracted from a captured image obtained by capturing the periphery of the vehicle using the search area. An overhead image generation process for generating an overhead image around the vehicle based on a photographed image of the surroundings of the vehicle;
A first straight line extraction process for extracting a straight line from the overhead image generated by the overhead image generation process;
Based on the straight line extracted by the first straight line extraction process, a search area setting process for setting a search area for searching for a straight line in a captured image obtained by photographing the periphery of the vehicle;
A second straight line extraction process for extracting a straight line from the photographed image obtained by photographing the periphery of the vehicle using the search area set by the search area setting process;
An image processing program for causing a computer to execute.

Images