JP2014107696A - On-vehicle image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct information related to a parking frame, the information provided to a vehicle and the driver.SOLUTION: A side-direction approach determination section 311 is configured to determine whether or not a near-side white-line end portion 23LE detected by a first white-line detection section 301 reaches a border 28 between an approach direction bird-view image 22e and side-direction bird-view images 22b, 22c. Further, a second white-line detection section 303 is configured to detect a white line from the side-direction bird-view images 22b, 22c when the side-direction approach determination section 311 determines that the near-side white-line end portion 23LE has reached the border 28.

Description

  The present invention relates to an in-vehicle image processing apparatus.

  A parking frame line is recognized from an image obtained by a camera attached to the rear part of the vehicle, and a frame line obtained by virtually extending the parking frame line is displayed so that the driver can make the vehicle parallel to the parking frame line. An apparatus for assisting parking is known (see Patent Document 1).

JP 2011-16405 A

  However, in the apparatus described in the above-described patent document, the length of the parking frame line photographed by the camera gradually decreases as the vehicle enters the parking frame. Therefore, there is a possibility that the angle of the virtually extending frame line is deviated from the parking frame line.

(1) The in-vehicle image processing apparatus according to the invention of claim 1 is an image obtained by photographing with a traveling direction detection unit that detects a traveling direction of the vehicle and cameras provided at the front, rear, and side of the vehicle. Among the acquired images, either one of an image obtained by photographing with a camera provided in front of the vehicle and an image obtained by photographing with a camera provided in the rear of the vehicle An image selection unit that selects an image corresponding to the traveling direction of the vehicle detected by the traveling direction detection unit from the image as a traveling direction image, and a traveling direction object that detects the characteristics of the object from the traveling direction image selected by the image selection unit Out of the object feature ends detected by the feature detection unit and the traveling direction object feature detection unit, the near-side object feature end that exists on the near side of the vehicle is photographed with a camera provided on the side of the vehicle. Lateral image obtained and the direction of travel The side approach determination unit that determines whether or not the boundary with the image has been reached and the side approach determination unit determine that the front object feature end has reached the boundary between the traveling direction image and the side image. Then, an object that detects an object using the feature of the object detected by the side object feature detection unit that detects the feature of the object from the side image, the traveling direction object feature detection unit, and the side object feature detection unit. A detection unit is provided.
(2) The in-vehicle image processing apparatus according to the invention of claim 2 acquires images obtained by photographing with cameras provided at the front, rear, and side of the vehicle, combines the acquired images, and performs an overhead view. Of the overhead image generated by the image composition unit that generates the image, the traveling direction detection unit that detects the traveling direction of the vehicle, and the image composition unit, the image obtained by photographing with the camera provided in front of the vehicle One of the front bird's-eye view image generated based on the image obtained by photographing the front bird's-eye image generated based on the image and the bird's-eye view image generated by the image synthesis unit with the camera provided behind the vehicle. From the overhead image, an image selection unit that selects an overhead image corresponding to the traveling direction of the vehicle detected by the traveling direction detection unit as a traveling direction overhead image, and a progress that detects a white line from the overhead image selected by the image selection unit Direction white line inspection Out of the overhead image generated by the image compositing unit, the front side white line end that is the end of the white line existing on the front side of the vehicle out of the outgoing line and the white line end detected by the traveling direction white line detection unit, A side approach determination unit that determines whether or not the boundary between the front overhead image generated based on the image obtained by photographing with a camera provided on the side of the vehicle and the traveling direction overhead image has been reached; When the side approach determination unit determines that the front white line end has reached the boundary between the front bird's-eye view image and the traveling direction bird's-eye view image, a side white line detection unit that detects a white line from the side bird's-eye view image is displayed. It is characterized by providing.
(3) The in-vehicle image processing apparatus according to the invention of claim 12 acquires images obtained by photographing with cameras provided at the front, rear, and side of the vehicle, and combines the acquired images to obtain an overhead view. Of the overhead image generated by the image composition unit that generates the image, the traveling direction detection unit that detects the traveling direction of the vehicle, and the image composition unit, the image obtained by photographing with the camera provided in front of the vehicle One of the front bird's-eye view image generated based on the image obtained by photographing the front bird's-eye image generated based on the image and the bird's-eye view image generated by the image synthesis unit with the camera provided behind the vehicle. An image selection unit that selects an overhead image corresponding to the traveling direction of the vehicle detected by the traveling direction detection unit from the overhead image as a traveling direction overhead image, and a white line that is detected from the traveling direction overhead image selected by the image selection unit. 1 white line inspection Based on the detection result of the exit portion and the first white line detection unit, the first parking frame detection unit for detecting the parking frame, and the end of the white line related to the parking frame detected by the first parking frame detection unit Out of the overhead images generated by the image composition unit, the image of the front white line end that is the end of the white line existing on the front side of the vehicle is an image obtained by photographing with a camera provided on the side of the vehicle. A side approach determination unit that determines whether or not the boundary between the front bird's-eye view image generated based on the traveling direction bird's-eye view image has been reached, and a side approach determination unit, where the front white line end is the front bird's-eye view image And a second white line detection unit that detects a white line from the side bird's-eye view image with reference to the detection result of the first parking frame detection unit. It is characterized by that.

  ADVANTAGE OF THE INVENTION According to this invention, the information regarding the parking frame provided with respect to a vehicle and a driver can be optimized.

It is a block diagram which shows the structure of the vehicle-mounted image processing apparatus by one embodiment of this invention. It is a figure which shows the example of the image of the picked-up image around the vehicle image | photographed with the camera mounted in the vehicle, and the bird's-eye view image synthesize | combined by the image synthetic | combination part. It is a control block diagram of the arithmetic processing which an arithmetic processing part performs. It is a flowchart of the process performed in the image selection part of an arithmetic processing part. It is a flowchart of the process performed in the 1st white line detection part of an arithmetic processing part. It is a flowchart of the process performed in the 1st parking frame detection part of a calculation process part. It is a schematic diagram which shows the example of the parking frame line which comprises a parking frame. It is a flowchart of the process performed in the side approach determination part of an arithmetic processing part. It is a figure which shows an example of the state as which the front bird's-eye view image was selected as a traveling direction bird's-eye view image. It is a flowchart of the process performed in the 2nd white line detection part of an arithmetic processing part. It is a figure which shows the bird's-eye view image which concerns on the process in a 2nd white line detection part. It is a figure explaining the process in a 2nd white line detection part. It is a flowchart of the process performed in the 2nd parking frame detection part of a calculation process part. It is a block diagram which shows the structure of the vehicle-mounted image processing apparatus by 2nd Embodiment of this invention. It is a control block diagram of the arithmetic processing which an arithmetic processing part performs. It is a flowchart of the process performed in the image selection part of an arithmetic processing part. It is a flowchart of the process performed in the 1st white line detection part of an arithmetic processing part. It is a schematic diagram which shows the example of the process performed in the 1st white line detection part of an arithmetic processing part. It is a schematic diagram which shows the example of the relationship of the process of the front camera and side camera in 2nd Embodiment of this invention. It is a flowchart of the process performed in the side approach determination part of an arithmetic processing part. It is a flowchart of the process performed in the 2nd white line detection part of an arithmetic processing part. It is a schematic diagram which shows the example of the process performed in the 2nd white line detection part of an arithmetic processing part. It is a schematic diagram which shows the example of the scene where the vehicle-mounted image processing apparatus by 3rd Embodiment of this invention operate | moves. It is a block diagram which shows the structure of the vehicle-mounted image processing apparatus by 3rd Embodiment of this invention. It is a control block diagram of the arithmetic processing which an arithmetic processing part performs. It is a schematic diagram which shows the example of the process performed in the 1st white line detection part of an arithmetic processing part. It is a flowchart of the process performed in the 1st lane detection part of an arithmetic processing part. It is a schematic diagram which shows the example of the process performed in the 2nd white line detection part of an arithmetic processing part. It is a flowchart of the process performed by the 2nd lane detection part of an arithmetic processing part.

[First embodiment]
With reference to FIGS. 1-13, 1st Embodiment of the vehicle-mounted image processing apparatus by this invention is described. FIG. 1 is a block diagram showing a configuration of an in-vehicle image processing apparatus 100 according to an embodiment of the present invention. An in-vehicle image processing apparatus 100 illustrated in FIG. 1 is used by being mounted on a vehicle 1 (see FIG. 2), and includes an image synthesis unit 2, an arithmetic processing unit 3, and an output unit 4. The in-vehicle image processing apparatus 100 realizes these functions by causing a computer (not shown) to execute a predetermined control program. FIG. 2 shows an example of images of captured images 21 a, 21 b, 21 c, 21 d around the vehicle captured by the cameras 1 a to 1 d mounted on the vehicle 1 and the overhead image 22 synthesized by the image composition unit 2. FIG.

  Cameras 1 a to 1 d mounted on the vehicle 1 are electronic cameras that capture images of the surroundings of the vehicle 1 in different shooting ranges, and are installed in various parts of the vehicle 1 such as a body, a bumper, and a door mirror. The shooting range of each of these cameras is determined so that the entire periphery of the vehicle 1 can be covered together. In the present embodiment, as shown in FIG. 2, the camera 1a has a shooting range in front of the vehicle 1, the camera 1b has a shooting range on the left side of the vehicle 1, the camera 1c has a shooting range on the right side of the vehicle 1, and the camera 1d has A description will be given assuming that the shooting range behind the vehicle 1 is shot. The captured images respectively acquired at predetermined frame rate intervals by the cameras 1 a to 1 d are output to the image composition unit 2.

  In the present embodiment, the four cameras 1a to 1d are described as capturing the respective shooting ranges as described above, but the number of cameras mounted on the vehicle 1 and the shooting range are not limited thereto. In addition, the shooting range including the cameras may not necessarily cover the entire periphery of the vehicle 1. If the periphery of the vehicle 1 can be photographed within an appropriate range, a photographed image can be acquired for any photographing range using any number of cameras.

  In FIG. 2, images 21a, 21b, 21c, and 21d on the left side of the page indicate captured images of the surroundings of the vehicle 1 obtained by shooting with cameras 1a to 1d mounted on the vehicle 1, and images 22 ( 22a, 22b, 22c, and 22d) show overhead images (overhead view images).

  The image composition unit 2 synthesizes the bird's-eye view image 22 that shows a bird's-eye view of the entire periphery of the vehicle based on the captured images 21a, 21b, 21c, and 21d acquired by the cameras 1a to 1d. The bird's-eye view image 22 is synthesized by connecting the photographed images 21a, 21b, 21c, and 21d of the cameras 1a to 1d after coordinate conversion according to the photographing direction. The bird's-eye view image synthesized by the image synthesis unit 2 is output to the arithmetic processing unit 3.

  The captured image 21a in the camera 1a, that is, the overhead image 22a generated from the captured image 21a in front of the vehicle 1 is referred to as a forward overhead image 22a. Similarly, the bird's-eye view image 22d generated from the captured image 21d behind the vehicle 1 is referred to as a rear bird's-eye view image 22d, and the bird's-eye view images 22b and 22c generated from the captured images 21b and 21c on the side of the vehicle 1 are respectively side-viewed images 22b. , 22c.

  The bird's-eye view image 22 is displayed on the display device 1M installed in the vehicle 1 on the vehicle 1 side, for example. The region (image region 22F) in which the overhead image 22 is displayed is composed of four image regions 22Fa, 22Fb, 22Fc, and 22Fd corresponding to the front, left side, right side, and rear of the vehicle 1, respectively. As described above, the images 22a, 22b, 22c, and 22d displayed in these image areas 22F are created based on images taken by the cameras 1a to 1d in FIG. In each of the image areas 22Fa, 22Fb, 22Fc, and 22Fd, a part of the parking frame line 23L constituting the parking frame is displayed.

  The arithmetic processing unit 3 recognizes a parking frame around the vehicle by performing a predetermined arithmetic processing based on the overhead image 22 synthesized by the image synthesizing unit 2. The contents of the arithmetic processing performed by the arithmetic processing unit 3 will be described in detail later. The recognition result of the parking frame by the arithmetic processing unit 3 is output to the output unit 4.

  The output unit 4 outputs parking frame information for parking frames around the vehicle based on the recognition result of the parking frame by the arithmetic processing unit 3. For example, information indicating the direction of the parking frame relative to the vehicle 1 and the distance to the parking frame is output as parking frame information. This parking frame information is output to a host vehicle control device (not shown) connected to the in-vehicle image processing device 100, and is used for parking support and travel control of the vehicle 1. For example, it automatically recognizes that there is a parking lot around it, and, for example, when it is a parking lot environment, it automatically switches and displays a bird's-eye view around the vehicle on the display device 1M described later. Can do. As a result, it is possible to suppress a situation in which it is erroneously detected as a parking lot on a public road, and to switch the video presented to the user at an appropriate timing.

  The output unit 4 outputs parking frame information as relative position information from the own vehicle based on the recognition result of the parking frame by the arithmetic processing unit 3. For example, the position information of the parking frame in the actual environment such as the end point coordinates of the left and right frame lines of the parking frame, the angle and intercept of the parking frame line is output. This parking frame information is output to a host vehicle control device (not shown) connected to the on-vehicle parking frame recognition device, and is used for vehicle parking assistance, travel control, and the like. For example, by calculating the travel route to the parking frame based on the relative position and orientation from the vehicle to the parking frame, and notifying the driver of the timing of brake and shift position change and the amount of steering angle operation, Parking assistance can be performed. As a result, a parking operation can be completed in a short time even for a driver who is unfamiliar with driving operations such as garage entry.

  Furthermore, the travel route to the parking frame is calculated based on the relative position and orientation from the own vehicle to the parking frame, the control amount of the vehicle forward / backward / turning is automatically calculated, and the vehicle is calculated according to the calculation result. The movement may be automatically controlled. This makes it possible to complete the parking operation safely and accurately even for a driver who is unfamiliar with driving operations such as garage entry.

  As described above, the image described above is displayed on the display device 1M in the vehicle 1, for example. The image displayed on the display device 1M may be a captured image 21a before being converted into the overhead image 22. In this case, it is preferable that the image displayed on the display device 1 </ b> M is switched according to the traveling direction of the vehicle 1. For example, when the shift lever of the vehicle 1 is switched to the forward direction, it may be determined that the vehicle 1 is moving forward, and the captured image 21a in front of the vehicle captured by the camera 1a may be displayed. On the other hand, when the shift lever of the vehicle 1 is switched to the reverse direction, it may be determined that the vehicle 1 is moving backward, and a captured image 21d behind the vehicle 1 captured by the camera 1d may be displayed. .

  Next, the contents of the arithmetic processing performed by the arithmetic processing unit 3 will be described. FIG. 3 is a control block diagram of the arithmetic processing performed by the arithmetic processing unit 3. As illustrated in FIG. 3, the arithmetic processing unit 3 includes an image selection unit 310, a first white line detection unit 301, a first parking frame detection unit 302, a side approach determination unit 311, and a second white line detection unit 303. And each control block of the 2nd parking frame detection part 304 is functionally provided. In the arithmetic processing unit 3, for example, each control block of FIG. 3 is realized by executing a program recorded in the memory corresponding to each of these control blocks by a microcomputer. The arithmetic processing unit 3 performs processing when the vehicle speed becomes equal to or lower than a predetermined speed (for example, 50 km / h), for example. The determination as to whether or not the vehicle speed is equal to or lower than a predetermined speed may be made based on the cycle of the vehicle speed pulse output from the vehicle side, or based on information about the vehicle speed output from a control device (not shown) of the vehicle. You may judge.

  The overhead image synthesized by the image synthesis unit 2 is input to the image selection unit 310 in the arithmetic processing unit 3. The image selection unit 310 detects information related to the shift position of the host vehicle as information related to the traveling direction of the host vehicle, and displays an overhead image corresponding to the traveling direction (front or rear) corresponding to the detected shift position. And the rear bird's-eye view image 22d are selected and output to the first white line detection unit 301 as the traveling direction bird's-eye view image 22e.

  The first white line detection unit 301 detects a white line drawn on the road surface from the input traveling direction overhead image 22e. That is, the first white line detection unit 301 detects a white line extending on the traveling direction side of the vehicle 1. The first white line detection unit 301 is also referred to as a traveling direction white line detection unit 301.

  Details of processing performed by the first white line detection unit 301 will be described later. The determination as to whether or not the vehicle speed is equal to or lower than a predetermined speed may be made based on the cycle of the vehicle speed pulse output from the vehicle side, and information on the vehicle speed output from a control device (not shown) of the vehicle. You may judge based on. The result of white line detection (recognition) by the first white line detection unit 301 is output to the first parking frame detection unit 302.

  The first parking frame detection unit 302 detects a parking frame based on the white line extraction result by the first white line detection unit 301. Details of processing performed by the first parking frame detection unit 302 will be described later. The detection result of the parking frame by the first parking frame detection unit 302 is output to the side approach determination unit 311 and the output unit 4 of FIG.

  The side approach determination unit 311 is based on the processing performed by the first parking frame detection unit 302, and is on the near side of the vehicle 1 among the ends of the white line related to the parking frame detected by the first parking frame detection unit 302. It is determined whether or not the front white line end that is the end of the white line that has reached the boundary between the traveling direction overhead image 22e and the side overhead images 22b and 22c. The contents of processing performed by the side entry determination unit 311 will be described later.

  The second white line detection unit 303 is based on the processing contents performed by the first parking frame detection unit 302 and the side entry determination unit 311, and white lines (parking frame line 23 </ b> L) in the side-view images 22 b and 22 c on the side of the vehicle 1. Is calculated, and a parking frame line 23L at the calculated predicted position is detected. The second white line detection unit 303 is also referred to as a side white line detection unit 303. Details of processing performed by the second white line detection unit 303 will be described later. The result of white line detection (recognition) by the second white line detection unit 303 is output to the second parking frame detection unit 304.

  Based on the detection result of the white line by the first white line detection unit 301 and the detection result of the second white line detection unit 303, the second parking frame detection unit 304 has a parking frame line 23L that appears in the traveling direction overhead image 22e, A parking frame is detected from the parking frame line 23L appearing in the overhead images 22b and 22c. The detection result of the parking frame by the second parking frame detection unit 304 is output to the output unit 4 of FIG.

  FIG. 4 is a flowchart of processing performed by the image selection unit 310 of the arithmetic processing unit 3. The process shown in this flowchart is repeatedly executed every time the arithmetic processing unit 3 performs the process. In step S301, the information regarding the bird's-eye view image 22 and the shift position is acquired, and the process proceeds to step S302. In step S302, an overhead image corresponding to the traveling direction (front or rear) corresponding to the shift position acquired in step S301 is selected from either the front overhead image 22a or the rear overhead image 22d, and is used as the traveling direction overhead image 22e. It outputs to the 1st white line detection part 301, and returns.

  FIG. 5 is a flowchart of processing performed by the first white line detection unit 301 of the arithmetic processing unit 3. The process shown in this flowchart is repeatedly executed every time the arithmetic processing unit 3 performs the process. In step S101, the traveling direction overhead image 22e output from the image selection unit 310 is sequentially scanned (progressive scanning), and a horizontal edge filter is applied to all scanning lines (all lines). Proceed to

  In step S103, the rising and falling peaks of luminance are extracted from the processing result in step S101, and the process proceeds to step S105. In step S105, based on the result of the process in step S103, a process of leaving only edges that form a pair with a rising peak of luminance and a falling peak of luminance at a predetermined interval (interval corresponding to the horizontal width of the parking frame line). Go to step S107.

  In step S107, grouping (grouping) for associating peaks estimated to be linearly arranged between rising peaks and falling peaks with respect to the edges left by the processing in step S105. To go to step S109. In step S109, filtering is performed on each group of peaks obtained in the process in step S107 according to the length of each group in order to remove noise. That is, among the linearly arranged peaks constituting one group, only the group in which the interval between two peaks corresponding to both ends of the straight line is longer than a predetermined length is left. When the process in step S109 is executed, the process proceeds to step S111.

  In step S111, for each group left by the processing in step S109, two groups corresponding to both ends of a straight line in an arbitrary group of rising edges and a group of falling edges that form a pair with the rising edges. The coordinates of the end points (upper and lower ends) are extracted and output to the first parking frame detection unit 302.

  FIG. 6 is a flowchart of processing performed by the first parking frame detection unit 302 of the arithmetic processing unit 3. The process shown in this flowchart is repeatedly executed every time the arithmetic processing unit 3 performs the process. In step S121, based on the information output from the first white line detection unit 301, information on any two white lines (parking frame lines) is selected, and the process proceeds to step S123.

  In step S123, it is determined whether or not the angle difference in the extending direction between the two white lines selected in step S121 is equal to or smaller than a predetermined value (Thθmax). That is, in step S123, it is determined whether or not the two white lines are substantially parallel. If a positive determination is made in step S123, the process proceeds to step S125, and it is determined whether or not the interval between the two white lines is within a predetermined range (ThWmin or more and ThWmax or less). That is, in step S125, it is determined whether or not two white lines are arranged at an interval considered as two white lines constituting the parking frame.

  If an affirmative determination is made in step S125, the process proceeds to step S127, and it is determined whether or not the deviation between the lower ends of the two white lines is within a predetermined range (ThBmin or more and ThBmax or less). Here, the shift of the lower ends of the two white lines will be described with reference to FIG. As shown to the left figure (a) of FIG. 7, it parks so that the vehicle 1 may be parked in parallel or at right angles to the moving direction of the vehicle 1 in the parking lot site (vertical direction and horizontal direction on the paper surface of FIG. 7). When the frame 23 is provided and the parking frame line 23 </ b> L is drawn so as to correspond to the parking frame 23, the lower end on the paper surface does not shift.

  However, as shown in the right figure (b) of FIG. 7, depending on the parking lot, the vehicle 1 is not parallel or perpendicular to the moving direction of the vehicle 1 (vertical direction and horizontal direction on the paper surface of FIG. 7). A parking frame 23 may be provided so as to park, and a parking frame line 23 </ b> L may be drawn so as to correspond to the parking frame 23. In this case, as indicated by the distance B, the lower end position of the parking frame line 23L constituting the parking frame 23 is shifted. This is because the lower end position of the parking frame line 23L is shifted so that the parking frame line 23L on the right side of the drawing does not protrude from the area where the vehicle 1 travels below the page.

  For example, in a highway service area or parking area, a line segment perpendicular to the extending direction of the parking frame line 23L from the lower end of the parking frame line 23L on the left side in FIG. It is determined that the crossing angle θ with the line connecting the lower ends of 23L is 0 degree, 30 degrees, or 45 degrees. Moreover, the range of the value which can be taken also about the width W of the parking frame 23 is decided. Therefore, in the present embodiment, the value of the distance B that can be taken when the intersection angle θ is one of 0 degree, 30 degrees, and 45 degrees is stored in advance in a memory (not shown) or the like.

  In step S127 of the present embodiment, the above-mentioned distance B stored in advance is compared with the distance B calculated from the front bird's-eye view image 22a, and the difference is within the predetermined range (ThBmin or more and ThBmax). It is determined whether or not two white lines are white lines (parking frame line 23L) constituting one parking frame 23.

  If an affirmative determination is made in step S127, the process proceeds to step S129, and the coordinates of the four corners of the rectangular parking frame 23 composed of two white lines are registered as position information for one parking frame 23. If step S129 is performed, it will progress to step S131 and it will be determined whether the process mentioned above about arbitrary two white lines was performed regarding all the white lines (parking frame line) based on the information output from the 1st white line detection part 301. Check. If an affirmative determination is made in step S131, the result obtained by the above-described processing is output to the side entry determination unit 311 and the process returns. If a negative determination is made in step S131, the process returns to step S121.

  If a negative determination is made in step S123, a negative determination is made in step S125, or a negative determination is made in step S127, the process proceeds to step S131.

  FIG. 8 is a flowchart of processing performed by the side approach determination unit 311 of the arithmetic processing unit 3. The process shown in this flowchart is repeatedly executed every time the arithmetic processing unit 3 performs the process. In step S311, based on the coordinates of the four corners of the parking frame 23 registered in step S129 in FIG. 6 of the first parking frame detection unit 302 described above, the two parking frame lines 23L constituting the parking frame 23 are displayed. Whether or not the front white line end 23LE, which is the end of the white line existing on the front side of the vehicle 1 among the ends, has reached the boundary 28 between the traveling direction overhead image 22e and the side overhead images 22b and 22c. judge. For example, FIG. 9 shows an example of a state where the shift position of the vehicle 1 is D (drive), the vehicle 1 is determined to be moving forward, and the front overhead image 22a is selected as the traveling direction overhead image 22e. FIG. Note that FIG. 9 shows a state in which the front white line end portion 23LE does not reach the boundary 28.

  If at least one of the two near-side white line ends 23LE existing in the left-right direction in the drawing as the vehicle 1 travels reaches the boundary 28, an affirmative determination is made in step S311 and the process proceeds to step S312. In step S312, in the side view images 22b and 22c that are in contact with the boundary 28 that is determined that the front white line end portion 23LE has reached the boundary 28 in step S311 among the two boundaries 28 in the illustrated left-right direction. Then, an instruction is output to the second white line detection unit 303 so as to start the white line detection process, and the process returns.

  If both the front white line ends 23LE existing in the left-right direction in the figure do not reach the boundary 28, a negative determination is made in step S311 and the process returns.

  FIG. 10 is a flowchart of processing performed by the second white line detection unit 303 of the arithmetic processing unit 3. The process shown in this flowchart is repeatedly executed by the second white line detection unit 303 when, for example, an instruction to start the white line detection process is input from the side approach determination unit 311. Note that the parking frame line 23L displayed on the side overhead images 22b and 22c (for example, the parking frame line 23L appearing on the overhead images 22b and 22c in FIG. 11) by the processing performed by the second white line detection unit 303. Is detected.

  In step S141, interlaced scanning is performed on the side overhead images 22b and 22c to thin out the scanned lines. Such thinned-out scanning is performed, and a horizontal edge filter is applied to all the scanning lines, that is, the lines that are not thinned out (for example, one field obtained by interlaced scanning).

  As described above, by applying the edge filter in the horizontal direction only to one field, for example, obtained by interlaced scanning, it is possible to reduce the load of white line detection processing on the side overhead images 22b and 22c.

  In addition, in FIG. 12 mentioned later, the scanning line 26 which concerns on the interlace scanning mentioned above is shown typically. If step S141 is performed, it will progress to step S143.

  In step S143, the rise and fall peaks of the luminance are extracted from the processing result in step S141, and the process proceeds to step S145. In step S145, from the result of the process in step S143, a process of leaving only edges that are paired with a luminance rising peak and a luminance falling peak at a predetermined interval (an interval corresponding to the width of the parking frame line). Go to step S147.

  In step S147, the extended line of the parking frame line 23L is extended to the regions of the side overhead images 22b and 22c based on the position information about the parking frame 23 registered by the first parking frame detection unit 302. For example, as shown in FIG. 11, the parking frame line extending from the information about the parking frame line 23 </ b> L constituting the parking frame 23 recognized by the first parking frame detection unit 302 to the area of the side overhead images 22 b and 22 c. An extension line 25 of 23L is calculated. In the example illustrated in FIG. 11, the center line in the extending direction of the parking frame line 23 </ b> L is extended as the extension line 25.

  When step S147 is executed, the process proceeds to step S149, and the intersection of each line (each scanning line 26) related to interlaced scanning and the extension line 25 calculated in step S147 is calculated. That is, as shown in the bird's-eye view images 22b and 22c in FIG. 12, the intersection 27 of each scanning line 26 and the extension line 25 is calculated. Note that the extension line 25, the scanning line 26, and the intersection point 27 shown in FIG. 11 and FIG. 12 are only shown in the figure for convenience of explanation, and the image displayed on the display device 1M includes The extension line 25, the scanning line 26, and the intersection point 27 are not displayed.

  When step S149 is executed, the process proceeds to step S151. On each scanning line 26, in the vicinity of each intersection point 27 (within a predetermined range corresponding to the width direction of the parking frame line 23L), the rising peak of luminance and the rising edge of luminance are detected. It is confirmed whether or not there is an edge that forms a pair with the falling peak, and the process proceeds to step S153.

  In step S153, the intersection 27 is sequentially traced from the uppermost line, that is, the scanning line 26 in the upper part of FIG. 12, and the intersection 27 where the edge pair confirmed to exist in step S151 does not exist is identified and identified. The intersection 27 on the scanning line 26 immediately above the intersection 27 is set as the lower end of the white line, that is, the lower end of the parking frame line 23L, and the coordinates of the intersection 27 are extracted and output to the second parking frame detection unit 304.

  FIG. 13 is a flowchart of processing performed by the second parking frame detection unit 304 of the arithmetic processing unit 3. For example, when the white line detection process in the second white line detection unit 303 described above is performed, the process shown in this flowchart is executed in the second parking frame detection unit 304. In step S161, it is determined whether or not the above-described second white line detection unit 303 has performed white line detection. If a positive determination is made in step S161, the process proceeds to step S163.

  In step S163, the lower end of the white line detected by the first white line detection unit 301 is replaced with the lower end of the parking frame line 23L detected by the second white line detection unit 303, and the traveling direction overhead image 22e and the side overhead image 22b, It is determined that the white line extending over 22c is, for example, one parking frame line 23L on each of the right side and the left side in FIG. 12, and it is assumed that one parking frame 23 is configured by these two white lines. Information on the position of the frame 23 is registered. The registered information is output to the output unit 4 and the process returns.

  If a negative determination is made in step S161, the process returns.

According to the above-described embodiment, the following operational effects can be obtained.
(1) The side of the front white line end 23LE detected by the first white line detection unit 301 is determined so as to determine whether or not the boundary 28 between the traveling direction overhead image 22e and the side overhead images 22b and 22c has been reached. An approach determination unit 311 is configured. Then, when the side approach determination unit 311 determines that the front white line end 23LE has reached the boundary 28, the second white line detection unit 303 is configured to detect the white line from the side overhead images 22b and 22c. Configured. Accordingly, the parking frame line 23L can be correctly recognized even if the parking frame line 23L spans the traveling direction overhead image 22e and the side overhead images 22b and 22c, so that the parking frame provided to the vehicle 1 and the driver is provided. 23 information can be optimized.

(2) The second white line detection unit 303 is configured to detect a white line based on the information regarding the extending direction of the white line related to the parking frame 23 detected by the first parking frame detection unit 302. Thereby, since the position of the white line in the side overhead images 22b and 22c can be predicted, the white line in the side overhead images 22b and 22c can be detected even when the interlaced scanning is performed. Therefore, the load of the white line detection processing in the second white line detection unit 303 can be reduced, and the performance of the arithmetic device related to the second white line detection unit 303 does not need to be requested more than necessary. Can contribute to down.

(3) More specifically, a parking frame line extending from the information about the parking frame line 23L constituting the parking frame 23 recognized by the first parking frame detection unit 302 to the area of the side overhead images 22b and 22c. It was configured to calculate an extension line 25 of 23L. And it was comprised so that the intersection of the extended line 25 and each line (each scanning line 26) concerning an interlaced scan might be calculated. In addition, it is configured to check whether or not there is an edge formed as a pair by the rising peak of luminance and the falling peak of luminance near each intersection 27. This makes it possible to easily detect a white line that extends over the traveling direction bird's-eye view image 22e and the side bird's-eye view images 22b and 22c, thereby reducing the load of white line detection processing in the second white line detection unit 303. Since it is not necessary to request the performance of the arithmetic device related to the white line detection unit 303 more than necessary, it is possible to contribute to the cost reduction of the in-vehicle image processing device 100.

(4) The scanning of the row direction overhead image 22e by the first white line detection unit 301 is sequentially performed, and the scanning of the side overhead images 22b and 22c by the second white line detection unit 303 is performed by interlaced scanning. . That is, the scanning density of the side overhead images 22b and 22c by the second white line detection unit 303 is configured to be coarser than the scanning density of the row direction overhead image 22e by the first white line detection unit 301. Thereby, since the processing speed concerning a white line detection can be improved, a white line can be detected without delay, and the information regarding a parking frame can be output without delay. Therefore, the processing time on the vehicle 1 side using the information about the parking frame can be shortened, which contributes to the control of the vehicle 1 using the information about the parking frame.

[Second Embodiment]
Next, a second embodiment of the present invention will be described below with reference to FIGS.

  FIG. 14 is a block diagram illustrating the configuration of an external environment recognition apparatus 200 according to the second embodiment. An in-vehicle image processing device 200 shown in FIG. 14 is used by being mounted on the vehicle 1, and includes an arithmetic processing unit 31 and an output unit 4. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  What is characteristic in the present embodiment is that the image processing unit 31 directly performs image processing without using the image synthesis unit 2.

  The contents of the arithmetic processing performed by the arithmetic processing unit 31 will be described. FIG. 15 is a control block diagram of arithmetic processing performed by the arithmetic processing unit 31. As illustrated in FIG. 12, the arithmetic processing unit 31 includes an image selection unit 361, a first white line detection unit 311, a first parking frame detection unit 302, a side approach determination unit 362, and a second white line detection unit 313. And each control block of the 2nd parking frame detection part 304 is functionally provided. Among the images 21a to 21d acquired from the cameras 1a to 1d attached to the vehicle, the arithmetic processing unit 31 includes a front camera image 21a in the first white line detector 311 and a right camera image 21b in the second white line detector. A left-side camera image 21c is used. In the arithmetic processing unit 31, for example, each control block of FIG. 12 is realized by executing a program recorded in the memory corresponding to each of these control blocks by a microcomputer. The arithmetic processing unit 31 performs processing when the vehicle speed becomes equal to or lower than a predetermined speed (for example, 50 km / h), for example. The determination as to whether or not the vehicle speed is equal to or lower than a predetermined speed may be made based on the cycle of the vehicle speed pulse output from the vehicle side, or based on information about the vehicle speed output from a control device (not shown) of the vehicle. You may judge. The recognition result of the parking frame by the arithmetic processing unit 31 is output to the output unit 4.

  The front camera image 21 a and the rear camera image 21 d are input to the image selection unit 361 in the arithmetic processing unit 3. The image selection unit 361 detects information on the shift position of the host vehicle as information on the traveling direction of the host vehicle, and displays a camera image corresponding to the traveling direction (front or rear) corresponding to the detected shift position as the front camera image 21a. And either one of the rear camera image 21d and the rear camera image 21d are output to the first white line detection unit 311 as the traveling direction camera image 21e.

  The first white line detection unit 311 receives the front camera image 21a and detects a white line drawn on the road surface. Details of processing performed by the first white line detection unit 311 will be described later. The result of white line detection (recognition) by the first white line detection unit 311 is output to the first parking frame detection unit 302.

  The second white line detection unit 313 calculates the predicted position of the white line in the right-side camera image 21b and the left-side camera image 21c based on the processing content performed by the first parking frame detection unit 302, and parks at the calculated predicted position. Detect borders. Details of processing performed by the second white line detection unit 313 will be described later. The result of white line detection (recognition) by the second white line detection unit 313 is output to the side approach determination unit 362 and the output unit 4 in FIG.

  The side approach determination unit 362 is based on the processing performed by the first parking frame detection unit 302, and is on the near side of the vehicle 1 among the ends of the white lines related to the parking frame detected by the first parking frame detection unit 302. It is determined whether or not the front white line ends 41LaE and 41LbE, which are the ends of the white lines existing in, have reached the boundaries 46L and 46R between the traveling direction camera image 21e and the side camera images 21b and 21c. The processing content performed by the side approach determination unit 362 will be described later.

  FIG. 17 is a flowchart of processing performed by the first white line detection unit 311 of the arithmetic processing unit 31. The processing shown in this flowchart is repeatedly executed every time the arithmetic processing unit 31 performs processing. In step S201, a processing line is set for the traveling direction camera image 21e of the vehicle 1. An example of processing line setting will be described with reference to FIG. FIG. 18A is an example of the traveling direction camera image 21e. A processing line 42 is set for this image. FIG. 18B shows an example in which lines are equally applied to the entire screen. In addition to this, a method of setting based on the detection result of the past white line or parking frame, a method of setting based on the behavior of the own vehicle, and the like can be considered, but the description is omitted here.

  In step S203, the horizontal edge filter is applied to all the processing lines set in step S201, and the process proceeds to step S205. In step S205, the rising and falling peaks of luminance are extracted from the processing result in step S203, and the process proceeds to step S207. In step S207, based on the result of the process in step S205, a process of leaving only edges that form a pair with a rising peak of luminance and a falling peak of luminance at a predetermined interval (an interval corresponding to the horizontal width of the parking frame line). Go to step S209.

  In step S209, with respect to the edge left by the processing in step S207, grouping (grouping) that associates peaks estimated to be linearly arranged between rising peaks and falling peaks as a group. I do. Here, since the camera image is used as it is in the present embodiment, the peak of the edge is not aligned on the straight line on the image as in the first embodiment. Therefore, in this embodiment, all edge points are converted into world coordinates using camera geometric information, and grouping is performed on the world coordinates. After grouping, the process proceeds to step S211. In step S211, filtering is performed on each group of peaks obtained by the processing in step S209 according to the length of each group in order to remove noise. That is, among the linearly arranged peaks constituting one group, only the group in which the interval between two peaks corresponding to both ends of the straight line is longer than a predetermined length is left. When the process in step S211 is executed, the process proceeds to step S213.

  In step S213, for each group left by the processing in step S211, two groups corresponding to both ends of the straight line in an arbitrary group of rising edges and a group of falling edges paired with the rising edges. The coordinates of the end points (upper end and lower end) are extracted and output to the side approach determination unit 362.

  FIG. 20 is a flowchart of processing performed by the side approach determination unit 362 of the arithmetic processing unit 31. In step S362, based on the coordinates of the four corners of the parking frame 23 registered in step S129 in FIG. 6 of the first parking frame detection unit 302 described above, the two parking frame lines 41La constituting the parking frame 23, Among the end portions of 41Lb, the front side white line end portions 41LaE and 41LbE, which are the end portions of the white line existing on the front side of the vehicle 1, are at the boundaries 46L and 46R between the traveling direction camera image 21e and the side camera images 21b and 21c. It is determined whether it has been reached. For example, FIG. 19 shows an example of a state where the shift position of the vehicle 1 is D (drive), the vehicle 1 is determined to be moving forward, and the front camera image 21a is selected as the traveling direction camera image 21e. FIG. FIG. 19 shows a state in which the front white line end portions 41LaE and 41LbE have not reached the boundaries 46L and 46R.

  When at least one of the two front side white line ends 23LE existing in the left-right direction in the drawing reaches the boundaries 46L and 46R as the vehicle 1 advances, an affirmative determination is made in step S521 and the process proceeds to step S523. . In step S523, of the two borders 46L and 46R existing in the horizontal direction in the figure, the front white line ends 41LaE and 41LbE touch the borders 46L and 46R that are determined to have reached the borders 46L and 46R in step S521. In the side camera images 21b and 21c, an instruction is output to the second white line detection unit 313 so as to start the white line detection process, and the process returns.

  If both of the front white line ends 41LaE and 41LbE existing in the left-right direction in the figure do not reach the boundaries 46L and 46R, a negative determination is made in step S311 and the process returns.

  FIG. 21 is a flowchart of processing performed by the second white line detection unit 313 of the arithmetic processing unit 31. The process shown in this flowchart is repeatedly executed by the second white line detection unit 313 when an instruction to start the white line detection process is input from the side approach determination unit 362, for example. Note that the parking frame line 41Lc (for example, the parking frame line 41Lc appearing in FIG. 19) displayed in the camera images 21b and 21c on the side of the vehicle 1 is detected by the processing performed by the second white line detection unit 313. . Hereinafter, the processing related to the left-side camera image 21b will be described using this flowchart, but the processing for the right-side camera image 21c is also the same.

  In step S221, the predicted position of the parking frame line 43L that is predicted to exist in the side camera images 21b and 21c is calculated based on the position information about the parking frame 23 registered by the first parking frame detection unit 302. FIG. 22 shows an example of processing for the left-side camera image 21b. Hereinafter, a description will be given based on the drawings. As shown in the figure, the predicted position of the parking frame line 41Lb that is predicted to exist in the side camera images 21b and 21c from the information about the line 41Lb constituting the parking frame 23 recognized by the first parking frame detection unit 302. An imaginary line 43L is calculated.

  In step S223, the virtual line 43L is superimposed on the side image 21b using the camera geometric information, and a processing line 44L is set around the line.

In step S225, a lateral edge filter is applied to the processing line set in step S223.
In this way, by applying a lateral edge filter whose Y-direction spacing is sparser than the processing line 42, it is possible to reduce the load of white line detection processing on the side camera images 21b and 21c.

  In step S227, the rise and fall peaks of the luminance are extracted from the processing result in step S225, and the process proceeds to step S229. In step S229, from the result of the process in step S227, a process of leaving only edges that form a pair with a rising peak of luminance and a falling peak of luminance at a predetermined interval (interval corresponding to the horizontal width of the parking frame line). Go to step S231.

  In step S <b> 231, an edge pair is extracted that is formed as a pair by the rising peak of luminance and the falling peak of luminance existing in the vicinity of each intersection 45 of each processing line 44 and virtual line 43.

  In step S233, the processing area is sequentially searched from the extended prediction line 43L and the intersection 47L of the line 43L to determine whether or not an edge pair exists. If the edge pair does not exist or if the search is successful up to the final line, the process is terminated on that line, and the process proceeds to step S235. In step S235, in step S233, the intersection 47L and the point immediately before the point at which the search is completed are registered as white lines.

[Third embodiment]
Next, a third embodiment of the present invention will be described below with reference to FIGS.

  FIG. 24 is a block diagram illustrating the configuration of an external environment recognition device 300 according to the third embodiment. An in-vehicle image processing device 300 shown in FIG. 24 is used by being mounted on the vehicle 1, and includes an arithmetic processing unit 32 and an output unit 4. The same components as those in the first embodiment and the second embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

What is characteristic in the present embodiment is that the traveling lane is detected from the white line extraction result instead of the parking frame. As shown in FIG. 23 (a), when the vehicle 1 is likely to deviate from the lane, it is preferable to check with the side camera whether the front wheels actually crossed the lane.
In this case, the present embodiment accurately recognizes the timing at which the front wheels depart from the lane using the side camera.

  The contents of the arithmetic processing performed by the arithmetic processing unit 32 will be described. FIG. 24 is a control block diagram of the arithmetic processing performed by the arithmetic processing unit 32. As shown in FIG. 24, the arithmetic processing unit 32 functionally controls each control block of the first white line detection unit 311, the first lane detection unit 322, the second white line detection unit 313, and the second lane detection unit 324. Have. Among the images 21a to 21d acquired from the cameras 1a to 1d attached to the vehicle, the arithmetic processing unit 32 includes a front camera image 21a in the first white line detector 311 and a right camera image 21b in the second white line detector. A left-side camera image 21c is used. In the arithmetic processing unit 32, for example, each control block of FIG. 20 is realized by executing a program recorded in a memory corresponding to each of these control blocks with a microcomputer. The arithmetic processing unit 32 performs the process when the vehicle speed becomes equal to or higher than a predetermined speed (for example, 40 km / h), for example. The determination as to whether or not the vehicle speed is equal to or lower than a predetermined speed may be made based on the cycle of the vehicle speed pulse output from the vehicle side, or based on information about the vehicle speed output from a control device (not shown) of the vehicle. You may judge. The recognition result of the parking frame by the arithmetic processing unit 32 is output to the output unit 4 and used for, for example, a lane departure warning.

  Since the processing of the first white line extraction unit 311 in the third embodiment is the same as the flow described in the second embodiment, a detailed description thereof is omitted. As shown in FIG. 26A, a processing area 42 is set for the white lines 41La and 41Lb shown in the front camera image 21a. The processing area is set in the vicinity of the past lane recognition result or set so as to process the entire screen. In the present embodiment, the front camera image 21a is divided into left and right, and the left lane and the right lane are detected from each region. The result of white line detection (recognition) by the first white line detection unit 311 is output to the first lane detection unit 322.

  FIG. 27 is a flowchart of processing performed by the first lane detection unit 322 of the arithmetic processing unit 32. The processing shown in this flowchart is repeatedly executed every time the arithmetic processing unit 32 performs processing. In step S301, based on the information output from the first white line detection unit 311, information on any two white lines is selected, and the process proceeds to step S303.

  In step S303, it is determined whether or not the angular difference in the extending direction between the two white lines selected in step S301 is equal to or less than a predetermined value (Thθmax). That is, in step S303, it is determined whether or not the two white lines are substantially parallel. If an affirmative determination is made in step S303, the process proceeds to step S305, and it is determined whether or not the interval between the two white lines is within a predetermined range (ThWmin or more and ThWmax or less). That is, in step S305, it is determined whether or not two white lines are lined up at an interval considered as two white lines constituting the lane.

  If an affirmative determination is made in step S305, the process proceeds to step S307, where the coordinates of the four corners composed of two white lines are registered as position information about the lane. When step S305 is executed, the process proceeds to step S307, and it is confirmed whether or not the above-described processing for any two white lines has been performed for all the white lines based on the information output from the first white line detection unit 311. If the determination in step S307 is affirmative, the result obtained by the above-described processing is output to the second white line detection unit 313, and the process returns. If a negative determination is made in step S309, the process returns to step S301.

  If a negative determination is made in step S303 or a negative determination is made in step S305, the process proceeds to step S309.

  Since the processing of the second white line detection unit 313 in the third embodiment is the same as the flow described in the second embodiment, a detailed description thereof is omitted. As shown in FIG. 28A, the virtual line 43 is drawn based on the information of the white line forming the lane detected from the front camera image 21a, and the processing area 44 is set around the virtual line. The white line detection result of the second white line detection unit is output to the second lane detection unit 324.

  FIG. 29 is a flowchart of processing performed by the second lane detection unit 324 of the arithmetic processing unit 32. The processing shown in this flowchart is repeatedly executed every time the arithmetic processing unit 32 performs processing. In step S321, it is determined whether the above-described second white line detection unit 313 has detected a white line. If a positive determination is made in step S321, the process proceeds to step S323.

  In step S323, the white line detected by the second white line detection unit 313 and the white line detected by the second white line detection unit 313 among the white lines detected by the first white line detection unit 311 are formed. The lane registered by the first lane detector 322 is rejected and newly registered. Then, the registered information is output to the output unit 4 and the process returns.

  If a negative determination is made in step S321, the process returns.

  As described above, the present invention is not limited to the detection of the parking frame, and when detecting an object that only partially enters the field of view of the front camera when turning, the side is determined based on the detection result of the front camera. A part of the object is detected by the side camera, and the object is detected using the detection results of the front camera and the side camera.

---- Modified example ---
(1) In the above description, for example, when the vehicle speed becomes a predetermined speed (for example, 50 km / h) or less, the above-described processing is performed in each unit of the arithmetic processing unit 3. It is not limited to. The above-mentioned predetermined speed of 50 km / h is an example, and the vehicle speed as a threshold for performing the above-described processing in each part of the arithmetic processing unit 3 may be a speed higher or lower than 50 km / h. Further, for example, the above-described processing is performed by each unit of the arithmetic processing unit 3 based on information indicating whether or not the current position of the vehicle 1 is in a parking lot from a navigation device mounted on the vehicle 1. May be.

(2) In the above description, the parking frame 23 is detected by the first parking frame detection unit 302 and the second parking frame detection unit 304, but the present invention is not limited to this. For example, you may comprise so that the process performed by the 1st parking frame detection part 302 may be performed by the 2nd parking frame detection part 304. FIG. That is, it is not essential to provide the first parking frame detection unit 302.
(3) You may combine each embodiment and modification which were mentioned above, respectively.

Note that the present invention is not limited to the embodiment described above, and is taken by a traveling direction detection unit that detects the traveling direction of the vehicle and cameras provided at the front, rear, and sides of the vehicle. The obtained image is acquired, and among the acquired images, an image obtained by photographing with a camera provided in front of the vehicle, and an image obtained by photographing with a camera provided at the rear of the vehicle. From either image, an image selection unit that selects an image corresponding to the traveling direction of the vehicle detected by the traveling direction detection unit as the traveling direction image, and a feature of the object is detected from the traveling direction image selected by the image selection unit. Of the object direction detected by the traveling direction object feature detecting unit and the end of the object feature detected by the traveling direction object feature detecting unit is provided on the side of the vehicle. Side image obtained by camera In the side approach determination unit that determines whether or not the boundary between the traveling direction image and the side approach determination unit is reached, the near-side object feature edge reaches the boundary between the traveling direction image and the side image. Is detected using the feature of the object detected by the side object feature detection unit that detects the feature of the object from the side image, the traveling direction object feature detection unit, and the side object feature detection unit. It includes an in-vehicle image processing apparatus having various structures, characterized in that it includes an object detection unit for detection.
Further, the present invention is not limited to the above-described embodiment, and acquires images obtained by photographing with cameras provided in front, rear, and side of the vehicle, and combines the acquired images. Among the overhead images generated by the image composition unit that generates the bird's-eye view image, the traveling direction detection unit that detects the traveling direction of the vehicle, and the image composition unit, the image is captured by a camera provided in front of the vehicle. Of the front overhead image generated based on the obtained image and the overhead image generated by the image composition unit, the rear overhead image generated based on the image obtained by photographing with the camera provided behind the vehicle An image selection unit that selects an overhead image corresponding to the traveling direction of the vehicle detected by the traveling direction detection unit from either one of the overhead direction images, and a white line from the traveling direction overhead image selected by the image selection unit Detect hex Of the white line ends detected by the direction white line detection unit and the white line detection unit of the traveling direction, the front white line end that is the end of the white line existing on the front side of the vehicle is an overhead image generated by the image composition unit. Of these, a side approach determination unit that determines whether or not a boundary between a front overhead image generated based on an image obtained by photographing with a camera provided on the side of the vehicle and a traveling direction overhead image has been reached. When the side approach determination unit determines that the front white line end has reached the boundary between the front bird's-eye view image and the traveling direction bird's-eye view image, the side white line detection unit that detects a white line from the side bird's-eye view image The vehicle-mounted image processing apparatus of various structures characterized by comprising.
Further, the present invention is not limited to the above-described embodiment, and acquires images obtained by photographing with cameras provided in front, rear, and side of the vehicle, and combines the acquired images. Among the overhead images generated by the image composition unit that generates the bird's-eye view image, the traveling direction detection unit that detects the traveling direction of the vehicle, and the image composition unit, the image is captured by a camera provided in front of the vehicle. Of the front overhead image generated based on the obtained image and the overhead image generated by the image composition unit, the rear overhead image generated based on the image obtained by photographing with the camera provided behind the vehicle An image selection unit that selects an overhead image corresponding to the traveling direction of the vehicle detected by the traveling direction detection unit from either one of the overhead direction images, and a white line from the traveling direction overhead image selected by the image selection unit Detect the second The white line detection unit, the first parking frame detection unit that detects the parking frame based on the detection result of the first white line detection unit, and the end of the white line related to the parking frame detected by the first parking frame detection unit The white line end on the near side, which is the end of the white line existing on the near side of the vehicle, is obtained by photographing with a camera provided on the side of the vehicle in the overhead view image generated by the image composition unit. In the side approach determination unit that determines whether or not the boundary between the forward bird's-eye view image generated based on the image and the traveling direction bird's-eye view image has been reached, A second white line detection unit that detects a white line from the side overhead image with reference to a detection result of the first parking frame detection unit when it is determined that the boundary between the overhead image and the traveling direction overhead image has been reached; The vehicle-mounted image processing apparatus of various structures characterized by comprising.

2 image compositing unit, 3 arithmetic processing unit, 4 output unit, 22 overhead image, 22a front overhead image, 22b, 22c side overhead image, 22d rear overhead image, 22e traveling direction overhead image, 100 in-vehicle image processing device, 301 1 white line detection unit, 302 first parking frame detection unit, 303 second white line detection unit, 304 second parking frame detection unit, 310 image selection unit, 311 side entry determination unit

Claims (12)

A traveling direction detector for detecting the traveling direction of the vehicle;
An image obtained by photographing with a camera provided at the front, rear, and side of the vehicle was obtained, and of the obtained images, obtained by photographing with a camera provided at the front of the vehicle. An image corresponding to the traveling direction of the vehicle detected by the traveling direction detection unit from either one of an image and an image obtained by photographing with a camera provided behind the vehicle. An image selector to select as,
A traveling direction object feature detection unit that detects a feature of the object from the traveling direction image selected by the image selection unit;
Out of the object features detected by the advancing direction object feature detector, the front object feature end located on the front side of the vehicle is obtained by photographing with a camera provided on the side of the vehicle. A side approach determination unit that determines whether or not the boundary between the side image and the traveling direction image has been reached,
When the side approach determining unit determines that the near-side object feature edge has reached the boundary between the traveling direction image and the side image, the side detecting the feature of the object from the side image An object feature detection unit;
An in-vehicle image processing apparatus comprising: an object detection unit that detects an object using a feature of an object detected by the traveling direction object feature detection unit and the side object feature detection unit. An image synthesizing unit that obtains images obtained by photographing with cameras provided in front, rear, and side of the vehicle, synthesizes the obtained images, and generates an overhead image;
A traveling direction detector for detecting a traveling direction of the vehicle;
Among the overhead images generated by the image composition unit, a front overhead image generated based on an image obtained by photographing with a camera provided in front of the vehicle, and an overhead image generated by the image composition unit Of these, in the traveling direction of the vehicle detected by the traveling direction detection unit, from any one of the overhead view images generated based on an image obtained by photographing with a camera provided behind the vehicle. An image selection unit that selects a corresponding overhead image as a traveling direction overhead image;
A traveling direction white line detection unit that detects a white line from the traveling direction overhead image selected by the image selection unit;
Among the white line ends detected by the advancing direction white line detection unit, the front white line end that is the end of the white line present on the front side of the vehicle is the vehicle out of the overhead image generated by the image composition unit. A side approach determination unit that determines whether or not a boundary between the front bird's-eye view image generated based on an image obtained by photographing with a camera provided on the side of the vehicle and the traveling direction bird's-eye view image has been reached,
When the side approach determination unit determines that the end of the front white line has reached the boundary between the front bird's-eye view image and the traveling direction bird's-eye view image, a side white line that detects a white line from the side bird's-eye view image A vehicle-mounted image processing apparatus comprising: a detection unit. The in-vehicle image processing apparatus according to claim 2,
The in-vehicle image processing apparatus, wherein the side white line detection unit detects a white line from the side bird's-eye view image with reference to at least information regarding an extending direction of the white line detected by the traveling direction white line detection unit. . The in-vehicle image processing device according to claim 3,
The in-vehicle image characterized in that the side white line detection unit predicts the position of the white line in the side overhead image with reference to at least information on the extending direction of the white line detected by the traveling direction white line detection unit. Processing equipment. The in-vehicle image processing device according to claim 4,
The side white line detection unit calculates an intersection of the predicted position of the white line in the side overhead image and the scanning line of the side overhead image, and scans to cross the white line existing in the side overhead image. The detected white line is detected by detecting the rise and fall of the luminance obtained and determining whether or not the position where the detected rise and fall of the luminance are coincident with the calculated intersection point. An in-vehicle image processing apparatus characterized by the above. In the in-vehicle image processing device according to any one of claims 2 to 5,
The in-vehicle image processing apparatus according to claim 1, wherein a scanning density of the side overhead image by the side white line detection unit is coarser than a scanning density of the traveling direction overhead image by the traveling direction white line detection unit. The in-vehicle image processing device according to claim 6,
The advancing direction white line detection unit detects a white line by sequentially scanning the advancing direction direction overhead image,
The in-vehicle image processing apparatus, wherein the side white line detection unit detects a white line by interlaced scanning the side overhead image. In the in-vehicle image processing device according to any one of claims 2 to 7,
Based on the detection result of the traveling direction white line detection unit, further comprising a traveling direction parking frame detection unit for detecting a parking frame,
The in-vehicle image processing apparatus, wherein the front white line end is an end of a white line existing on the front side of the vehicle among the end of the white line related to the parking frame detected by the traveling direction parking frame detection unit. . In the in-vehicle image processing device according to any one of claims 2 to 8,
An in-vehicle image processing apparatus further comprising a parking frame detection means for detecting a parking frame based on at least information on the white line detected by the side white line detection unit.   The in-vehicle image processing apparatus according to claim 1, wherein the object feature is a white line, and the object is a parking frame.   The in-vehicle image processing apparatus according to claim 1, wherein the object feature is a white line, and the object is a lane. An image synthesizing unit that obtains images obtained by photographing with cameras provided in front, rear, and side of the vehicle, synthesizes the obtained images, and generates an overhead image;
A traveling direction detector for detecting a traveling direction of the vehicle;
Among the overhead images generated by the image composition unit, a front overhead image generated based on an image obtained by photographing with a camera provided in front of the vehicle, and an overhead image generated by the image composition unit Of these, in the traveling direction of the vehicle detected by the traveling direction detection unit, from any one of the overhead view images generated based on an image obtained by photographing with a camera provided behind the vehicle. An image selection unit that selects a corresponding overhead image as a traveling direction overhead image;
A first white line detection unit for detecting a white line from the traveling direction overhead image selected by the image selection unit;
A first parking frame detection unit for detecting a parking frame based on a detection result of the first white line detection unit;
Of the white line end portions related to the parking frame detected by the first parking frame detection unit, the front white line end portion that is the end portion of the white line existing on the front side of the vehicle is generated by the image composition unit. The side which determines whether the boundary of the front bird's-eye view image produced | generated based on the image obtained by image | photographing with the camera provided in the side of the said vehicle among the images and the said advancing direction bird's-eye view image was reached. A direction entry determination unit;
When it is determined by the side approach determination unit that the front white line end has reached the boundary between the front overhead image and the traveling direction overhead image, refer to the detection result of the first parking frame detection unit. An in-vehicle image processing apparatus comprising: a second white line detection unit that detects a white line from the side overhead image.

Images