JP6948365B2 - Programs, devices, and methods for calculating the vanishing point - Google Patents

Description

The present invention relates to a program, an apparatus, and a method for calculating a vanishing point in a captured image.

Conventionally, various techniques using vanishing points in images taken by a camera placed in a vehicle have been proposed. For example, Patent Document 1 below detects a target vehicle image and a vanishing point from images captured in front of the vehicle, and calculates a distance (inter-vehicle distance) to the target vehicle based on the target vehicle image and the vanishing point. The device to be used is disclosed. Further, for example, in Patent Document 2 below, a vanishing point in the field of view on the front side of the vehicle is calculated based on a photographed image of the scenery in front of the vehicle, and the headlight of the vehicle is controlled based on the vanishing point. The device to be used is disclosed.

Japanese Unexamined Patent Publication No. 2014-167397 Japanese Unexamined Patent Publication No. 2019-10962

However, if the direction of the camera changes due to vibration of the vehicle body or the like while the vehicle is running, the position of the vanishing point in the captured image may also change. Such a change in the position of the vanishing point may affect the above-mentioned calculation of the inter-vehicle distance and various processes using the vanishing point such as control of the headlight.

One of the objects of the embodiment of the present invention is to support an appropriate calculation of a vanishing point in an image captured by a camera arranged in a vehicle. Other objects of the embodiments of the present invention will become apparent by reference to the entire specification.

A program according to an embodiment of the present invention includes a step of calculating a first vanishing point in an image captured by a camera arranged in a vehicle on one or a plurality of computers, and a point on a lane boundary in the captured image. A step of detecting a plurality of estimated vanishing points and calculating a second vanishing point based on at least the first vanishing point and the plurality of vanishing points is executed. Here, in the present specification, the "vanishing point" is defined as a point where parallel straight lines are gathered in perspective and perspective projection, and includes "vanishing point", "point at infinity" and the like.

An apparatus according to an embodiment of the present invention includes an estimation point detection unit configured to detect a plurality of estimation points estimated to be points on a lane boundary in an image captured by a camera arranged in a vehicle. A vanishing point calculation unit configured to calculate a vanishing point in the captured image is provided, and the vanishing point calculation unit is a first vanishing point calculation unit based on at least the first vanishing point and the plurality of estimated points in the captured image. 2 It is configured to calculate the vanishing point.

A method according to an embodiment of the present invention includes a step of calculating a first vanishing point in an image captured by a camera arranged in a vehicle, and a plurality of estimated points estimated to be points on a lane boundary line in the captured image. Is included, and a step of calculating a second vanishing point based on at least the first vanishing point and the plurality of estimated points is provided.

Various embodiments of the present invention support the proper calculation of vanishing points in images captured by a camera placed in a vehicle.

FIG. 6 is a configuration diagram schematically showing a configuration of an in-vehicle device 10 according to an embodiment of the present invention. FIG. 6 is a flow chart illustrating a process related to calculation of a vanishing point executed by the in-vehicle device 10. The figure for demonstrating the estimation point in this embodiment. FIG. 5 is a flow chart illustrating a process executed by the vanishing point calculation unit 114 of the vehicle-mounted device 10 when grouping a plurality of estimated points. The figure for demonstrating the procedure of grouping of estimated points. The figure for demonstrating the procedure of grouping of estimated points. The figure for demonstrating the procedure for calculating a temporary vanishing point. FIG. 5 is a flow chart illustrating a process related to calculation of an inter-vehicle distance executed by the in-vehicle device 10. The figure which illustrates the photographed image 20 in which a vehicle is detected. The figure for demonstrating the method of calculating the inter-vehicle distance.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram schematically showing a configuration of an in-vehicle device 10 according to an embodiment of the present invention. The device 10 is configured as a general computer, and as shown in the figure, a computer processor 11, a main memory 12, an input / output I / F 13, a communication I / F 14, and a storage (storage device) 15. Each of these components is electrically connected via a bus or the like (not shown). The in-vehicle device 10 is mounted and used in a vehicle such as an automobile.

The computer processor 11 is configured as a CPU, a GPU, and the like, reads various programs stored in the storage 15 and the like into the main memory 12, and executes various instructions included in the programs. The main memory 12 is composed of, for example, a DRAM or the like.

The input / output I / F 13 includes various input / output devices for exchanging information with a user or the like. The input / output I / F 13 includes, for example, a pointing device (for example, a touch panel), an information input device such as a physical button, a voice input device such as a microphone, and an image input device such as a camera. Further, the input / output I / F 13 includes an image output device (display device) such as a display and an audio output device such as a speaker.

In this embodiment, the input / output I / F 13 includes a camera 131. The in-vehicle device 10 is installed so that the front of the vehicle on which the device 10 is mounted is the field of view of the camera 131 and the optical axis of the camera 131 is parallel to the road surface.

The communication I / F 14 is implemented as hardware such as a network adapter, various communication software, and a combination thereof, and is configured to realize wired or wireless communication.

The storage 15 is composed of, for example, a magnetic disk, a flash memory, or the like. The storage 15 stores various programs including an operating system, various data, and the like. In the present embodiment, the storage 15 includes a program for an in-vehicle device 151 for causing the computer to function as the in-vehicle device 10 of the present embodiment, and a point on a lane marking of a road sign included in a captured image taken by the camera 131. The trained model 152 for estimating point detection used for detecting the estimated point estimated to be, and the trained model 153 for vehicle detection used for detecting the vehicle (predetermined object) also included in the captured image are stored. There is.

In the present embodiment, the vehicle-mounted device 10 may be configured by using a plurality of devices, each of which has the above-mentioned hardware configuration.

Next, the functions of the vehicle-mounted device 10 of the present embodiment will be described. As shown in FIG. 1, the computer processor 11 of the apparatus 10 is an estimation point detection unit 112 configured to detect a plurality of estimation points estimated to be points on a lane marking in a captured image captured by the camera 131. , The vanishing point calculation unit 114 configured to calculate the vanishing point in the image captured by the camera 131, the vehicle detection unit 116 configured to detect the vehicle in the image captured by the camera 131, and the inter-vehicle distance calculation. It is configured to function as an inter-vehicle distance calculation unit 118, which is configured to do so. These functions are realized by the computer processor 11 operating in cooperation with hardware such as the main memory 12 and various programs and data stored in the storage 15 and the like. For example, the main memory 12 It is realized by executing the instruction included in the program loaded in.

Next, the operation of the in-vehicle device 10 of the present embodiment will be described. First, the operation related to the calculation of the vanishing point (auto-calibration of the camera 131) will be described. FIG. 2 is a flow chart illustrating a process related to calculation of a vanishing point executed by the in-vehicle device 10. These processes are executed when the in-vehicle device 10 is started, and then are periodically and repeatedly executed.

First, as shown in FIG. 2, the in-vehicle device 10 acquires a captured image captured by the camera 131 (step S100), and a plurality of estimates presumed to be points on the lane marking of the road marking in the acquired captured image. A point is detected (step S110). Acquisition of these captured images and detection of estimated points are executed by the estimated point detecting unit 112. Specifically, the estimation point detection unit 112 acquires an image taken by the camera 131 at that time (an image corresponding to the scenery in front of the moving vehicle on which the in-vehicle device 10 is mounted). Further, in the present embodiment, the estimation point detection unit 112 detects the intersection of the virtual horizon and the division line at a predetermined height in the acquired captured image as the estimation point.

FIG. 3 is a diagram for explaining an estimated point in the present embodiment, and illustrates a captured image 20 as an example of a captured image of the camera 131. In the photographed image 20, the three lane markings ML1 to ML3 drawn on the roadway extend forward (in the direction of travel of the vehicle), and each of these lane markings ML1 to ML3 has three different heights. An intersection with each of the virtual horizontal lines y1 to y3 (that is, "three points on each lane marking at three predetermined heights") is detected as an estimated point EP.

In the present embodiment, the detection of the estimated points in the captured image is performed by using the above-described trained model 152 for detecting the estimated points. The trained model 152 for estimating point detection is configured as, for example, a convolutional neural network (CNN) model, and when a captured image is input, the position of the estimated point included in the captured image is specified. It is configured to output position identification information (for example, coordinate information and reliability). Such a trained model 152 is generated by machine learning using training data. The learning data includes, for example, a large number of learning images taken from a moving vehicle. The learning image is, for example, a still image corresponding to each frame constituting a moving image shot through a camera provided so as to shoot the front of the vehicle. The learning image is associated with position-specific information at the intersection of each of the one or more division lines included in the image and each of the three virtual horizontal lines y1 to y3.

The detection of the estimated points using the trained model is an example of a method for detecting the estimated points, and in another embodiment of the present invention, the estimated points are detected by using another method. For example, in other embodiments, the estimated points are detected by image analysis using a rule-based algorithm.

Return to the flow chart of FIG. When the estimated points are detected in this way, the vehicle-mounted device 10 then groups a plurality of estimated points (step S120). The grouping is executed by the vanishing point calculation unit 114. In the present embodiment, the vanishing point calculation unit 114 groups a plurality of estimated points based on the positions of the previously registered vanishing points (first vanishing points). When the previously registered vanishing point does not exist (for example, immediately after the in-vehicle device 10 is started), a predetermined position (for example, the center of gravity (center) of the captured image) is set as the position of the previous vanishing point.

FIG. 4 is a flow chart illustrating a process executed by the vanishing point calculation unit 114 of the vehicle-mounted device 10 when grouping a plurality of estimated points. As shown in the figure, the vanishing point calculation unit 114 first groups the estimated points on the lowest (lowermost) horizontal line y1 among the three virtual horizontal lines y1 to y3. To identify the unprocessed estimated points (step S200). The unprocessed estimation points are specified in order from, for example, the estimation points on the left side to the estimation points on the right side.

Next, the vanishing point calculation unit 114 calculates the intersection of the estimated point on the specified horizon y1 and the virtual straight line passing through the previous vanishing point and the virtual horizon y2 located one above the horizon y1. (Step S210). FIG. 5 is a diagram for explaining a procedure for grouping estimated points. In step S210, for example, as shown in FIG. 5, the intersection IP11 (coordinate information) of the straight line SL1 passing through the estimated point EP1 on the horizon y1 and the previous vanishing point VP_prev and the horizon y2 is calculated.

Return to the flow chart of FIG. When the intersection of the estimated point on the horizon y1 and the straight line passing through the previous disappearance point and the horizon y2 is calculated in this way, the vanishing point calculation unit 114 then calculates a predetermined range in the horizontal direction (for example, ± x pixels) from the calculated intersection. It is determined whether or not there is an estimated point on the horizon y2 within the range) (step S220), and if there is an estimated point on the horizon y2 within the predetermined range (YES in step S220), the estimated point Is selected as an estimation point in the same group as the estimation point on the horizon y1 (step S230), while there is no estimation point on the horizon y2 within a predetermined range (NO in step S220), on the horizon y1. The intersection of the estimated point and the straight line passing through the previous disappearance point and the horizontal line y2 is selected as an estimated point in the same group as the estimated point on the horizontal line y1 (step S240).

For example, in the example of FIG. 5, since the estimated point EP2 on the horizontal line y2 exists in a predetermined range R (right side of the intersection IP11) in the horizontal direction from the intersection IP11 of the straight line SL1 and the horizontal line y2, the estimated point EP2 is , Selected as an estimated point in the same group as the estimated point EP1 on the horizon y1. On the other hand, in the example of FIG. 6, since the estimated point on the horizontal line y2 does not exist in the predetermined range R in the horizontal direction from the intersection IP11 between the straight line SL1 and the horizontal line y2, the intersection IP11 is the estimated point EP1 on the horizontal line y1. Is selected as the estimated point for the same group as. In this way, when some estimated points are missing, for example, there is a case where an obstacle or the like exists near the intersection of the virtual horizon and the lane marking in the captured image and the lane marking is hidden. Conceivable. In this way, when some estimated points are missing, the vanishing point is calculated by regarding the other points (the intersection of the straight line passing through the previous vanishing point and the horizon) as the estimated points and proceeding with the processing. The processing speed is increased.

Returning to the flow chart of FIG. 4, the vanishing point calculation unit 114 then calculates the intersection of the estimated point on the horizontal line y1 and the virtual straight line passing through the previous vanishing point and the uppermost horizontal line y3 (step). S250), it is determined whether or not there is an estimated point on the horizontal line y3 within a predetermined range in the horizontal direction (for example, a range of ± x pixels) from the calculated intersection (step S260), and on the horizontal line y3 within the predetermined range. (YES in step S260), the estimated point is selected as an estimated point in the same group as the estimated point on the horizon y1 (step S270), while on the horizon y3 within a predetermined range. If the estimation point does not exist (NO in step S260), the estimation point on the horizon y1 and the intersection of the straight line passing through the previous vanishing point and the horizon y3 are selected as the estimation points in the same group as the estimation point on the horizon y1. (Step S280).

For example, in the examples of FIGS. 5 and 6, since the estimated point EP3 on the horizontal line y3 exists in a predetermined range R (right side of the intersection IP12) in the horizontal direction from the intersection IP12 of the straight line SL1 and the horizontal line y3, the estimation is performed. The point EP3 is selected as the estimated point in the same group as the estimated point EP1 on the horizon y1. As a result, in the example of FIG. 5, the estimated point EP1 on the horizontal line y1, the estimated point EP2 on the horizontal line y2, and the estimated point EP3 on the horizontal line y3 are grouped into the same group. On the other hand, in the example of FIG. 6, the estimated point EP1 on the horizontal line y1, the intersection IP11 on the horizontal line y2, and the estimated point EP3 on the horizontal line y3 are grouped in the same group.

Returning to the flow chart of FIG. 4, the vanishing point calculation unit 114 then determines whether or not an unprocessed estimated point on the horizon y1 exists (remains) (step S290), and unprocessed estimation. If a point exists (YES in step S290), the process returns to the identification of the unprocessed estimated point (step S200), while if there is no unprocessed estimated point (NO in step S290), the figure. The process in the flow chart of No. 4 is completed. When the processing in the flow chart of FIG. 4 is completed, the same number of groups as the number of estimated points on the horizontal line y1 are created.

Return to the flow chart of FIG. When the grouping of the plurality of estimated points is performed in this way, the vehicle-mounted device 10 next determines whether or not the number of groups created by the grouping is 2 or more (step S130). The determination is executed by the vanishing point calculation unit 114.

Then, when the number of groups is less than 2 (that is, 1) (NO in step S130), the in-vehicle device 10 returns to the acquisition of the captured image (step S100), and the captured image of the camera 131 at that time. Is newly acquired, and the process proceeds to the detection of the estimated point (step S110) again.

On the other hand, when the number of groups is 2 or more (YES in step S130), the vehicle-mounted device 10 calculates and accumulates a temporary vanishing point (second vanishing point) (step S140). The calculation and accumulation of the temporary vanishing point is executed by the vanishing point calculation unit 114. In the present embodiment, the vanishing point calculation unit 114 calculates an estimated vanishing point that is presumed to be a lane marking based on a plurality of estimated points belonging to the same group, and a temporary vanishing point based on the intersection of the plurality of estimated vanishing points. Is calculated.

FIG. 7 is a diagram for explaining a procedure for calculating a temporary vanishing point. In the example of FIG. 7, of the three estimated points EP11, EP21, and EP31 on the horizontal line y1, the estimated point EP11 on the left side is the same as the estimated point EP12 on the left side on the horizontal line y2 and the estimated point EP13 on the left side on the horizontal line y3. Grouped into groups. Similarly, the left-right center estimation point EP21 on the horizontal line y1 is grouped in the same group as the left-right center estimation point EP22 on the horizontal line y2 and the left-right center estimation point EP23 on the horizontal line y3. The right estimated point EP31 on the horizontal line y1 is grouped in the same group as the right estimated point EP32 on the horizontal line y2 and the right estimated point EP33 on the horizontal line y3.

In the present embodiment, the vanishing point calculation unit 114 calculates the slope of a straight line passing through the two estimation points for each combination of the two estimation points (there are three combinations) in the three estimation points belonging to the same group. Then, the representative value (for example, the average value or the median value) of the inclination of each of the three combinations is used as the inclination of the estimated division line, and a straight line having the inclination and passing through the estimated point on the horizontal line y1 belonging to the same group is defined as the inclination. Calculated as an estimated lane marking.

For example, in the example of FIG. 7, the slope of a straight line passing through the estimation points EP11 and EP12, which is a combination of two estimation points in the three estimation points EP11, EP12, and EP13 belonging to the same group, and a straight line passing through the estimation points EP11 and EP13. And the representative value of the slope of the straight line passing through the estimated points EP12 and EP13 are calculated, and the straight line having the slope of the calculated representative value and passing through the estimated point EP11 on the horizontal line y1 is calculated as the estimated lane marking EL1. Will be done. Similarly, the estimated lane line EL2 is calculated based on the three estimated points EP21, EP22, and EP23 belonging to the same group, and the estimated lane marking EL3 is calculated based on the three estimated points EP31, EP32, and EP33 belonging to the same group. It is calculated.

As described above, in the present embodiment, when calculating the estimated division line, the estimated point on the horizontal line y1 located at the lowermost side of the three virtual horizontal lines y1 to y3 is used as a reference (horizontal line). The estimated points are grouped based on the straight line passing through the estimated point on y1 and the previous vanishing point, and the estimated dividing line is calculated so as to pass through the estimated point on the horizontal line y1). However, in another embodiment of the present invention, when calculating the estimated division line, the estimated point on the horizon y2 is used as a reference, or the estimated point on the horizon y3 is used as a reference.

Further, the method for calculating such an estimated lane marking is an example, and in another embodiment of the present invention, the estimated lane marking is calculated by using another method. For example, in other embodiments, a straight line obtained by applying the least squares method to three estimated points belonging to the same group is used as the estimated lane marking.

In the present embodiment, the vanishing point calculation unit 114 calculates the intersections of a plurality of estimated vanishing points as temporary vanishing points, and when there are a plurality of such intersections (that is, when the number of estimated vanishing points is 3 or more). Calculates the representative value (average value or median value) of a plurality of intersections as a temporary vanishing point.

For example, in the example of FIG. 7, among the three estimated lane markings EL1 to EL3, the intersection IP21 of the estimated lane markings EL1 and EL2, the intersection IP22 of the estimated lane markings EL1 and EL3, and the intersection IP23 of the estimated lane markings E21 and EL3. The temporary vanishing point is calculated based on the above, and specifically, the point specified by the representative value of the X coordinate of each of these intersections IP21, IP22, and IP23 and the representative value of the Y coordinate is the temporary vanishing point ( (Coordinate information of) is calculated.

Returning to the flow chart of FIG. 2, in step S140, the calculated information on the temporary vanishing point (for example, the coordinate information is included) is accumulated in the storage 15. After calculating and accumulating the temporary vanishing points in this way, the in-vehicle device 10 next determines whether or not the number of accumulated temporary vanishing points is equal to or greater than the threshold value Nvp (step S150). The determination is executed by the vanishing point calculation unit 114. Then, when the number of accumulated temporary vanishing points is less than the threshold value Nvp (NO in step S150), the in-vehicle device 10 returns to the acquisition of the captured image (step S100), and the camera 131 at that time The captured image is newly acquired, and the process proceeds to the detection of the estimated point (step S110) again.

On the other hand, when the number of accumulated temporary vanishing points is equal to or greater than the threshold value Nvp (YES in step S150), the in-vehicle device 10 has a vanishing point (third vanishing point) based on the plurality of accumulated temporary vanishing points. ) Is calculated and registered (step S160). The calculation and registration of the vanishing point is executed by the vanishing point calculation unit 114.

In the present embodiment, the vanishing point calculation unit 114 calculates the point specified by the representative value of the X coordinate and the representative value of the Y coordinate of each of the accumulated temporary vanishing points as the current vanishing point. (Filtering). The calculated vanishing point information (for example, including coordinate information) is stored in the storage 15. In this way, by calculating the current vanishing point based on the plurality of temporary vanishing points, the accurate calculation of the vanishing point is promoted.

As described above, the processing in the flow diagram illustrated in FIG. 2 is repeatedly executed periodically (for example, each time a predetermined time elapses after the calculation and registration of the vanishing point are performed) after the in-vehicle device 10 is started. Will be done. Then, in the next grouping of estimated points (step S120), the vanishing point newly registered this time is used as the previous vanishing point.

As described above, in the present embodiment, the vanishing point is calculated repeatedly, so that it is possible to deal with a case where the position of the vanishing point changes according to a change in the orientation of the camera due to vibration of the vehicle body or the like. Further, since the previous vanishing point is used in the calculation of the vanishing point this time (more specifically, the grouping of a plurality of estimated points), the simplification and speeding up of the processing are promoted.

The operation related to the calculation of the vanishing point has been described above. Next, the operation related to the calculation of the inter-vehicle distance will be described. FIG. 8 is a flow chart illustrating a process related to calculation of the inter-vehicle distance executed by the in-vehicle device 10.

First, as shown in FIG. 8, the in-vehicle device 10 detects the vehicle in the captured image of the camera 131 (step S300). The detection of the vehicle is executed by the vehicle detection unit 116.

In the present embodiment, the detection of the vehicle in the captured image is performed by using the above-described trained model 153 for vehicle detection. The trained model 153 for vehicle detection is configured as, for example, a convolutional neural network model, and when a captured image is input, region identification information (for example,) that specifies a rectangular detection region surrounding the vehicle included in the captured image is used. , Coordinate information of each of the lower left vertex and the upper right vertex of the rectangular area) is output. Such a trained model 153 is generated by machine learning using the training data. The learning data includes, for example, a large number of learning images taken from a moving vehicle. The learning image is, for example, a still image corresponding to each frame constituting a moving image shot through a camera provided so as to shoot the front of the vehicle. The learning image is associated with the area identification information of the rectangular detection area that surrounds the vehicle included in the image.

For example, as illustrated in FIG. 9, the vehicle CR included in the captured image 20 is detected as a rectangular detection region DA. In addition, in another embodiment of the present invention, other objects such as pedestrians are detected in place of or in addition to the vehicle.

Note that the vehicle detection using the trained model is an example of a method for detecting a vehicle, and in another embodiment of the present invention, the vehicle is detected by using another method. For example, in other embodiments, the vehicle is detected by image analysis using a rule-based algorithm.

Returning to the flow chart of FIG. 8, the in-vehicle device 10 subsequently calculates the inter-vehicle distance based on the positional relationship between the vehicle detection region and the vanishing point in the captured image (step S310). The calculation of the inter-vehicle distance is executed by the inter-vehicle distance calculation unit 118.

In the present embodiment, the inter-vehicle distance calculation unit 118 is specifically based on the difference in the height direction between the lower side of the detection area of the vehicle and the vanishing point in the captured image and the height of the camera 131 from the road surface. , Calculate the inter-vehicle distance. As the vanishing point, the latest vanishing point registered at that time is used. Here, in the captured image 20 illustrated in FIG. 9, the y-coordinate (height) of the vanishing point VP is yVP, and the y-coordinate of the lower side of the detection area DA is yDA.

FIG. 10 shows the height H of the camera 131 from the road surface, the difference Δy between the y-coordinate yVP of the vanishing point VP in the captured image 20 and the y-coordinate yDA of the lower side of the detection area DA, and the distance between the camera 131 and the vehicle CR. The relationship with the inter-vehicle distance L will be illustrated. The inter-vehicle distance L is calculated by the following formula (1).

L = α (H) / Δy ... (1)

Here, α (H) is a constant calculated based on the height H. The height H (height of the camera 131 from the road surface) is measured when the in-vehicle device 10 is mounted on the vehicle and is stored in the storage 15 in advance.

The processing of the flow chart illustrated in FIG. 8 is repeatedly executed, that is, the detection of the vehicle included in the captured image and the calculation of the inter-vehicle distance to the detected vehicle are repeatedly executed.

The in-vehicle device 10 according to the present embodiment described above detects a plurality of estimated points estimated to be points on the lane marking of the road sign in the captured image of the camera 131, and these plurality of estimated points and the previous vanishing point. Since the vanishing point is calculated based on the above, the appropriate calculation of the vanishing point is supported.

In the above-described embodiment, the temporary vanishing point is calculated based on the presumed lane marking line, but in addition to the road marking lane marking, various boundaries of the lane in which the vehicle travels. The temporary vanishing point may be calculated based on the estimated boundary line estimated to be a line (for example, the boundary line between the road and the sidewalk).

In the above-described embodiment, the intersections of three virtual horizon lines having different heights and the lane markings are detected as estimated points, but the intersections of two or four or more horizon lines and the lane markings are detected. It may be detected as an estimation point. For example, when the intersection of two horizontal lines and a lane marking is detected as an estimation point, a straight line passing through two estimation points belonging to the same group is calculated as an estimation lane marking.

In the above-described embodiment, the inter-vehicle distance is calculated based on the vanishing point, but in another embodiment of the present invention, other processing based on the vanishing point is executed, for example, disappearance. It is configured to perform point-based headlight control.

In another embodiment of the present invention, at least a part of the functions of the in-vehicle device 10 is realized by a cloud server (a server connected via a communication network, etc.) or the like.

The processes and procedures described herein are implemented by software, hardware, or any combination thereof, other than those expressly described. For example, the processes and procedures described herein are realized by implementing logic corresponding to the processes and procedures on a medium such as an integrated circuit, a volatile memory, a non-volatile memory, or a magnetic disk. Further, the processes and procedures described in the present specification can be implemented as a computer program corresponding to the processes and procedures and executed by various computers.

Even if it is explained that the processes and procedures described herein are performed by a single device, software, component, module, such processes or procedures are performed by multiple devices, multiple software, multiple devices. Can be executed by a component of, and / or multiple modules. The software and hardware components described herein can also be realized by integrating them into fewer components or by breaking them down into more components.

In the present specification, when the components of the invention are described as either singular or plural, or even when they are described without limitation to either singular or plural, they should be understood separately in the context. Except for, the component may be either singular or plural.

10 In-vehicle device 11 Computer processor 112 Estimated point detection unit 114 Vanishing point calculation unit 116 Vehicle detection unit 118 Inter-vehicle distance calculation unit 13 Input / output I / F
131 Camera 15 Storage 151 In-vehicle device program 152 Learned model for estimated point detection 153 Learned model for vehicle detection 20 Captured image DA detection area EL Estimated lane line EP Estimated point ML lane marking point VP Vanishing point

Claims (13)

For one or more computers
The step of registering the first vanishing point in the image taken by the camera placed in the vehicle,
A step of detecting a plurality of estimated points estimated to be points on a lane boundary line in the captured image and calculating a second vanishing point based on at least the first vanishing point and the plurality of estimated points. Let it run
program. The step of calculating the second vanishing point includes detecting the intersection of a virtual horizon of a predetermined height and a lane boundary line in the captured image as an estimation point.
The program of claim 1. In the step of calculating the second vanishing point, a plurality of estimated boundary lines estimated to be lane boundaries in the captured image are calculated based on at least the first vanishing point and the plurality of estimated points, and the plurality of estimated vanishing points are calculated. Including calculating the second vanishing point based on at least the estimated boundary line of
The program of claim 1 or 2. In the step of calculating the second vanishing point, the plurality of estimated points are grouped based on at least the first vanishing point, and one estimated boundary line is calculated based on at least a plurality of estimated points belonging to the same group. Including that
The program of claim 3. The step of calculating the second vanishing point includes calculating the second vanishing point based on at least the intersection of the plurality of estimated boundaries.
The program of claim 3 or 4. The step of calculating the second vanishing point includes calculating the second vanishing point via statistical processing for a plurality of intersections of three or more estimated boundaries.
The program of claim 5. The step of calculating the second vanishing point includes calculating the representative value of the plurality of intersections as the second vanishing point.
The program of claim 6. The one or a plurality of computers are made to repeatedly execute the step of calculating the second vanishing point, and the predetermined number of the second vanishing points is calculated according to the calculation of two or more predetermined numbers of the second vanishing points. Perform the step of calculating the third vanishing point via statistical processing on the points,
The program according to any one of claims 1 to 7. The step of calculating the third vanishing point includes calculating a representative value of the predetermined number of the second vanishing points as the third vanishing point.
The program of claim 8. The camera and the predetermined object are further detected by the one or a plurality of computers in the captured image, and based on at least the positional relationship between the third vanishing point and the predetermined object in the captured image. To perform the step of calculating the distance based on the distance to the object of
The program of claim 8 or 9. The step of registering the first vanishing point comprises registering the third vanishing point as a new first vanishing point.
The program according to any one of claims 8 to 10. An estimation point detection unit configured to detect a plurality of estimation points estimated to be points on the lane boundary in an image captured by a camera placed on the vehicle, and an estimation point detection unit.
A vanishing point calculation unit configured to calculate and register a vanishing point in the captured image is provided.
The vanishing point calculation unit is configured to calculate a second vanishing point based on at least the first vanishing point registered earlier and the plurality of estimated points.
Device. The process of registering the first vanishing point in the image taken by the camera placed in the vehicle, and
A step of detecting a plurality of estimated points estimated to be points on a lane boundary line in the captured image and calculating a second vanishing point based on at least the first vanishing point and the plurality of estimated points. ,
Method.

Images