JP7116613B2 - Image processing device and image processing method - Google Patents

Description

TECHNICAL FIELD The present invention relates to an image processing device and an image processing method, and more particularly to technology for recognizing and managing parking frames by image processing.

Patent Literature 1 discloses detecting white lines from an image captured by a camera and detecting an area between two white lines as a parking space. Japanese Patent Laid-Open No. 2004-100000 discloses updating detected parking space information using the amount of movement of the vehicle from the immediately preceding processing cycle to the current processing cycle, which is output by the vehicle motion acquisition unit.

JP 2017-87758 A

In the method disclosed in Patent Document 1, information about the traveling direction and speed of the vehicle is acquired via the CAN bus, the amount of movement of the vehicle is obtained from the information, and the current position of the parking space is estimated. With this configuration, the position of the parking space may deviate from the appropriate position when estimation errors in the amount of movement of the vehicle accumulate. Also, due to the effects of lens distortion and the like, white lines cannot be properly detected from a captured image at a position distant from the camera, and a suitable parking space may not be detected.

In view of the above problems, an object of the present invention is to provide a technique capable of improving the reliability of information regarding a parking frame obtained based on an image captured by a camera.

In order to achieve the above object, the image processing apparatus of the present invention includes a recognition unit that recognizes a parking frame based on a captured image obtained by a camera that captures the surroundings of a vehicle, and position information of the parking frame that has been recognized in the past. Based on the observation information including the position information of the parking frame obtained from the current captured image, the management unit updates management information including, based on the management information and the observation information, An update type is classified for each division line that constitutes the parking frame to be managed, and the management information is updated based on the classification (first configuration).

In the image processing apparatus having the first configuration, the management unit performs a process of obtaining estimation information including position information of the parking frame based on the movement information of the vehicle and the management information, and classifies the update type as: , the configuration (second configuration) performed based on the estimation information and the observation information.

In the image processing apparatus having the second configuration, the management unit classifies the lane markings whose position information is included in both the observation information and the estimation information into a first update type, It is preferable that the lane markings containing information are classified into a second update type, and the lane markings containing position information only in the estimated information are classified into a third update type (third structure).

In the image processing apparatus having the third configuration, when updating the management information, the position information of the lane marking of the first update type is updated to the value of the observation information, and the lane marking of the second update type is updated to the value of the observation information. is provisionally registered with the value of the observation information, and the position information of the lane marking of the third update type is a correction value obtained based on the observation information and the estimation information. It is preferable that the configuration (fourth configuration) is updated to the value corrected by .

In the image processing apparatus having the fourth configuration, the management unit collectively corrects the values of the estimated information including the position information of the lane markings of the third update type with the same correction value. (Fifth configuration) is preferable.

In the image processing device having any one of the first to fifth configurations, the camera is a side camera provided on the side of the vehicle, and the recognition unit uses a partial area of the captured image to It is preferable that it is the structure (6th structure) which recognizes the said parking frame.

In order to achieve the above object, an image processing method of the present invention is an image processing method executed by an image processing device, which recognizes a parking frame based on a photographed image obtained by a camera that photographs the surroundings of a vehicle. and a management step of updating management information including the position information of the parking frame recognized in the past based on observation information including the position information of the parking frame obtained from the currently captured image, wherein the management The step includes a process of classifying an update type for each lane line constituting the parking frame to be managed based on the management information and the observation information, and updating the management information based on the classification (second 7 configuration).

ADVANTAGE OF THE INVENTION According to this invention, the reliability of the information regarding a parking frame obtained based on the picked-up image of a camera can be improved.

A diagram showing a configuration example of a parking assistance system Schematic diagram for explaining the search area for lane markings Table showing examples of update types of lane markings that make up parking spaces Flowchart illustrating management processing Schematic diagram for explaining management processing Flowchart showing a specific example of update processing shown in FIG. Schematic diagram showing a comparison between the current recognition image of the lane marking and the past recognition image Schematic diagram for explaining the calculation method of the correction value

Exemplary embodiments of the invention are described in detail below with reference to the drawings. In the following description, the straight traveling direction of the vehicle, which is the direction from the driver's seat to the steering wheel, is referred to as the "forward direction." Also, the direction in which the vehicle travels straight ahead and is directed from the steering wheel toward the driver's seat is called the "rearward direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and directed from the right side to the left side of the driver who is facing forward is referred to as the "left direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and directed from the left side to the right side of the driver who is facing forward is referred to as the "right direction." A vehicle equipped with a parking assistance system including the image processing device of the present invention is referred to as "own vehicle".

<1. Parking assistance system>
The image processing device of the present invention can be applied to, for example, a parking assistance system. FIG. 1 is a diagram showing a configuration example of a parking assistance system to which an image processing device 1 of the present invention is applied. As shown in FIG. 1, the parking assistance system includes an image processing device 1, an imaging unit 2, a parking control ECU (Electronic Control Unit) 3, an EPS (Electric Power Steering)-ECU 4, and an in-vehicle network 5. Prepare.

The image processing device 1 is connected to the imaging unit 2, and is also connected to the parking control ECU 3 and the EPS-ECU 4 via an in-vehicle network 5 such as a CAN (Controller Area Network). The image processing device 1 recognizes a parking frame based on an image captured by the imaging unit 2 and manages the recognized parking frame. That is, the image processing device 1 functions as a parking frame management device. In addition, the image processing device 1 calculates the target parking position, for example, assuming that the vehicle should be parked in an empty parking space that is closest to the vehicle when the vehicle stops. Details of the image processing apparatus 1 will be described later.

A display device such as a liquid crystal display may be connected to the image processing device 1, for example. The image processing device 1 may generate a display image in which an index indicating the target parking position is superimposed on the captured image, and display the image on the display device.

The photographing unit 2 includes a camera for photographing the surroundings of the own vehicle. In this embodiment, the camera has a fisheye lens. The imaging unit 2 includes a side camera 20 provided on the side of the own vehicle. The side camera 20 is included in a camera that captures an image of the surroundings of the vehicle, and is provided, for example, in a door mirror of the vehicle. The side cameras 20 preferably include a left side camera and a right side camera. The left side camera has an optical axis along the left-right direction of the vehicle, and photographs the left direction of the vehicle. The right side camera has an optical axis along the left and right direction of the own vehicle, and photographs the right direction of the own vehicle. In addition to the side cameras 20, the photographing unit 2 preferably includes a front camera for photographing the front of the vehicle and a back camera for photographing the rear of the vehicle.

The parking control ECU 3 calculates the steering amount based on the target parking position received from the image processing device 1 and the output of a clearance sonar sensor (not shown), and transmits information on the steering amount to the EPS-ECU 4 .

The EPS-ECU 4 performs automatic steering during parking operation of the own vehicle based on the information regarding the steering amount received from the parking control ECU 3 . In addition, in this embodiment, the driver is in charge of the accelerator operation and the brake operation.

<2. Image processing device>
As shown in FIG. 1 , the image processing apparatus 1 includes a control section 11 and a storage section 12 . The control unit 11 controls the entire image processing apparatus 1 . The control unit 11 is configured by a computer including, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The storage unit 12 nonvolatilely stores computer programs and data necessary for the operation of the control unit 11 . As the storage unit 12, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, or the like can be used.

The video acquisition unit 111, search unit 112, recognition unit 113, and management unit 114 shown in FIG. In other words, the image processing device 1 includes a video acquisition unit 111 , a search unit 112 , a recognition unit 113 and a management unit 114 .

At least one of the functional units 111 to 114 included in the control unit 11 may be configured by hardware such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). Each functional unit 111 to 114 included in the control unit 11 is a conceptual component. A function performed by one component may be distributed to a plurality of components, or a function possessed by a plurality of components may be integrated into one component.

The video acquisition unit 111 temporally continuously acquires analog or digital captured images from each camera of the imaging unit 2 including the side camera 20 at a predetermined cycle (for example, 1/30 second cycle). Then, when the acquired temporally continuous captured images (acquired video) are analog, the video acquisition unit 111 converts the analog captured images into digital captured images (A/D conversion).

The search unit 112 searches for a lane marking that forms a parking frame based on the image captured by the side camera 20 output from the image acquisition unit 111 . When the side cameras 20 include a left side camera and a right side camera, the search unit 112 searches for lane markings in the images captured by each camera.

The searching unit 112 starts searching for lane markings, for example, when the vehicle speed of the own vehicle becomes equal to or lower than a predetermined speed. In the present embodiment, the search unit 112 detects the lane markings from the image captured by the side camera 20 at intervals of 100 ms, for example, using image processing such as edge extraction. In addition, the division lines are drawn on the paved surface of the parking lot with white lines or yellow lines.

FIG. 2 is a schematic diagram for explaining a search area for lane markings. FIG. 2 shows a state in which a vehicle V has entered a parking lot and a plurality of lane markings SL are present on the left side of the vehicle V. As shown in FIG. Each partition line SL extends in the left-right direction. That is, each partition line SL extends in a direction parallel to the optical axis of the left side camera 20L. The plurality of demarcation lines SL are arranged at predetermined intervals in the front-rear direction. In FIG. 2, a region R surrounded by a dashed line schematically shows a photographing region of the left side camera 20L of the own vehicle V. As shown in FIG.

The image captured by the left side camera 20L has high resolution in the front direction along the optical axis of the left side camera 20L, and the resolution decreases as the distance from the optical axis increases in the front-rear direction due to the effects of lens distortion and the like. For this reason, as shown in the graph in FIG. 2, when the marking line SL is detected from the entire area of the image captured by the left side camera 20L, the reliability of the information on the marking line SL detected from the front direction is high. Information on the marking line SL detected from a position away from the optical axis in the front-rear direction is less reliable. Such a tendency also applies to the case of the right side camera. That is, in the configuration in which the lane marking SL is detected from the entire area of the image captured by the side camera 20, information on the lane marking SL with high reliability and information on the lane marking SL with low reliability may be mixed.

In consideration of this point, in the present embodiment, the search unit 112 searches for the lane marking SL by narrowing down the region from the image captured by the side camera 20 where highly reliable lane marking information can be obtained. It is configured. The search unit 112 searches for the lane marking SL using a narrow area centered in the front direction along the optical axis of the side camera 20 in the image captured by the side camera 20 as a search area. The search area has a size that can contain about three lane markings SL forming the parking frame P, for example.

Returning to FIG. 1 , the recognition unit 113 recognizes the parking frame P based on the information regarding the lane marking SL detected by the search unit 112 . In other words, the recognition unit 113 recognizes the parking frame P based on the captured image obtained by the side camera 20 . The recognition unit 113 obtains the parallelism between the two lane markings SL, the width between the lane markings SL, and the like, and recognizes the parking frame based on these results. For example, the recognition unit 113 recognizes, as the parking frame P, an area between two partition lines SL arranged in parallel with a predetermined interval. The predetermined interval is, for example, the width of a standard parking area for general public use, which is stipulated by laws and regulations related to parking lots.

In the present embodiment, as described above, the search unit 112 searches for the lane marking SL by narrowing down the image captured by the side camera 20 to narrow areas with high image reliability. For this reason, the recognition unit 113 recognizes the parking frame P using a partial area including the central portion of the image captured by the side camera 20 . Specifically, the recognition unit 113 recognizes the parking frame P using an image of a narrow area centered on the front direction along the optical axis of the side camera 20 among the images captured by the side camera 20 . The narrow area may be, for example, an area wide enough to accommodate about one to two parking frames P, and in the present embodiment, the narrow area is an area wide enough to fit about two parking lots P. According to this, the recognition part 113 can recognize the parking frame P narrowing down to the lane marking SL obtained from the area|region where the reliability of an image is high among picked-up images. Therefore, the reliability of the parking frame P recognized by the recognition unit 113 can be improved.

The management unit 114 manages the information regarding the parking frame P recognized by the recognition unit 113 as management information 121 in chronological order. The management information 121 is stored in the storage unit 12 and updated as appropriate. Note that the management unit 114 does not need to manage all the parking spaces recognized by the recognition unit 113 in the past. The information of P may be appropriately excluded from the management targets. As a result, the required storage capacity of the storage unit 12 can be kept low.

In this embodiment, the management unit 114 converts the management information 121 including the position information of the parking frame P recognized in the past into the observation information including the position information of the parking frame P obtained from the currently captured image of the side camera 20. Update based on. Note that the position information of the parking stall P includes the position information of the lane marking SL forming the parking stall P. The management information 121 has position information about the parking frame P and the lane marking SL, for example, as a combination of a plurality of vertex coordinates. Each parking frame P may be managed as a combination of vertex coordinates of the partition lines SL forming the parking frame P.

Specifically, based on the management information 121 and the observation information, the management unit 114 classifies the update type for each lane line SL that constitutes the parking frame P to be managed. The management unit 114 updates the management information 121 based on the update type classification. According to this, it is possible to update each lane marking SL constituting the parking frame P to be managed by an appropriate method according to the classification, and to improve the reliability of the management information 121. .

More specifically, the management unit 114 performs a process of obtaining estimated information including position information of the parking frame P based on the movement information of the own vehicle V and the management information 121 . The estimated information is for estimating to which position the position of each parking frame P included in the management information 121 has moved due to the movement of the host vehicle V. FIG. The estimated information includes information about the lane marking SL that forms the parking frame P. As shown in FIG. The movement information of the own vehicle V is, for example, information such as the steering angle, vehicle speed, and shift of the own vehicle V acquired from the in-vehicle network 5 . In this case, the management unit 114 estimates the amount of movement of the vehicle V from the movement information of the vehicle V, and estimates the current position of each parking frame P recognized in the past based on the estimated amount of movement and the management information 121. do. When information on a plurality of parking spaces P is included in the management information 121, the management unit 114 estimates the current position of each of the plurality of parking spaces P. FIG.

The movement information of the own vehicle V is not limited to the measured information such as steering angle and vehicle speed, and may be information obtained by processing the measured information such as the estimated amount of movement of the own vehicle V, for example. In this case, the estimated amount of movement of the own vehicle V may be obtained by the parking control ECU 3, for example. Then, the management unit 114 estimates the current position of the parking frame P based on the estimated movement amount of the own vehicle V acquired from the parking control ECU 3 and the information on the parking frame P stored in the past as the management information 121. You can

In the present embodiment, the classification of the update type of the lane marking SL forming the parking stall P is performed based on the estimation information and the observation information. According to this, it is possible to update the management information 121 by giving priority to the results of observation information with high reliability, and the reliability of the management information 121 can be improved.

FIG. 3 is a table showing an example of the update type of the lane marking SL forming the parking frame P. As shown in FIG. In FIG. 3 , the observation value “◯” corresponds to the lane marking SL for which the lane marking information exists in the observation information. In FIG. 3, the observation value "x" corresponds to the lane marking SL for which the lane marking information does not exist in the observation information. In FIG. 3 , an estimated value of “◯” corresponds to a lane marking SL for which lane marking information exists in the estimated information. In FIG. 3, an estimated value of "x" corresponds to a lane marking SL for which lane marking information does not exist in the estimated information.

A lane marking SL for which lane marking information exists in the observed information and estimated information has an observed value of "o" and an estimated value of "o", and is classified as the first update type. A lane marking SL for which the observation information contains the lane marking information but the estimation information does not contain the lane marking information has an observed value of "o" and an estimated value of "x", and is classified into the second update type. A lane marking SL with no lane marking information in the observation information but with lane marking information in the estimated information has an observed value of "x" and an estimated value of "○", and is classified as the third update type. . Note that a lane marking SL for which lane marking information does not exist in the observation information and the estimation information cannot be processed and therefore is not given an update type.

That is, the management unit 114 classifies the lane marking SL whose position information is included in both the observation information and the estimation information into the first update type, and classifies the lane marking SL whose position information is included only in the observation information into the second update type. , and the lane marking SL whose position information is included only in the estimated information is classified into the third update type. According to this, the position information of the plurality of lane markings SL acquired in time series can be appropriately classified, and the management information 121 of the parking frame P can be updated appropriately.

In this embodiment, the management unit 114 updates the management information 121 based on these three update types. FIG. 4 is a flowchart illustrating management processing executed by management unit 114 . It should be noted that the management processing shown in FIG. 4 will be explained with appropriate reference to the schematic example shown in FIG. 5 in order to facilitate understanding.

FIG. 5 is a schematic diagram for explaining management processing. The own vehicle V which has entered the parking lot passes through the respective lane marking lines SL1 to SL10 in the order of the first lane marking SL1, the second lane marking SL2, the third lane marking SL3, . . . . As the host vehicle V moves forward, the recognition unit 113 sequentially recognizes the lane markings SL1 to SL10 as lane markings forming the parking frame P based on the captured images sequentially acquired from the left side camera 20L.

In the example shown in FIG. 5, the first lane marking SL1 to the ninth lane marking SL9 are already recognized as the lane marking SL forming the parking frame P by the recognition unit 113, and the lane marking information of these lane markings is included in the management information 121. is That is, the management unit 114 can obtain estimated information about the first lane marking SL1 to the ninth lane marking SL9. Further, the eighth lane SL8 to the tenth lane SL10 are recognized as the lane markings forming the parking frame P from the image currently captured by the left side camera 20L.

The management processing (management process) by the management unit 114 is started, for example, when the recognition unit 113 completes the recognition processing of the parking frame P for the current captured image. First, the management unit 114 acquires observation information (step S1). In the example shown in FIG. 5, the management unit 114 acquires information on three lane markings SL8 to SL10, 8th to 10th, in step S1.

Next, the management unit 114 acquires estimation information (step S2). The management unit 114 calculates an estimated amount of movement of the own vehicle V from the time when the captured image immediately before the current captured image is obtained to the time when the current captured image is obtained. Then, the management unit 114 obtains estimated information based on the estimated amount of movement and the management information 121 currently stored in the storage unit 12 . In the example shown in FIG. 5, the management unit 114 obtains the estimated positions of the first to ninth lane markings SL1 to SL9 in step S2. Note that step S2 may be performed prior to step S1, or may be performed in parallel with step S1.

After acquiring the observation information and the estimation information, the management unit 114 classifies each lane marking SL according to the update type (step S3). Specifically, the management unit 114 determines to which of the first update type, the second update type, and the third update type each lane marking SL belongs.

In the example shown in FIG. 5, the eighth lane marking SL8 and the ninth lane SL9 correspond to lane markings for which lane marking information exists in the observation information and the estimation information. Therefore, the eighth lane marking SL8 and the ninth lane marking SL9 are classified into the first update type. The tenth lane line SL10 corresponds to a lane line for which lane line information exists in the observation information but does not exist in the estimated information. Therefore, the tenth lane marking SL10 is classified as the second update type. The first to seventh lane markings SL1 to SL7 correspond to lane markings for which the observation information has no lane marking information but the estimation information has lane marking information. Therefore, the first to seventh lane marking lines SL1 to SL7 are classified into the third update type.

Note that the management unit 114 determines whether each lane marking SL included in the observation information is the same as any of the lane marking SL recognized in the past based on the position obtained from the observation information and the position obtained from the estimated information. is determined by comparing In the present embodiment, even if the position obtained from the observation information and the position obtained from the estimation information do not completely match, if the position difference is within a predetermined threshold, it is determined to be the same lane marking SL. to decide.

After classifying the update type of each lane marking SL, the management unit 114 updates the management information 121 currently stored in the storage unit 12 based on the classification result (step S4). When updating the management information 121, the position information of the lane marking SL of the first update type is updated to the value of the observation information (observation update). Also, the position information of the lane marking SL of the second update type is provisionally registered with the value of the observation information (new registration). At the stage of provisional registration, the lane marking SL is not treated as a lane marking SL recognized in the past in the classification of the update type, nor is it used to calculate a correction value, which will be described later. When provisional registration is performed multiple times, the lane marking SL is officially registered. Thereby, the reliability of the management information 121 can be further improved. However, a configuration may be adopted in which the first observation is not temporarily registered but is officially registered. In this case, the first update type and the second update type may not be divided into two, but may be classified as the same. Furthermore, the position information of the lane marking SL of the third update type is updated to a value obtained by correcting the value of the estimation information with a correction value obtained based on the observation information and the estimation information.

In the example shown in FIG. 5, the management unit 114 updates the position information of the eighth lane marking SL8 and the ninth lane SL9 to the value of the observation information. The management unit 114 provisionally registers the position information of the tenth lane marking SL10 with the value of the observation information. The management unit 114 updates the position information of the first to seventh lane marking lines SL1 to SL7 with values obtained by correcting each estimated value with a correction value. By these updates, the update of the management information 121 including the information of the parking frame P is completed.

In the present embodiment, observation information can be obtained using a highly reliable region of the captured image, so the reliability of the lane marking information included in the observation information can be increased. That is, it is possible to improve the reliability of the update information of the lane markings of the first update type and the second update type that are updated with the observed values. Then, regarding the position information of the lane marking SL of the third update type that cannot be recognized from the captured image and needs to be updated with the result estimated from the movement amount of the own vehicle V, the estimated value is corrected with the correction value obtained from the observation information. to obtain update information. For this reason, it is possible to prevent the lane markings of the third update type from deviating from their original positions due to accumulation of errors in estimating the amount of movement of the vehicle during the updating process. That is, according to this embodiment, the reliability of the management information 121 of the parking frame P can be improved.

A specific example of the update process (step S4) shown in FIG. 4 will be described. FIG. 6 is a flowchart illustrating a specific example of the update processing shown in FIG. 4; After classifying the update types of each lane marking SL, the management unit 114 confirms whether or not the first update type exists in the classified update types (step S41). In the example shown in FIG. 5, since the two lane markings SL8 and SL9 are classified into the first update type, the management unit 114 determines that the first update type exists.

When determining that the first update type exists (step S41 Yes), the management unit 114 calculates a correction value (step S42). A correction value is used to correct the estimated information. A method of calculating the correction value will be described with reference to FIGS. 7 and 8. FIG.

FIG. 7 is a schematic diagram showing a comparison between the current recognition image I(t) of the lane marking SL and the past recognition image I(t−1). In FIG. 7, the solid line is the current recognition image I(t), and the dashed line is the past recognition image I(t-1). The current recognition image I(t) is acquired from the image captured by the left side camera 20L of the vehicle V(t), and the past recognition image I(t−1) is the image of the vehicle V(t− This is obtained from the captured image of the left side camera 20L in 1). The past in FIG. 7 refers to the time one cycle (the imaging cycle of the camera) before the present. As shown in FIG. 7, due to the movement of the own vehicle V, the position of the lane marking SL recognized by the captured image also moves. Also, the type of lane marking SL recognized by the photographed image changes as the vehicle V moves. That is, in the past recognition image I(t−1), the 7th to 9th division lines SL7 to SL9 were recognized, but in the present recognition image I(t), the 8th to 10th division lines Three demarcation lines SL8-SL10 are recognized.

FIG. 8 is a schematic diagram for explaining a method of calculating correction values. FIG. 8 is a diagram obtained by extracting only the current recognition image I(t) of FIG. 7 and adding items necessary for explaining the method of calculating the correction values. When calculating the correction value, the management unit 114 compares the lane marking information included in the observation information with the lane marking information included in the estimated information.

The example shown in FIG. 8 will be described specifically. The management unit 114 calculates the current position (t) and the estimated position (position indicated by the dashed-dotted line) estimated from the position of the current (t) and the position of one cycle before (t−1) for the eighth lane SL8 and the ninth lane SL9. are compared to calculate the positional difference value Δ. The management unit 14 uses this difference value Δ as a correction value.

The difference value Δ obtained using the eighth marking line SL8 and the difference value Δ obtained using the ninth marking line SL9 may or may not be the same. When a plurality of difference values Δ having different values are acquired, for example, the average value of the plurality of difference values may be used as the correction value. As another method, a difference value Δ that is determined to be highly reliable among a plurality of difference values Δ may be used as the correction value. For example, it is estimated that the lane marking information recognized at the center of the captured image is more reliable than the lane marking information recognized at a position shifted from the center. For this reason, the lane marking information recognized from a position closer to the central position of the captured image may be regarded as having a high degree of reliability, and the difference value Δ obtained from the lane marking information may be used as the correction value. In this example, the number of difference values to be obtained is two, but this number may be changed as appropriate, and may be one or three or more.

After calculating the correction value, the management unit 114 performs a correction process (step S43). Specifically, the management unit 114 sets the position information of the lane marking SL of the third update type to a value obtained by correcting the value of the estimated information with a correction value. In the example shown in FIG. 8, it is determined that the estimated movement amount M of the host vehicle V is estimated to be smaller than the actual amount, and the estimated movement amount M is corrected to be increased.

Note that in the present embodiment, the management unit 114 collectively corrects the values of the estimated information including the position information of the plurality of lane markings SL of the third update type with the same correction value. In the example shown in FIG. 5, the management unit 114 collectively corrects the estimated information values with the same correction value for the position information of the first to seventh marking lines SL1 to SL7. According to this, the lane marking SL of the third update type for which predictive updating is performed can be efficiently corrected to a highly reliable value and updated.

After performing the correction process, the management unit 114 updates each lane marking information to an update value determined according to each update type, and updates and registers the management information 121 regarding the parking frame P (step S44). Since the method of obtaining the update value determined according to each update type has been described above, the description is omitted here.

By the way, if it is determined in step S41 that the lane marking SL of the first update type does not exist, the correction value cannot be calculated based on the observation information. Therefore, when it is determined that the lane marking SL of the first update type does not exist (No in step S41), the management unit 114 sets the correction value to zero and proceeds to step S43. In other words, the management unit 114 does not substantially perform the estimation information correction process when it is determined that the lane marking SL of the first update type does not exist.

Note that a state in which no lane marking SL of the first update type exists is a case where there is no recognition information of the lane marking SL in the observation information, or only newly recognized lane marking information exists in the observation information. Sometimes. The former is, for example, when the own vehicle V passes through the area where the parking frame P exists, when an object (stationary object or moving object) exists between the lane marking SL and the side camera 20, or when the lane marking SL is For example, the lane marking SL cannot be recognized because it is covered with shadows or the lane marking SL is about to disappear. The latter is, for example, when the vehicle V has just started approaching the existing area of the parking frame P, when an object exists between the lane marking SL and the side camera 20, or when the lane marking SL is in the shadow. For example, a part of the marking lines SL cannot be recognized because they are covered or the marking lines SL are about to disappear.

Further, even if it is determined that the lane marking SL of the first update type exists, the lane marking SL that should be recognized in the current captured image may not be recognized due to the presence of an object, shadow, or the like. However, due to the presence of the lane marking SL of the first update type, such a lane marking SL can be updated and registered with a value obtained by correcting the estimated value obtained by estimating the movement amount of the vehicle with a correction value.

<3. Notes>
Various modifications can be made to the various technical features disclosed in this specification without departing from the gist of the technical creation in addition to the above-described embodiments. That is, the above embodiments should be considered as examples in all respects and not restrictive, and the technical scope of the present invention is not defined by the description of the above embodiments, but by the scope of claims. All changes that come within the meaning and range of equivalency of the claims are to be understood. In addition, multiple embodiments and modifications shown in this specification may be implemented in combination to the extent possible.

In the above, the case where the recognition unit 113 uses the side camera 20 to recognize the parking frame P has been described as an example. It can also be applied to a configuration in which the parking frame is recognized by using the parking frame.

1 image processing device 20 side camera (an example of a camera)
113... Recognition part 114... Management part 121... Management information P... Parking slot SL... Section line V... Own vehicle (vehicle)

Claims (7)

A recognition unit that recognizes a parking frame based on a captured image obtained by a camera that captures the surroundings of the vehicle;
A management unit that updates management information including position information of the parking frame recognized in the past based on observation information including position information of the parking frame obtained from a currently captured image;
with
The management unit classifies an update type for each lane line forming the parking frame to be managed based on the management information and the observation information, and updates the management information based on the classification. . The management unit performs a process of obtaining estimated information including position information of the parking frame based on the movement information of the vehicle and the management information,
2. The image processing apparatus according to claim 1, wherein said update type classification is performed based on said estimation information and said observation information. The management department
Classifying the lane markings for which position information is included in both the observation information and the estimation information as a first update type;
classifying the lane markings for which position information is included only in the observation information into a second update type;
3. The image processing apparatus according to claim 2, wherein said lane markings whose position information is included only in said estimation information are classified into a third update type. When updating the management information,
the position information of the lane marking of the first update type is updated to the value of the observation information;
the position information of the lane marking of the second update type is provisionally registered with the value of the observation information;
The position information of the lane marking of the third update type according to claim 3, wherein the value of the estimated information is updated to a value obtained by correcting the value of the estimated information with a correction value obtained based on the observation information and the estimated information. Image processing device. 5. The image processing apparatus according to claim 4, wherein the management unit collectively corrects the values of the estimated information including the position information of the lane markings of the third update type with the same correction value. The camera is a side camera provided on the side of the vehicle,
The image processing device according to any one of claims 1 to 5, wherein the recognition unit recognizes the parking frame using a partial area including a central portion of the captured image. An image processing method executed by an image processing device,
A recognition step of recognizing a parking frame based on a captured image obtained by a camera that captures the surroundings of the vehicle;
A management step of updating management information including position information of the parking frame recognized in the past based on observation information including position information of the parking frame obtained from a currently captured image;
with
In the management step, based on the management information and the observation information, an update type is classified for each lane line that constitutes the parking frame to be managed, and the management information is updated based on the classification. Image processing method.

Images