JP2022113526A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To detect a parking area accurately regardless of environment.SOLUTION: An image processing apparatus 100 includes: an image synthesis unit 120 which synthesizes captured images obtained in a camera to generate a synthesized image; a map creation unit 131 which calculates a hue value of the synthesized image to create a hue map; a cumulative map creation unit 132 which accumulates hue values of hue maps for each predetermined area, in a predetermined range, for a predetermined period, to create a cumulative map 220; a parking area extraction unit 134 which extracts a parking area 300 from the cumulative map 220; and an output processing unit 140 which outputs the parking area 300.SELECTED DRAWING: Figure 2

Description

The present invention relates to image processing technology for processing color images.

There is technology for detecting white lines in a parking lot using an image captured by an in-vehicle camera. For example, in Patent Document 1, "Based on the detection result of the first white line detection unit, the first parking frame detection unit detects the parking frame in the front bird's-eye view image, and refers to the detection result of the first parking frame detection unit. Then, based on the detection results of the first white line detection unit and the second white line detection unit, the front bird's-eye view image and the side bird's-eye view image are detected by the second white line detection unit. A technique is disclosed in which a second parking frame detection unit detects a parking frame existing over an overhead image (summary excerpt).

There is also an image processing technique for detecting yellow lines. For example, in Patent Document 2, "In the image processing apparatus, the line segment extraction unit includes a calculation unit, a correction unit, and an extraction unit. A saturation is calculated based on the difference between the R value or the G value and the B value.The correction unit corrects the luminance value of the pixel that satisfies a predetermined yellow condition using the calculated saturation as an enhancement amount. , Detecting marking lines displayed on the road surface based on corrected luminance values (summary excerpt)” technology is disclosed.

JP 2014-106738 A JP 2020-095621 A

The technique described in Patent Literature 1 recognizes a parking frame based on the luminance difference between the road surface and the parking frame line. For this reason, it is difficult to recognize in an environment with little luminance difference, for example, when it is cloudy, at night, indoors, or the like.

The technology described in Patent Literature 2 considers color information and recognizes a parking frame. However, this technology adds the strength of the yellow component to the base luminance, and the recognition target is limited to general yellow (orange) parking frame lines.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a technique for accurately detecting a parking area regardless of the environment.

The image processing apparatus of the present invention includes an image synthesizing unit that synthesizes captured images obtained by a camera to generate a synthesized image, and a map creation unit that calculates the hue value of the synthesized image and creates a hue map. an accumulated map creation unit for creating an accumulated map by accumulating the hue values of the hue map for each predetermined area in a predetermined range for a predetermined period; and a parking area for extracting a parking area from the accumulated map. An area extraction unit and an output processing unit that outputs the parking area are provided.

According to the present invention, the parking area can be detected with high accuracy regardless of the environment. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing for demonstrating the outline|summary of embodiment of this invention. 1A and 1B are a hardware configuration diagram and a functional block diagram, respectively, of an image processing apparatus according to an embodiment of the present invention; FIG. It is a flowchart of parking area identification processing of the embodiment of the present invention. It is an explanatory view for explaining map creation processing of an embodiment of the present invention. (a) and (b) are explanatory diagrams for explaining an accumulation map and data registered in association with each grid of the accumulation map, respectively, according to the embodiment of the present invention. FIG. 5 is an explanatory diagram for explaining accumulation map creation processing according to the embodiment of this invention; (a) is a flowchart of certainty calculation processing according to the embodiment of the present invention, and (b) and (c) respectively explain an example of a brightness increase/decrease value table and a hue increase/decrease value table of the embodiment of the present invention. It is an explanatory view for doing. It is a flow chart of parking area extraction processing of the embodiment of the present invention. 4(a) and 4(b) are flowcharts of a colored portion extraction process and an achromatic part extraction process, respectively, according to the embodiment of the present invention. It is an explanatory view for explaining accumulation map creation processing of a modification of the present invention. (a) and (b) are explanatory diagrams for explaining a parking area extraction method according to a modification of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the present invention is not limited to this embodiment, and all application examples consistent with the idea of the present invention are included in its technical scope. Also, unless otherwise specified, each component may be singular or plural.

First, an outline of this embodiment will be described. FIG. 1 is a diagram for explaining the outline of this embodiment. In this embodiment, every time an image (captured image) is acquired by the camera mounted on the vehicle 910, the captured image is synthesized and converted to generate the bird's-eye view image 210. FIG. A bird's-eye view image 210 is an image that looks down on the surroundings of the vehicle from above the vehicle 910 . Then, the hue value and brightness value of each pixel 211 and 212 of the overhead image 210 are calculated.

After that, hue values and lightness values in a predetermined range for a predetermined period are accumulated for each predetermined area (grid 222) to generate an accumulation map 220 for the predetermined range. At this time, the reliability of the accumulated value of each grid 222 may be calculated. Then, using these accumulated values, a parking area 300 within a predetermined range is extracted. In this embodiment, the parking area 300 includes the area surrounded by the parking frame line 310 and the parking space 320 .

In this embodiment, the above processing is realized by the image processing device 100 mounted on the vehicle. First, the image processing apparatus 100 of this embodiment will be described.

[Hardware configuration]
FIG. 2A is a hardware configuration diagram of the image processing apparatus of this embodiment. As shown in the figure, the image processing apparatus 100 of this embodiment includes a CPU (Central Processing Unit) 111, a memory 112, a storage device 113, an input device 114, an output device 115, a data I/F 116, and a , provided. Furthermore, a communication I/F or the like may be provided. The image processing apparatus 100 is implemented by, for example, a general-purpose information processing apparatus (computer).

A CPU 111 is a main control unit (main processor) that controls the entire image processing apparatus 100 according to a predetermined program. It may be implemented in a microprocessor unit (MPU).

A memory 112 is a temporary storage area and a work area. For example, a RAM (Random Access Memory) or the like is used. The RAM may be integrated with the CPU 111 .

The storage device 113 stores data used by the CPU 111 for processing, data generated by the processing, and the like. A ROM (Read Only Memory), a flash memory, a hard disk, or the like is used.

The input device 114 receives input of operation instructions. Operation keys such as a power key, a volume key, a home key, and a touch sensor may be provided. It should be noted that the touch sensor is arranged as a touch panel so as to overlap a display, which will be described later. It should be noted that a microphone may be used when receiving voice input.

The output device 115 outputs the processing result. For example, a display, a speaker, a lamp, etc. may be provided.

The data I/F 116 is an interface that transmits and receives data to, for example, an ECU (Electronic Control Unit) in the vehicle, and receives data from the vehicle-mounted camera 191 (see FIG. 2B).

In addition, the vehicle-mounted camera 191 includes an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and photographs the surroundings of the vehicle 910 in which the camera is mounted.

The number of in-vehicle cameras 191 is not limited as long as they can photograph 360 degrees around the vehicle. For example, four cameras each having a horizontal angle of view of 90 degrees or more may photograph the front, rear, left side, and right side of the vehicle. Alternatively, one 360-degree camera (omnidirectional camera) may be used. Hereinafter, in this embodiment, a case in which a plurality of in-vehicle cameras 191 are provided will be described as an example.

[Function block]
In this embodiment, the image processing device 100 identifies the parking area 300 around the vehicle 910 using the captured image acquired by the vehicle-mounted camera 191 as described above. The functional configuration of the image processing apparatus 100 of this embodiment that realizes this will be described below. FIG. 2B is a functional block diagram of the image processing apparatus 100 of this embodiment.

The image processing apparatus 100 includes an image synthesizing section 120 , an arithmetic processing section 130 and an output processing section 140 . Arithmetic processing unit 130 includes map creating unit 131 , accumulated map creating unit 132 , and parking area extracting unit 134 . The accumulation map creating unit 132 also includes a certainty calculating unit 133 .

The image synthesizing unit 120 performs an image synthesizing process of synthesizing the captured images acquired by the plurality of in-vehicle cameras 191 and generating a synthesized image (overhead image 210). The bird's-eye view image 210 is synthesized by connecting the photographed images after performing coordinate transformation according to the photographing direction. Various well-known processes can be used for this process. The image synthesizing process is performed at predetermined intervals, for example, each time a photographed image is input from each vehicle-mounted camera 191 .

Note that the synthesized image generated here is not limited to the overhead image 210 . Any image may be used as long as the surroundings of the vehicle 910 can be grasped.

Note that the overhead image 210 includes RGB color information (hereinafter referred to as RGB values) for each pixel. Although the format of the color information depends on the image output format of the camera, here it is assumed that each vehicle-mounted camera 191 outputs the color information in RGB format.

The calculation processing unit 130 performs predetermined calculations on the bird's-eye view image 210 to recognize the parking area 300 around the vehicle.

Map creator 131 creates a hue map and a lightness map from bird's-eye view image 210 created by image synthesizer 120 . The map creation unit 131 converts the RGB values of each pixel of the latest bird's-eye view image 210 into the HSV format, and creates a hue map and a brightness map of the region corresponding to the bird's-eye view image 210 .

Note that HSV is a method of expressing a color by three elements of Hue (H), Saturation, and Value/Brightness (V). In the present embodiment, hue (H) and brightness (V) are used among these. Therefore, the hue map maps the color information of the range of the bird's-eye view image 210 around the vehicle 910, and the brightness map maps the brightness of the same range.

Accumulation map creation unit 132 creates an accumulation map 220 by accumulating the hue values of the hue map and the brightness values of the brightness map created by map creation unit 131 in units of predetermined regions (grids). Note that the accumulation map 220 is obtained by accumulating hue values and lightness values of grids corresponding to the same position on the real space using a predetermined number of hue maps and lightness maps that go back a predetermined period from the current time. Accumulation is performed for a predetermined range wider than the bird's-eye view image 210 .

The certainty calculation unit 133 calculates the certainty of each grid of the accumulation map 220 . Here, the degree of certainty is the degree of certainty of the value of the grid.

The parking area extraction unit 134 extracts two types of parking areas 300 from the accumulation map 220 including the certainty generated by the accumulation map generation unit 132 . Here, a parking area 300 defined by a parking space 320, which is a predetermined plane area, and a parking area 300 defined by a pair of parking frame lines 310 are extracted.

The output processing unit 140 generates an output image of the parking area 300 extracted by the parking area extraction unit 134 and outputs the image from the output device 115 . Alternatively, it outputs to the ECU via the data I/F 116 . For example, the ECU may use this information for automatic parking control or the like.

Each function of the image processing apparatus 100 is implemented by, for example, the CPU 111 loading a program stored in the storage device 113 into the memory 112 and executing the program. A part of the image processing apparatus 100 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[Parking area identification process]
FIG. 3 shows the flow of parking area identification processing by the arithmetic processing unit 130 of the present embodiment. This processing is executed each time the image synthesizing unit 120 synthesizes images to generate the bird's-eye view image 210 .

First, the map creation unit 131 executes map creation processing (step S1101). Thereby, the map creating unit 131 creates a hue map in which the pixel values at each pixel position of the bird's-eye view image 210 are the hue values, and a lightness map in which the lightness values are the lightness values. Details of the map creation process will be described later.

Next, the accumulation map creation unit 132 executes accumulation map creation processing for creating the accumulation map 220 (step S1102). Details of the accumulation map creation process will be described later. At this time, the certainty calculation unit 133 calculates the certainty of each grid.

Then, the parking area extraction unit 134 extracts the parking area 300 from the accumulation map (step S1103), and ends the process.

Note that the output processing unit 140 uses the parking area 300 extracted by the arithmetic processing unit 130 as an output image and displays it on a display or the like. In this embodiment, for example, the parking area 300 is superimposed on the original overhead image 210 and displayed.

The details of each processing of the above arithmetic processing will be described below.

[Create map]
The map creation unit 131 creates a hue map in which each pixel value is an (H) value and a brightness map in which each pixel value is a brightness (V) value from the bird's-eye view image 210 created by the image synthesis unit 120 and in which each pixel value is an RGB value. do. As shown in FIG. 4, the hue map and the brightness map are obtained by mapping color information and brightness information around the vehicle 910 on the bird's-eye view image 210 .

なお、各画素において、ＲＧＢの３つの値のうち、最大のものをＭＡＸ、最小のものをＭＩＮとする。 As described above, each pixel of the overhead image 210 has an RGB value. The conversion formula for calculating the HV value from the RGB values is as shown in formula (1) below.

Note that for each pixel, among the three values of RGB, the maximum value is MAX and the minimum value is MIN.

For example, when the pixel of the overhead image 210 corresponding to the pixel 211 in FIG. 4 is (R, G, B)=(128, 128, 128), the hue value and brightness value of the pixel 211 are )=(undefinable, 128). Similarly, when the pixel of the overhead image 210 corresponding to the pixel 212 is (R, G, B)=(140, 170, 220), the hue value and brightness value of the pixel 212 are (H, V)=(159). .5,220).

A portion (MIN=MAX) where the hue cannot be calculated is recorded on the hue map as an undefinable region. Here, the undefinable pixels are pixels whose RGB values are all the same, that is, grayscale (achromatic) pixels such as white, gray, and black. As described above, in the present embodiment, even if the hue (H) cannot be calculated, the information is held as H=impossible to define in the hue map.

[Create accumulation map]
The accumulation map 220 is a map that accumulates and holds hue values and brightness values of hue maps and brightness maps that have been created in the past. The accumulation map 220 is a map having a two-dimensional grid, and as shown in FIG. 5A, the map is larger in size than the hue map and the brightness map. Each grid may correspond to the pixels of the original bird's-eye view image 210 of the hue map and the lightness map, or may include a plurality of pixels. Hereinafter, when there is no particular need to distinguish between a hue map and a lightness map, they will be referred to as a hue map or the like.

In the present embodiment, for example, the accumulation map 220 is twice as large in height and width as the hue map (overhead image 210). For example, if the hue map or the like is a rectangle centered on the vehicle 910 and 8 m on a side, the accumulated map 220 is also a rectangle centered on the vehicle 910 and 16 m on a side.

Note that the size ratio between the accumulation map 220 and the hue map or the like (overhead image 210) is not limited to this. It is sufficient if the size of the accumulation map 220 is larger than the hue map or the like. Also, the size of the accumulation map 220 may be a fixed value, or may be changed according to the parking area 300 to be extracted.

Each grid 221 of the accumulation map 220, as shown in FIG. 5B, holds information on hue accumulation value, lightness accumulation value, and certainty. The hue accumulation value and the lightness accumulation value are accumulated hue and lightness values, respectively, for a predetermined period of time in the past corresponding to the same position in the physical space, such as a hue map of a predetermined region. The number of accumulations is determined in advance.

The degree of certainty is calculated by judging the state of the current hue accumulated value or lightness accumulated value based on past reliability information.

Vehicle-mounted camera 191 is mounted on vehicle 910 . Therefore, as shown in FIG. 6, the area photographed by the vehicle-mounted camera 191 changes as the vehicle 910 moves. That is, the photographing ranges of the photographed images are different, and the range of the physical space covered by the bird's-eye view image 210 generated from the photographed images is also different. Therefore, the range of the hue map etc. corresponding to the bird's-eye view image 210 is also different.

FIG. 6 shows a map 210A and accumulated map 220A when vehicle 910 is at position A, and a map 210B and accumulated map 220B when vehicle 910 is at position B, as an example. In this way, as the vehicle 910 moves from position A to position B, the range of the real space covered by the bird's-eye view image 210 corresponding to the map and the range of the real space covered by the accumulated map 220 accordingly change.

The pixel position of each pixel of bird's-eye view image 210, which is the source of the hue map and the brightness map, is specified in a local coordinate system fixed to vehicle 910 (hereinafter referred to as vehicle coordinate system). The vehicle coordinate system is, for example, a coordinate system whose origin is the center of gravity of the vehicle 910 .

Accumulation map creation unit 132 converts the coordinate values of each pixel of the hue map and the lightness map in the local coordinate system into coordinate values in the world coordinate system fixed in space. The world coordinate system is, for example, a coordinate system whose origin is a specific position on a predetermined road surface. The world coordinate system may be, for example, latitude and longitude.

The conversion is performed using, for example, vehicle information acquired via the ECU and information from the vehicle-mounted camera 191 . Specifically, using the vehicle information such as acceleration information, speed information, etc., the mounting position of the vehicle-mounted camera 191, the imaging field of view, etc., the change in the attitude of the vehicle 910, the amount of movement, the imaging range in the real space, etc. Calculate Then, using it, the coordinate position of each pixel in the latest bird's-eye view image 210 (that is, the hue map and the lightness map) is specified in a predetermined world coordinate system.

Then, the hue value and the lightness value are accumulated as the hue accumulation value and the lightness accumulation value, respectively, in the grids of the accumulation map that include the same coordinate values in the world coordinate system. When accumulating, the information may be accumulated in association with the time information.

Note that accumulation is performed in units of grids containing one or more predetermined pixels. When one grid includes a plurality of pixels, statistical values of the values (hue value, brightness value) of the plurality of pixels forming each grid are used. A statistical value is, for example, an average value.

For example, in the example of FIG. 6, assuming that only the maps 210A and 210B have been calculated at that time, no values are registered in the grid 225 . Also, the values of the map 210A are registered in the grid 226. FIG. The grid 227 registers the values of the map 210A and the values of the map 210B. The values of the map 210B are registered in the grid 228. FIG.

[Confidence calculation process]
Each grid in the accumulation map 220 also accumulates a certainty. The certainty calculation processing by the certainty calculation unit 133 will be described below.

FIG. 7A shows the flow of the certainty calculation process. The certainty calculation process calculates the certainty for each grid. For example, the initial value of the certainty is 0, and the value is increased or decreased according to the variance of the hue value or lightness value of the grid. Note that the certainty calculation process is executed each time the accumulation map 220 is updated. The certainty factor may be calculated for all the grids of the accumulation map 220 each time it is updated, or the certainty factor may be calculated only for the grids whose values have been updated. Confidence calculation processing for a specific grid (referred to as a target grid) will be described below.

First, the certainty calculation unit 133 determines the hue accumulated value of the target grid. Here, it is determined whether or not all of them cannot be defined (step S1201).

If everything is undefinable, that is, if the area has been achromatic all the time since the start of accumulation (S1201; Yes), the lightness accumulation value is used to decide whether to increase or decrease the degree of certainty. Here, first, the variance of the brightness accumulated value (brightness variance value) is calculated (step S1202).

Then, the increase/decrease value of the confidence factor is determined according to the calculated brightness variance value (step S1203). Increase/decrease values corresponding to brightness variance values are defined in advance as a brightness increase/decrease value table. Here, an example of the brightness increase/decrease value table 410 is shown in FIG. 7(b). For example, using a predetermined first threshold, if the brightness variance value is less than the first threshold, it is increased by the first fixed value, and if it is equal to or greater than the first threshold, it is decreased by the first fixed value. Therefore, here, as the increase/decrease value, if the variance value is less than the predetermined first threshold value, plus the first fixed value, and if the variance value is equal to or greater than the first threshold value, minus the first fixed value is decided.

That is, in the present embodiment, when the brightness variance value is less than the first threshold value, it is determined that the brightness is stable, and the degree of certainty is accordingly increased. Conversely, if it is equal to or greater than the first threshold, the degree of certainty is reduced because the lightness is unstable. The first threshold for determining the increase/decrease is determined in advance.

Then, using the determined increase/decrease value, the latest confidence factor is determined (step S1204). Here, the increase/decrease value determined in step S1203 is applied to the current certainty factor. For example, if the brightness variance value is less than a predetermined first threshold value, a predetermined first fixed value is added to the current confidence value. On the other hand, if the variance is greater than or equal to the first threshold, the first fixed value is subtracted from the current certainty.

Also, if there is a value that is not undefinable even once in the past (S1201; No), the accumulation value of that hue is used to determine the increase or decrease of the degree of certainty. Here, first, the variance of the hue accumulation value (hue variance value) is calculated (step S1205).

Then, an increase/decrease value is determined by combining the calculated hue dispersion value and the number of accumulation values that cannot be defined (step S1206). These are defined in advance as the hue increase/decrease value table 420 . For example, classification is performed according to whether the number of undefinable items is less than a predetermined threshold value, and the increase/decrease value is determined depending on whether the hue dispersion value is less than the predetermined threshold value.

An example of the hue increase/decrease value table 420 is shown in FIG. For example, if the number of undefinable accumulated values is less than a fourth threshold, the hue variance value is compared with a second threshold. If it is less than the second threshold, it is increased by a predetermined second fixed value. On the other hand, if it is equal to or greater than the second threshold, it is decreased by the second fixed value. Also, when the number of undefinable accumulated values is equal to or greater than the fourth threshold, the hue variance value is compared with the third threshold. If it is less than the third threshold, it is increased by a predetermined third fixed value. On the other hand, if it is equal to or greater than the third threshold, it is decreased by the third fixed value.

Here, between the second fixed value and the third fixed value, the second fixed value is set larger. This is to make the degree of influence on confidence when the number of undefinables is small larger than that when the number of undefinables is large. As for dispersion, the reason is the same as for brightness. The second threshold, third threshold, and fourth threshold are determined in advance.

Then, the process proceeds to step S1204, the determined increase/decrease value is reflected in the current certainty factor, and the process ends.

[Parking frame extraction process]
Next, parking area extraction processing by the parking area extraction unit 134 will be described. The parking area extraction unit 134 extracts the parking area 300 from the created accumulation map 220 . In the present embodiment, an area obtained by combining two parking frame lines 310 and a parking space 320 shown in FIG.

A flow of parking area extraction processing by the parking area extraction unit 134 is shown in FIG. This processing is performed, for example, each time the accumulation map 220 is updated.

The parking area extraction unit 134 first classifies each grid of the accumulation map 220 into a colored portion or an achromatic portion (step S1301). Here, a portion whose hue accumulation values are all undefinable is classified as an achromatic portion, and other portions, that is, portions with hue accumulation values other than undefinable, are classified as colored portions.

Next, a chromatic portion extraction process (step S1302) is performed, and then an achromatic portion extraction process (step S1303) is performed. In the colored portion extraction processing, the parking area is specified in the colored portion, and its coordinates and type (shape) are acquired. Further, in the achromatic portion extraction processing, the parking area is specified by the achromatic portion, and its coordinates and type (shape) are acquired.

Now, the process of extracting the colored portion will be described. FIG. 9(a) is a processing flow of the colored portion extraction processing by the parking area extraction unit 134 of the present embodiment.

First, the parking area extraction unit 134 scans the latest accumulation map 220 (step S1401). Then, a point (hue change area) where the tendency of hue changes is extracted (step S1402). The hue change areas are, between (adjacent) grids arranged in the scanning direction, points where the color changes from achromatic to chromatic (boundary points; boundary areas), places where the color changes to achromatic (boundary points), and In the change from coloring to coloring, it is either a point (hue change point) where the amount of change in hue value is equal to or greater than a threshold value.

Here, the parking area extraction unit 134 identifies the boundary points according to the above classification. On the other hand, regarding the hue change point, for example, the average value of the accumulated values of a specific grid is compared with the same average value of grids adjacent to the grid, and it is determined whether the amount of change is a threshold value or more. It is discriminated, and if it is equal to or greater than the threshold value, it is specified as a hue change point.

After extracting such a grid as a hue change area, the parking area extraction unit 134 identifies a predetermined pair of grids (step S1403). Here, a pair corresponding to a predetermined interval (the width of the parking frame line or the width of the parking space) is specified from among the extracted hue change areas. For the horizontal width of the frame line and the horizontal width of the space, values within a predetermined range such as the number of grids are stored in advance. When specifying, a pair corresponding to the horizontal width of the frame line or a pair corresponding to the horizontal width of the space is determined, and identifiable information (identification information) is added. In addition, when extracting, the direction does not matter.

Next, the parking area extraction unit 134 performs grouping on the extracted pairs, and then performs filtering (step S1404). Grouping is a process of associating, as one group, pairs of hue change points of the same kind that are arranged in a straight line. A pair of the same kind is a pair to which the same identification information is assigned, and in this embodiment, it is a pair of frame lines or a pair of spaces. Filtering is a process of deleting groups whose length is less than a threshold as invalid groups.

The parking area extraction unit 134 determines the remaining groups after filtering as parking line candidates or parking space candidates, and extracts the coordinate values and types (parking line candidate, parking space candidate) of each area in the world coordinate system. It is acquired (step S1405), recorded, and the process ends.

Next, extraction processing of an achromatic portion will be described. FIG. 9B is a processing flow of extraction processing of an achromatic portion by the parking area extraction unit 134. As shown in FIG.

This processing is basically the same as the processing of the colored portion. First, the accumulation map 220 is scanned (step S1501).

Next, in the achromatic portion extraction processing, a brightness change region is extracted (step S1502). The lightness change areas are the points where the achromatic color changes to the chromatic color (boundary points), the points where the chromatic color changes to the achromatic color (boundary points), and the achromatic to achromatic color areas between the grids arranged in the scanning direction. In the change, the amount of change in the brightness value is either a point (brightness change point) where the amount of change is equal to or greater than the threshold. Note that the lightness change point is specified by using the average value of a plurality of accumulated values in the same manner as in the color portion extraction process.

Then, for the extracted lightness change area, a pair of grids is identified (step S1503), similar to the processing of the colored portion, and then grouping and filtering are performed (step S1504). The area is determined as a candidate, and the coordinate values and types (parking frame line candidate, parking space candidate) of each area are obtained (step S1405), recorded, and the process ends.

Returning to FIG. 8, the parking area extraction unit 134 identifies the parking frame line and the parking space from the information of the parking frame line candidate and the parking space candidate identified in the colored portion extraction processing and the achromatic portion extraction processing (step S1304). Here, among the extracted parking frame candidates, a set in which the interval between two adjacent lines is within a predetermined threshold and the angle difference between the two lines is less than the threshold is extracted, and the parking frame constituting the parking area is extracted. Let it be a set of lines. Also, the parking space candidate is directly used as the parking space.

Next, filtering is performed according to certainty (step S1305). Here, the average value of the certainty factors of the grids included in the parking closing line and the parking space specified in step S1304 is calculated. An area in which the calculated average value of certainty is equal to or greater than a predetermined threshold value is determined as the parking area 300 (step S1306), and the process ends.

The determined parking area 300 may be stored in the storage device 113 by outputting the coordinate values in the world coordinate system to the output unit.

As described above, the image processing apparatus 100 of the present embodiment combines the images captured by the on-vehicle camera 191 to generate a composite image (overhead image 210), and the hue value of the composite image. A map creation unit 131 that calculates and creates a hue map, and accumulates the hue values of the hue map for each predetermined region (grid) within a predetermined range for a predetermined period to create an accumulation map 220. a parking area extraction unit 134 for extracting a parking frame from the accumulation map 220; and an output processing unit for outputting the parking frame.

Thus, according to the present embodiment, the parking area is specified using the hue and brightness calculated from the bird's-eye view image 210 . Specifically, parking area candidates are first detected using the hue, and parking area candidates are detected using the lightness of areas for which the hue cannot be obtained. Since it is not a brightness value like RGB, it can be detected with high accuracy even in an environment with little brightness difference, such as when it is cloudy, at night, or indoors. In addition, since the detection is performed using the hue value, there are no color restrictions.

In addition, a hue value, that is, a grayscale region in which color information cannot be obtained is detected using a lightness value in a complementary manner. For this reason, it is possible to deal with a wider range of applications.

According to the image processing apparatus of the present embodiment, various parking spaces such as a lawn, a parking space for the disabled, etc., painted in a specific color, and a shopping mall, etc., which are color-coded by area, can be used. A parking area can be appropriately extracted in a parking lot.

Furthermore, according to this embodiment, parking area candidates detected using hue and lightness are filtered using certainty. Therefore, parking area 300 can be identified with higher accuracy.

As a result, according to the present embodiment, the parking area can be accurately detected regardless of the environment.

<Modification 1>
In addition, although the certainty factor is calculated in the above embodiment, the certainty factor does not have to be calculated. The parking closing line candidate and parking space candidate calculated in step S1304 may be directly determined as the parking closing line and parking space.

Also, the certainty factor may be determined by a method other than the above method. For example, the reciprocal of the variance value, the reciprocal of the standard deviation of the accumulated value, or the like may be used.

Alternatively, the parking closing line candidate and parking space candidate calculated in step S1304 may be directly determined as the parking closing line and parking space, and stored together with the degree of certainty. Then, for example, when the parking area extraction process is performed in the same area on another day, the arithmetic processing unit 130 performs the extraction process only when the degree of certainty is less than a predetermined degree, and when the degree of certainty is greater than or equal to the predetermined value, the storage area is stored. The designated parking area may be used as it is.

<Modification 2>
Also, in the above embodiment, the range of the accumulated map 220 in the real space (world coordinate system) changes as the vehicle 910 moves, but the range is not limited to this. The range of the accumulation map 220 may be fixed on the real space. For example, as shown in FIG. 10, the range of the accumulation map 220 is the same regardless of whether the vehicle 910 is at position A or at position B. FIG. The range of the accumulation map 220 to be created is specified in advance by the world coordinate system or the like.

By configuring in this way, for example, when the area of the entire parking lot is known in advance, a large number of hue values and brightness values of the parking lot area can be accumulated, and the extraction accuracy is improved accordingly.

<Modification 3>
In the above-described embodiment, the parking area extraction unit 134 uses the average value of accumulated values to specify the hue change point in the colored portion extraction process and the brightness change point in the achromatic part extraction process. However, it is not limited to this. For example, a weighted average may be used in which the reciprocal of the distance between the own vehicle (vehicle 910) and the point at the time each value is acquired is weighted.

A distance value corresponding to each pixel of the overhead image 210 (distance from the vehicle 910 to the position in the real space corresponding to the pixel) is calculated based on the image captured by the camera 191 . Then, as shown in FIGS. 11A and 11B, when creating an accumulation map 220, the accumulation map creating unit 132 calculates the distance value of each grid and accumulates it together with the hue value and the lightness value. In addition to the camera 191, the vehicle 910 may be equipped with a separate distance sensor, if necessary. In this case, the map creating unit 131 of the image processing device 100 calculates a distance value corresponding to each pixel of the bird's-eye view image 210 using the output of this distance sensor.

When calculating the hue change point and the lightness change point, the parking area extraction unit 134 uses this accumulated distance value as the hue value of the grid, and calculates a weighted average value in which the closer the vehicle (vehicle 910) is, the higher the weight is. Calculate That is, if the i-th registered distance accumulation value is L_i, the weight Wi is calculated by 1/Li. A weighted average value obtained by multiplying the i-th registered hue value Hi by this weight is used as the grid value.

For example, in the example of FIG. 11A, since L2 is closest to the own vehicle (vehicle 910), the value of weight W2=1/L2 is the largest, and the weight of hue value H2 acquired at this time is the same. become bigger.

The parking area extraction unit 134 calculates this weighted average value for each grid, compares it with the weighted average values of adjacent grids, and identifies hue change points.

This makes it possible to determine the change in the hue value of each grid using a value weighted according to the distance at the time of acquisition. Therefore, it is possible to specify the hue change point and the lightness change point with higher accuracy.

<Other Modifications>
Each threshold during processing can be set arbitrarily. The increase/decrease value of the certainty factor may be freely input by the user via the input device 114, for example.

Furthermore, detection of the parking area by luminance may also be combined.

Furthermore, the present embodiment is not limited to parking area detection only. For example, it may be used to detect white lines on the road.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. Further, the above-described embodiments are described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and part of the configuration of the embodiment can be added, deleted, or replaced with another configuration. is.

Further, each of the configurations, functions, processing units, processing means, etc. described above may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function may be placed in a recording device such as a memory, a hard disk, or an SSD, or a recording medium such as an IC card, an SD card, or an optical disc. In addition, the control lines and information lines indicate those considered to be necessary for explanation, and not all the control lines and information lines are necessarily indicated on the product. For example, in practice, it may be considered that almost all configurations are interconnected.

100: Image processing device, 111: CPU, 112: Memory, 113: Storage device, 114: Input device, 115: Output device, 116: Data I/F, 120: Image synthesizing unit, 130: Arithmetic processing unit, 131: Map creation unit 132: Accumulation map creation unit 133: Confidence calculation unit 134: Parking area extraction unit 140: Output processing unit 191: In-vehicle camera
210: bird's-eye view image, 210A: map, 210B: map, 211: pixel, 212: pixel, 220: accumulated map, 220A: accumulated map, 220B: accumulated map, 221: grid, 222: grid, 225: grid, 226: grid, 227: grid, 228: grid,
300: Parking area, 310: Parking frame line, 320: Parking space, 410: Brightness increase/decrease value table, 420: Hue increase/decrease value table, 910: Vehicle

Claims (9)

an image synthesizing unit that synthesizes captured images acquired by a camera and generates a synthesized image;
a map creation unit that calculates hue values of the composite image and creates a hue map;
an accumulation map creation unit that creates an accumulation map by accumulating the hue values of the hue map for each predetermined area in a predetermined range for a predetermined period;
A parking area extraction unit that extracts a parking area from the accumulation map;
and an output processing unit that outputs the parking area. The image processing device according to claim 1,
The map creation unit calculates a brightness value and creates a brightness map for an area of the composite image in which the hue value cannot be calculated,
The image processing apparatus, wherein the accumulation map creation unit also accumulates the brightness values of the brightness map for each of the predetermined regions within the predetermined range for the predetermined period. The image processing device according to claim 2,
The parking area extraction unit extracts a hue change area, which is the area where the hue value changes by a predetermined hue change threshold or more, and the area where the hue value can be calculated and the area where the hue value cannot be calculated. A boundary area is specified, and a parking frame line candidate and a parking space candidate are specified using the hue change area and the boundary area. The image processing device according to claim 3,
The parking area extraction unit identifies a brightness change area, which is the area in which the brightness value changes by a predetermined brightness change threshold or more, in the accumulation map, further uses the brightness change area, and extracts the parking frame line candidate and An image processing device that identifies the parking space candidate. The image processing device according to claim 3 or 4,
The accumulation map creation unit includes a certainty calculation unit that calculates the certainty for each of the predetermined areas of the accumulation map using the accumulated values,
The image processing device, wherein the parking area extracting unit extracts, as the parking area, a candidate whose degree of certainty is equal to or greater than a predetermined threshold from among the parking frame line candidates and the parking space candidates. The image processing device according to claim 5,
The image processing apparatus, wherein the degree of certainty is determined according to a hue variance value that is a variance of the hue value and a lightness variance value that is a variance of the lightness value. The image processing device according to any one of claims 1 to 6,
The captured image is an image of the surroundings of the vehicle equipped with the camera,
The image processing device, wherein the synthesized image is a bird's-eye view image of the vehicle. an image synthesizing step of synthesizing captured images acquired by a camera to generate a synthesized image;
a map creation step of calculating hue values of the composite image and creating a hue map;
an accumulation map creating step of creating an accumulation map by accumulating the hue values of the hue map for each predetermined region in a predetermined range for a predetermined period;
a parking area extracting step of extracting a parking area from the accumulation map;
and an output step of outputting the parking area. the computer,
image synthesizing means for synthesizing captured images acquired by a camera to generate a synthesized image;
map creating means for calculating hue values of the synthesized image and creating a hue map;
accumulation map creating means for creating an accumulation map by accumulating the hue values of the hue map for each predetermined area in a predetermined range for a predetermined period;
Parking area extracting means for extracting a parking area from the accumulation map;
output processing means for outputting the parking area;
A program to function as

Images