WO2023021726A1

WO2023021726A1 - Sign recognition device and sign recognition method

Info

Publication number: WO2023021726A1
Application number: PCT/JP2022/004760
Authority: WO
Inventors: ユイビンツーン; 雄飛椎名; 健永崎
Original assignee: 日立Astemo株式会社
Priority date: 2021-08-18
Filing date: 2022-02-07
Publication date: 2023-02-23
Also published as: DE112022002739T5; JPWO2023021726A1

Abstract

Provided is a processing device capable of improving the performance of recognizing an object in an image such as a restriction lifting sign. This sign recognition device is characterized by comprising: a candidate sign recognition unit that recognizes a candidate sign of a predetermined type from an image; a paired sign search unit that associates, if a candidate for a first sign is recognized as the candidate sign of the predetermined type, the first sign with a second sign installed in correspondence with the first sign; a color information extraction unit that extracts, if there is a second sign installed in correspondence with the first sign, information on the color of the second sign associated with the first sign; and a threshold value determination unit that determines a threshold value for extracting information on the color of the candidate for the first sign on the basis of the reliability of the information on the color of the second sign extracted by the color information extraction unit.

Description

Sign recognition device and sign recognition method

The present invention relates to a sign recognition device and a sign recognition method used in an in-vehicle sensing device such as a stereo camera device.

　In recent years, the demand for various recognition functions of in-vehicle sensing devices has also increased toward the realization of advanced vehicle control that responds to the environment outside the vehicle, such as automatic driving control and driving support control. A stereo camera device, which is a type of in-vehicle sensing device, is a device that simultaneously detects visual information based on images and distance information of objects in the image. , three-dimensional objects, road surfaces, road signs, signage signs, etc.) can be grasped in detail, greatly contributing to the safety improvement of automatic driving control and driving support control.

Some vehicles equipped with a stereo camera device use the recognized traffic signs for acceleration and deceleration control of the vehicle. In addition, Euro NCAP, which is an evaluation index for advanced driver assistance systems, also has evaluation items related to speed assistance systems (SAS), and their importance is increasing. In Japan as well, the importance of automatic speed control functions (ISA, Intelligent Speed Assistance) that suppress overspeeding due to driver carelessness or erroneous operation is increasing, and the recognition accuracy of the sign recognition function that acquires speed limit information is increasing. become necessary. Furthermore, recognition of conditional speed limits such as valid sections, time limits, and vehicle type limits is required, so there is a demand for an expansion of recognizable traffic signs.

Due to the above circumstances, various technologies related to in-vehicle camera devices that recognize the environment in front of the vehicle have been proposed. For example, Patent Document 1 is cited as a conventional technology that focuses on improving the recognition performance of deregulation signs (signs that release regulations such as speed limits and overtaking prohibitions). A camera device capable of improving detection accuracy is provided.Arithmetic processing unit (searching unit) searches for a diagonal line candidate from an image (diagonal line search 504).Arithmetic processing unit (selecting unit) selects a detected diagonal line candidate. from the image of the selected oblique line candidate of the deregulation sign is selected (diagonal line determination 505). identify the deregulation sign (identification process 506).".

WO2020/071132

Since the camera device of Patent Document 1 identifies the deregulation sign from the image of the diagonal candidate, if the captured image has an image pattern as shown in FIG. A diagonal line can be correctly identified as a deregulation sign.

However, in the camera device of Patent Document 1, when an image pattern as shown in FIG. Since it resembles the shaded part of the deregulation sign, there is a possibility that the post above the sign will be misidentified as the deregulation sign.

The present invention has been made in view of the above problems, and its object is to use an image pattern similar to the design of a deregulation sign, such as a pole, a utility pole, a tree branch, etc., as a non-existent deregulation sign. An object of the present invention is to provide a sign recognition device and a sign recognition method that suppress erroneous detection.

In order to solve the above-described problems, the sign recognition apparatus of the present invention includes a sign candidate recognition unit that recognizes a sign candidate of a predetermined type from an image, and a first sign candidate that recognizes a first sign candidate as the sign candidate of the predetermined type. If so, there is a paired sign searching unit that associates the first sign with the second sign installed corresponding to the first sign, and the second sign installed corresponding to the first sign. a color information extraction unit for extracting color information of the second sign associated with the first sign; and a threshold determination unit for determining a threshold when extracting color information of a candidate for a sign.

According to the sign recognition device and the sign recognition method of the present invention, it is possible to suppress false detection of image patterns similar to the design of deregulation signs, such as poles, utility poles, tree branches, etc., as non-existent deregulation signs. be able to.

Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a functional block diagram showing a schematic configuration of a stereo camera device of one embodiment; FIG. 4 is a flowchart of basic processing of the stereo camera device of one embodiment. 4 is a flowchart of sign recognition processing of the sign recognition device of one embodiment. 4 is a flowchart of circular object extraction processing of the sign recognition device of one embodiment. Speed limit sign image pattern Explanatory drawing of the center estimation process and radius estimation process with respect to the edge image of a speed limit sign. 4 is a flowchart of effective line search processing of the sign recognition device of one embodiment. FIG. 7 is an explanatory diagram of left and right edge search processing in FIG. 6; Explanatory drawing of the straight line detection process of FIG. FIG. 7 is an explanatory diagram of the effective straight line determination process in FIG. 6; Image diagram of paired sign search processing in Japan. An image diagram of paired sign search processing in another country. 4 is a flowchart of paired sign search processing of the sign recognition device of one embodiment. An image diagram showing the position of the deregulation pair sign in Japan. An image diagram showing the position of a deregulation pair sign in another country. The figure explaining the object similar to the design of the deregulation sign.

A sign recognition device according to an embodiment of the present invention will be described below with reference to the drawings. Note that portions having the same function in each drawing may be denoted by the same reference numerals, and repeated description may be omitted.

<Schematic configuration of stereo camera device>
FIG. 1 is a functional block diagram showing a schematic configuration of a stereo camera device 100 according to one embodiment of the invention. The stereo camera device 100 is a type of in-vehicle sensing device mounted in a vehicle that executes automatic driving control and driving support control, and is based on image information of a shooting target area in front of the vehicle. It is a device that recognizes white lines, pedestrians, other vehicles, other three-dimensional objects, traffic lights, traffic signs, lighting lamps, etc.). Also, the stereo camera device 100 determines a control policy (brake, steering, etc.) of the own vehicle according to the environment outside the vehicle, and provides the control policy to an ECU (Electronic Control Unit) via an in-vehicle network CAN (Controller Area Network). It is an output device. Then, the ECU controls the braking system and the steering system of the own vehicle according to the control policy of the stereo camera device 100, thereby safely decelerating the own vehicle and causing the own vehicle to avoid obstacles.

As shown in FIG. 1, the stereo camera device 100 of this embodiment has a camera 1 and a sign recognition device 2. As shown in FIG. The camera 1 is composed of a left camera 1L and a right camera 1R arranged side by side, and outputs a pair of left and right images (left image P _L and right image P _R ) synchronously photographed in front of the vehicle.

The sign recognition device 2 recognizes traffic signs in the image captured by the camera 1. Specifically, the sign recognition device 2 includes a processor such as a CPU (Central Processing Unit), a memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). One or more computer units. Each function of the sign recognition device 2 is realized by the processor executing a program stored in the ROM. The RAM and SSD store data such as intermediate data for computation by the program and images of the camera 1 . As described above, the stereo camera device 100 is a device capable of recognizing the environment outside the vehicle (white lines on the road, pedestrians, other vehicles, etc.) other than traffic signs. Since the recognition function is focused on, the device 2, which should be called the environment recognition device, will be called the sign recognition device 2 in this embodiment.

As shown, the sign recognition device 2 includes an image input interface 21, an image processing unit 22, an arithmetic processing unit 23, a storage unit 24, a CAN interface 25, a monitoring processing unit 26, and interconnecting them. It has an internal bus 27 that Each part will be outlined below.

The image input interface 21 is an interface that controls the imaging device of the camera 1 and captures the captured image. The image captured by the image input interface 21 is transmitted to the image processing section 22 and the arithmetic processing section 23 via the internal bus 27 .

The image processing unit 22 compares the left image _PL captured by the imaging device of the left camera 1L and the right image _PR captured by the imaging device of the right camera 1R, and determines device-specific After performing image correction such as deviation correction and noise interpolation, the left and right images after correction are stored in the storage unit 24 . Further, the image processing unit 22 extracts mutually corresponding portions between the corrected left and right images, and then calculates distance information of each pixel on the image based on the parallax between the corresponding portions. Then, the calculated distance information is stored in the storage unit 24 .

The arithmetic processing unit 23 uses the corrected left and right images and the distance information stored in the storage unit 24 to recognize various objects necessary for perceiving the environment around the vehicle. Various objects recognized here include, for example, pedestrians, other vehicles, other obstacles, traffic lights, traffic signs, tail lamps and headlights of vehicles, and the like. Some of these recognition results and intermediate calculation results are recorded in the storage unit 24 . Furthermore, the arithmetic processing unit 23 determines a control strategy for the own vehicle using the recognition results of various objects.

The storage unit 24 stores data during or after processing by the image processing unit 22 and the arithmetic processing unit 23 . Further, various information necessary for recognizing various objects is registered in advance in the storage unit 24 . Note that the storage unit 24 is specifically a memory such as the above-described RAM or SSD, and includes an image buffer 24a and a discriminator 24b, which will be described later.

The CAN interface 25 is an interface that transmits various objects recognized by the arithmetic processing unit 23 and vehicle control policies determined by the arithmetic processing unit 23 to the in-vehicle network CAN. The ECU connected to the in-vehicle network CAN executes braking of the own vehicle, warning to the driver, etc., based on the object recognition result and control policy transmitted via the in-vehicle network CAN.

The monitoring processing unit 26 monitors whether any of the above-described units are operating abnormally, or whether an error has occurred during data transfer, and is a mechanism for preventing abnormal operations.

<Basic processing of the stereo camera device>
Here, basic processing by the stereo camera device 100 of the present embodiment will be described using the flowchart of FIG.

First, in step S1L, the image processing unit 22 causes the left camera 1L to capture the left image _PL . Next, in step S2L, the image processing unit 22 performs processing such as image correction that absorbs the unique characteristics of the imaging element on the captured left image _PL , and then corrects the left image _PL . is stored in the image buffer 24 a of the storage unit 24 .

Further, in synchronism with steps S1L and S2L, in step S1R, the image processing unit 22 causes the right camera 1R to capture the right image _PR . Next, in step S2R, the image processing unit 22 performs processing such as image correction that absorbs the unique characteristics of the imaging element on the captured right image _PR , and then corrects the right image _PR . is stored in the image buffer 24 a of the storage unit 24 .

In step S3, the image processing unit 22 compares the corrected left image _PL and right image _PR stored in the image buffer 24a, and calculates the parallax between the left and right images. The parallax between the left and right images makes it clear where a given point on the object corresponds to where on the left and right images, so the distance to the object can be calculated by the principle of triangulation. . Note that the distance information obtained in this step is also stored in the storage unit 24 .

In step S4, the arithmetic processing unit 23 recognizes various objects using the corrected left and right images and the distance information stored in the image buffer 24a. Objects to be recognized include pedestrians, other vehicles, other three-dimensional objects, traffic signs, traffic lights, tail lamps, and the like. The discriminator 24b is used as necessary for various object recognitions in this step. The discriminator 24b stores and records, for example, features of an object to be recognized as machine learning data. Details of the sign recognition processing (step S40) executed in this step will be described later.

In step S5, the arithmetic processing unit 23 considers the recognition results of various objects in step S4 and the state of the own vehicle (speed, steering angle, etc.) to determine a vehicle control policy. For example, if the own vehicle exceeds the speed limit, a control policy such as issuing a warning to the passengers or braking the own vehicle is determined. In this embodiment, as will be described below, the arithmetic processing unit 23 determines control of the own vehicle based on information on the sign recognized in step S4.

In step S6, the CAN interface 25 outputs the various objects recognized in step S4 and the control policy of the own vehicle determined in step S6 to the external ECU through the in-vehicle network CAN. As a result, the ECU can issue a warning to the occupants or brake the vehicle in accordance with the control policy determined by the stereo camera device 100 .

<Sign recognition processing>
Next, the sign recognition process (step S40), which is one aspect of the various object recognition processes (step S4) in FIG. 2, will be described using the flowchart in FIG.

First, in step S41, the arithmetic processing unit 23 extracts an image pattern including a circular object from the post-correction image in the image buffer 24a. This processing is divided into edge image generation processing (step S41a), center estimation processing (step S41b), and radius estimation processing (step S41c). Each process will be described in turn.

First, in the edge image generation process (step S41a), the arithmetic processing unit 23 generates an edge image _PE1 by extracting edge components of the post-correction image in the image buffer 24a. For example, if an image pattern of a speed limit sign as shown in FIG. 5A is captured in part of the post-correction image, an edge image P _E1 as shown in FIG. 5B is generated in this step.

Next, in the center estimation process (step S41b), the arithmetic processing unit 23 estimates the center of the circular object. Specifically, as shown in FIG. 5B, the arithmetic processing unit 23 draws a line segment Ln from each edge of the edge image _PE1 in the normal direction, and draws a line segment Ln at a point where a certain number or more of intersections of the line segments Ln overlap. Assume C. In the example of FIG. 5B, the center C is presumed to be between the number "11" and the number "0".

In the radius estimation process (step S41c), the arithmetic processing unit 23 estimates the radius of the circular object based on the histogram of the edge image _PE1 . For example, when estimating the radius of the circular object in FIG. 5B, a histogram is created with the horizontal axis representing the distance from the center C and the vertical axis representing the number of edges. Specifically, first, a circle having a radius of r=0 is arranged at the center C, and the radius r of this circle is gradually increased. Then, when this circle overlaps an edge, the number of overlapped edges is counted as the number of edges at that distance. By continuing this process until the radius r reaches a predetermined value (a radius appropriate for a traffic sign), the histogram of circular objects in FIG. 5B can be generated. In the histogram based on FIG. 5B generated in this way, the horizontal axis position corresponding to the distance from the center C to the first edge group E1 and the horizontal axis position corresponding to the distance from the center C to the second edge group E2 Therefore, it can be assumed that both the first edge group E1 and the second edge group E2 are substantially annular. In this case, the arithmetic processing unit 23 estimates that the second edge group E2 having the larger radius is the outer diameter of the circular object.

Through the above-described processing, the arithmetic processing unit 23 can extract a circular object and its features (center position, radius) from the corrected image in the image buffer 24a.

In step S42, the arithmetic processing unit 23 acquires color information of the circular object extracted in step S41. This is because the color information of this sign is used as a reference when judging the authenticity of the deregulation sign in step S4b, which will be described later. Note that step S42 may be processed after step S43, which will be described later, or may be processed in parallel with step S43.

At step S43, the arithmetic processing unit 23 identifies whether the circular object extracted at step S41 is a traffic sign candidate. Specifically, the arithmetic processing unit 23 performs identification processing using the classifier 24b on the image pattern of the corrected image in the image buffer 24a, which corresponds to the position of the circular object extracted in step S41. . As a result, it is recognized whether or not the circular object extracted in step S41 is a traffic sign candidate, and if the circular object is a traffic sign candidate, the content of the traffic sign (for example, speed limit) is recognized. A first identification score representing the likelihood of a traffic sign when identified as a traffic sign candidate is output in this identification process and stored in memory. The traffic sign candidates identified in this step are hereinafter referred to as "main signs". If a circular object is not extracted in step S41, this step may be omitted.

In steps S44-45 and 47-4b, candidates for deregulation signs are searched.

First, in step S44, the arithmetic processing unit 23 searches for an effective straight line that can be a candidate for a deregulation sign from the corrected image in the image buffer 24a. This effective line search processing is divided into left and right edge search processing (step S44a), line detection processing (step S44b), and effective line determination processing (step S44c). Each process will be described in order with reference to 7C.

First, in the left and right edge search processing (step S44a), the arithmetic processing unit 23 scans the pixels of the corrected image in the image buffer 24a row by row from left to right, as shown in FIG. 7A. If the left edge E _L and the right edge E _R can be found within a certain distance, they are extracted as a pair of left and right edges. By applying such processing to the entire area of the post-correction image, an edge image P _E2 ( FIG. 7B is an excerpt of the edge image) can be generated by extracting a pair of left and right edges from the post-correction image.

Next, in the straight line detection process (step S44b), the arithmetic processing unit 23 detects a straight line portion from the edge image _PE2 generated in step S44a. The edge image P _E2 illustrated in FIG. 7B has a portion in which pairs of left and right edges are continuous in two or more rows and a portion in which there is only one row. In this step, a portion in which two or more rows of pairs of left and right edges continue vertically is detected as a straight line.

In the effective straight line determination process (step S44c), the arithmetic processing unit 23 determines whether the straight line detected in step S44b is a valid candidate for the deregulation sign, and extracts a valid straight line that can be a candidate. Japan's deregulation sign has a conspicuous thick blue slanted line inside a circular sign (see the upper sign in Figure 11(a)). There are many release signs (eg Germany, Poland, Sweden, etc.). Therefore, in this step, considering the design features of the hatched portion of the deregulation sign, straight portions that can be candidates for the deregulation sign are extracted from the post-correction image.

When the stereo camera device 100 is used, the distance to the straight line detected in step S44b can be calculated. Therefore, assuming that the deregulation sign exists at the calculated distance, the oblique line of the deregulation sign on the corrected image The width and height of the part can also be approximated. Therefore, if the width of the edge pair illustrated in FIG. 7B is too narrow or too wide, the straight line formed by the edge pair is determined as an invalid straight line that cannot be a candidate for the deregulation sign, and is excluded from the valid straight lines. can do.

Similarly, as exemplified in FIG. 7C, the height of the straight portion is short and does not reach the lower limit of the effective range that is considered appropriate as the height of the hatched portion of the deregulation sign (FIG. 7C (b) ), or longer than the upper limit (FIG. 7C(c)), these straight lines are determined as invalid straight lines that cannot be candidates for deregulation signs, and are excluded from valid straight lines. On the other hand, when the height of the straight portion falls within the effective range (FIG. 7C(a)), the straight portion is extracted as an effective straight line.

By the processing in step S44 described above, the image patterns of utility poles and supports that are significantly different in form from the shaded portion of the deregulation sign are excluded from the deregulation sign candidates, and the straight portions that may be deregulation signs are excluded. can extract only the image pattern of

Next, in step S45, the arithmetic processing unit 23 determines whether an effective straight line has been extracted by the processing in step S44. If the effective straight line is not extracted, the process proceeds to step S46, and if the effective straight line is extracted, the process proceeds to step S47.

In step S46, the arithmetic processing unit 23 tracks the main sign (traffic sign) candidate identified in step S43. Here, “tracking” refers to the shape information and position information of the sign candidate obtained in step S41, the color information obtained in step S42, and the color It refers to storing information in memory in association with time series. Specifically, the type of the traffic sign candidate identified in step S43 (for example, a speed limit sign) is registered in the tracking list L1, which is a list for registering the type of traffic sign being imaged by the stereo camera device 100. . Note that when the own vehicle passes through the main sign candidate registered in the tracking list L1 and becomes unable to take an image of the main sign candidate, the information on the main sign candidate registered in the tracking list L1 is not recognized. It is assumed that it is transcribed to list L2.

On the other hand, in step S47, the arithmetic processing unit 23 calculates color information of the effective straight line extracted in step S44. For example, in the case of searching for a candidate for a deregulation sign in Japan (see FIG. 11(a)), the arithmetic processing unit 23 arranges a window having a shape that accommodates an effective straight line in the image, and Calculate the blue color score. Here, the blue score is calculated by, for example, the following method. That is, if the color of each pixel in the image is defined by the three primary colors red (R), green (G), and blue (B), the area of the blue pixel in the window is divided by the total area of the window. By doing so, the blue score is calculated. The calculated blue score is stored in memory. Although blue is used as an example here, color information other than blue may be extracted according to the sign design. It says.

As mentioned above, for example, the deregulation sign in Japan is designed with a conspicuous blue diagonal line (see Fig. 11(a)), so the blue score in the window that actually includes the deregulation sign is considered to be high. Therefore, by focusing on the blue score of the window, for example, effective straight lines clearly other than blue, such as white straight lines and black straight lines, are excluded from candidates for deregulation signs. Only effective straight lines can be extracted.

However, when capturing an image of a deregulation sign illuminated by the morning sun or sunset, or when capturing an image of a deregulation sign in cloudy or rainy weather, the color of the deregulation sign may be weakened in the image. In order not to exclude such deregulation signs from the candidates, the threshold for comparison with the blue score is set low in this step. As a result, in this step, not only actual deregulation signs, but also effective straight lines that are not deregulation signs (for example, poles, utility poles, tree branches, etc.) may be extracted as deregulation sign candidates. Candidates for deregulation signs at this step are not reliable enough.

In step S48, candidates for deregulation signs are identified. This is performed in the same manner as the identification of the main mark in S42. Specifically, the arithmetic processing unit 23 uses the discriminator 24b to discriminate the deregulation sign candidates. At this time, a second identification score representing the likeness of the deregulation sign is output in this identification process and stored in memory. The deregulation sign candidates identified in this step are hereinafter referred to as "deregulation signs". Note that step S48 may be processed before step S47, or may be processed in parallel with step S47.

In step S49, the arithmetic processing unit 23 tracks the actual sign identified in step S42 and the candidates for the deregulation sign extracted in step S48. Again, "tracking" has the same meaning as described in step S46. Specifically, in the tracking list L1, which is a list for registering the signs being imaged by the stereo camera device 100, the type of the main sign candidate identified in step S43 (for example, a speed limit sign) and the type extracted in step S48. Register the deregulation sign (candidate). Note that if the vehicle passes through a traffic sign and cannot take an image of the traffic sign registered in the tracking list L1, the information on the traffic sign registered in the tracking list L1 is transferred to the recognized list L2. shall be transcribed.

In step S4a, the arithmetic processing unit 23 searches for a real sign (for example, speed limit sign, no overtaking) paired with the deregulation sign candidate extracted in step S48. The method of searching for paired signs in this step differs from country to country. For example, in Japan, as shown in FIG. 8A, this sign is placed below the deregulation sign. should be associated as On the other hand, in other countries, as shown in FIG. It suffices to associate the registered deregulation signs as a pair sign. By using either search method, it is possible to search for this sign that is paired with the derestriction sign regardless of the country.

Here, the details of the pair sign search process in step S4a will be described using the flowchart of FIG. This flowchart is divided into processing from step S4aa to step S4ad and processing from step S4af to step S4ah.

From step S4aa to step S4ad, the arithmetic processing unit 23 searches for pair signs based on the tracking list L1. This is a search method for paired signs in countries where this sign and deregulated sign are installed at the same location, as illustrated in FIG. 8A.

First, in step S4ab, the arithmetic processing unit 23, based on the tracking information registered in the tracking list L1 in step S49 (information for specifying the sign such as the set position, size, type, and tracking number of the sign) Check if there is a main sign paired with the deregulation sign. For example, as shown in FIG. 8A, if a deregulation sign and a speed limit sign are registered in the tracking list L1, they are extracted as pair sign candidates.

Next, in step S4ac, the arithmetic processing unit 23 confirms the arrangement of the deregulation sign and the pair sign candidate, and determines that both are pair signs if their installation positions are close. For example, as shown in FIG. 10A, a speed limit sign, which is the first paired sign candidate, is installed on the same post as the deregulation sign, and a second pair sign candidate is installed on a post far from the deregulation sign. If there is an overtaking prohibition sign, the speed limit sign installed on the same pole as the deregulation sign is determined as a pair sign, and the overtaking prohibition sign in the distance is not determined as a pair sign. In addition, when a plurality of permanent signs are installed on the same post as the deregulation sign, the plurality of permanent signs may be extracted as a pair of signs.

In step S4ad, the arithmetic processing unit 23 stores information on the paired signs extracted in step 48c in the storage unit 24.

At step S4ae, the arithmetic processing unit 23 confirms whether the pair indicator has been registered. Then, when the pair indicator is registered, the process of step S4a is terminated and the process proceeds to step S4b. On the other hand, if the pair indicator is unregistered, the process proceeds to step S4af.

From step S4af to step S4ah, the arithmetic processing unit 23 searches for pair signs based on the tracking list L1 and the recognized list L2. As illustrated in FIG. 8B, this is a search method for paired signs in countries where the main sign and deregulation sign are installed in separate locations.

First, in step S4ag, the arithmetic processing unit 23, based on the recognized information registered in the recognized list L2 (previously recognized sign information such as the setting position, size, type, and registration number of the sign), It is checked whether or not there is a regular sign paired with the deregulation sign registered in L1. For example, as shown in FIG. 8B, if a previously imaged speed limit sign is registered in the recognized list L2, it is extracted as a pair sign candidate. Since there is a possibility that a large number of previously recognized sign information are registered in the recognized list L2, in this step, the most recently registered main sign is determined as a pair sign candidate. For example, as shown in FIG. 10B, a speed limit sign, which is the first pair of candidate signs, is installed on the pole in front of the deregulation sign, and the second pair of sign candidates is installed on the pole in front of the speed limit sign. If a no overtaking sign is installed, the speed limit sign registered most recently is determined to be a paired sign, and the previously registered no overtaking sign is determined not to be a paired sign.

In FIG. 9, both the processing from step S4aa to step S4ad corresponding to the environment in FIG. 8A and the processing from step S4af to step S4ah corresponding to the environment in FIG. 8B are executed. If the country in which the vehicle is traveling can be identified using It is good also as a structure to carry out.

After completing the process of step S4a shown in FIG. 9, the main sign that is paired with the deregulation sign can be extracted regardless of the installation mode of the deregulation sign (FIGS. 8A and 8B). However, in step S45, as described above, there is a possibility that an effective straight line that is not actually a deregulation sign is extracted as a candidate. By also referring to the color information of the actual sign), it is confirmed whether or not the candidate extracted in step S48 is truly a deregulation sign. The color information of the main sign is acquired in step S48) and stored in the memory.

In step S4b, the arithmetic processing unit 23 determines whether it is a time zone in which it is possible to perform the authenticity determination process (step S4c) of the candidate deregulation sign. As will be described later, in step S4c, the color information of the paired signs is used. reliability is low. Therefore, in this step, for example, based on the time information acquired from the built-in timer of the ECU or the GPS, it is determined whether it is daytime or nighttime. Proceeding to step S4c, it is determined that it is difficult to obtain highly reliable color information at night, and the process proceeds to step S4d. It should be noted that if the candidate for this sign is not extracted in step S43, the process of step S4c cannot be executed, and if the candidate for the deregulation sign is not extracted in step S48, the truth of the candidate is determined. Since the process of step S4c for determining is not necessary in the first place, step S4c is avoided in these cases as well, and the process proceeds to step S4c.

In step S4c, the arithmetic processing unit 23 determines whether the candidates for deregulation signs extracted in step S48 are true or false. reject. As described above, in step S48, candidates for deregulation signs are extracted based on the blue score of the effective straight line, but deregulation signs illuminated with colored light and deregulation signs in cloudy or rainy weather are excluded from the candidates. In order to avoid this, the threshold for comparison with the blue score of the effective straight line was set to be low, which allowed the extraction of effective straight lines that were not actual deregulation signs as candidates. Therefore, in this step, by referring to the color score of the main sign acquired in step S42 and adjusting the threshold value used to determine the authenticity of the candidates for the deregulation sign, the deregulation sign It is now possible to accurately judge the authenticity of a candidate.

For example, as shown in Fig. 11, if this sign is a speed limit sign with a red ring around the sign, the red score within the window shaped to accommodate the red ring is considered to be a high value. However, if this sign is illuminated by colored light, cloudy or rainy weather, clear weather, etc., the red score of this sign (speed limit sign) will change. In that case, the blue score of the deregulation sign would also change to the same extent.

Therefore, in this step, when the red score of this sign (speed limit sign) is low, the threshold of the blue score for judging the authenticity of the candidate of the deregulation sign is also set low. If it is high, the blue score threshold is also set high. In other words, the color information (for example, blue score (that is, confidence in being blue)) acquired when recognizing this sign is used to determine the threshold of color information for recognition as a deregulation sign. In addition, not the color information itself acquired when recognizing this sign, but the rate of change in color information acquired when recognizing this sign against the color information of the template image of this sign, to recognize it as a deregulation sign A threshold for color information may be determined. Together, the color information of the sign and the rate of change of the color information of the sign are referred to as the "reliability" of the sign.

By using the threshold value of color information (for example, blue score) adjusted in this way, it is possible to accurately determine the authenticity of candidates for deregulation signs regardless of the environment.

Finally, in step S4d, the arithmetic processing unit 23 refers to the tracking list L1 or the recognized list L2 to determine the result of identifying the sign. That is, if the process proceeds from step S45 to step S46, only the main marker candidate identified in step S43 is determined as the identification result as a marker. On the other hand, if the process proceeds from step S45 to step S47, for example, the main sign candidate identified in step S43 is identified as a pair sign of the deregulation sign candidate extracted in steps S48 and S4c. Then, when the pair sign is recognized in this step, a control policy that cancels the regulation indicated by this sign is determined in the subsequent step S5. Here, "determine" means outputting the information obtained from the recognized sign to the subsequent control process (S5 in FIG. 2) as the result of the recognition process.

According to the stereo camera device 100 of the present embodiment described above, not only can the deregulation sign shown in FIG. Since it is possible to suppress erroneous detection of the upper ends of the posts having similar thicknesses as the deregulation sign, it is possible to improve the reliability of sign detection.

In the above-described embodiment, the stereo camera device 100 composed of two cameras was used, but one camera or three or more cameras may be used. Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

DESCRIPTION OF SYMBOLS 100... Stereo camera apparatus, 1... Camera, 1L... Left camera, 1R... Right camera, 2... Sign recognition apparatus, 21... Image input interface, 22... Image processing part, 23... Operation process part, 24... Storage part, 24a ... image buffer, 24b ... discriminator, 25 ... CAN interface, 26 ... monitoring processing unit, 27 ... internal bus, CAN in-vehicle network

Claims

a sign candidate recognition unit that recognizes sign candidates of a predetermined type from an image;
a paired sign searching unit that, when a first sign candidate is recognized as a sign candidate of the predetermined type, associates the first sign with a second sign installed corresponding to the first sign;
a color information extraction unit for extracting color information of the second sign associated with the first sign when there is a second sign installed corresponding to the first sign;
a threshold determination unit configured to determine a threshold for extracting the color information of the first marker candidate based on the reliability of the color information of the second marker extracted by the color information extraction unit;
A sign recognition device comprising:
The sign recognition device according to claim 1,
The paired sign searching unit determines whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, uses a threshold value different from the threshold value.
The sign recognition device according to claim 1,
further comprising a sign candidate storage unit that stores the recognition results of the sign candidate recognition unit in time series,
The paired sign searching unit determines whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, and extracting a second sign associated with the first sign from recognition results stored in the sign candidate storage unit.
The sign recognition device according to claim 1,
A sign recognition device, characterized in that when information on a plurality of first signs is registered in a tracking list, a first sign closest to a second sign is associated as a pair sign.
The sign recognition device according to claim 1,
transferring the information of the first sign registered in the tracking list to the recognized list when the vehicle passes the first sign;
A sign recognition device, wherein a second sign candidate registered in the tracking list and a first sign registered in the recognized list are associated as a pair sign.
In the sign recognition device according to claim 5,
A sign recognition device, wherein when information on a plurality of first signs is registered in the recognized list, the most recently registered first sign is associated as a pair sign.
a step of recognizing candidate signs of a predetermined type from an image captured by an in-vehicle camera;
when a first sign candidate is recognized as a sign candidate of the predetermined type, associating the first sign with a second sign installed corresponding to the first sign;
if there is a second sign installed corresponding to the first sign, extracting color information of the second sign associated with the first sign;
determining a threshold for extracting the color information of the candidate for the first marker based on the reliability of the extracted color information for the second marker;
A sign recognition method, comprising:
In the sign recognition method according to claim 7,
It is determined whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, it is different from the threshold A sign recognition method characterized by using a threshold.
In the sign recognition method according to claim 7,
further comprising a step of storing recognition results in chronological order;
It is determined whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, the stored recognition result is determined. A method of recognizing a sign, comprising extracting a second sign associated with the first sign from among them.
In the sign recognition method according to claim 7,
A sign recognition method characterized by associating a first sign closest to a second sign as a pair sign when information on a plurality of first signs is registered in a tracking list.
In the sign recognition method according to claim 7,
transferring the information of the first sign registered in the tracking list to the recognized list when the vehicle passes the first sign;
a step of associating a second marker candidate registered in a tracking list and a first marker registered in the recognized list as a paired marker;
A sign recognition method, comprising:
The sign recognition method according to claim 11,
A sign recognition method, wherein when information on a plurality of first signs is registered in the recognized list, the most recently registered first sign is associated as a pair sign.