WO2023021726A1 - Sign recognition device and sign recognition method - Google Patents
Sign recognition device and sign recognition method Download PDFInfo
- Publication number
- WO2023021726A1 WO2023021726A1 PCT/JP2022/004760 JP2022004760W WO2023021726A1 WO 2023021726 A1 WO2023021726 A1 WO 2023021726A1 JP 2022004760 W JP2022004760 W JP 2022004760W WO 2023021726 A1 WO2023021726 A1 WO 2023021726A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- sign
- candidate
- recognition
- registered
- image
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 54
- 238000000605 extraction Methods 0.000 claims abstract description 7
- 239000003550 marker Substances 0.000 claims description 9
- 238000012545 processing Methods 0.000 abstract description 94
- 239000000284 extract Substances 0.000 abstract description 6
- 230000003831 deregulation Effects 0.000 description 82
- 230000008569 process Effects 0.000 description 34
- 230000006870 function Effects 0.000 description 10
- 238000012937 correction Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 238000013461 design Methods 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000003702 image correction Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000033228 biological regulation Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000011217 control strategy Methods 0.000 description 1
- 230000002074 deregulated effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/582—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of traffic signs
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
Definitions
- the present invention relates to a sign recognition device and a sign recognition method used in an in-vehicle sensing device such as a stereo camera device.
- a stereo camera device which is a type of in-vehicle sensing device, is a device that simultaneously detects visual information based on images and distance information of objects in the image. , three-dimensional objects, road surfaces, road signs, signage signs, etc.) can be grasped in detail, greatly contributing to the safety improvement of automatic driving control and driving support control.
- Some vehicles equipped with a stereo camera device use the recognized traffic signs for acceleration and deceleration control of the vehicle.
- Euro NCAP which is an evaluation index for advanced driver assistance systems
- SAS speed assistance systems
- ISA automatic speed control functions
- ISA Intelligent Speed Assistance
- recognition of conditional speed limits such as valid sections, time limits, and vehicle type limits is required, so there is a demand for an expansion of recognizable traffic signs.
- Patent Document 1 is cited as a conventional technology that focuses on improving the recognition performance of deregulation signs (signs that release regulations such as speed limits and overtaking prohibitions).
- a camera device capable of improving detection accuracy is provided.
- Arithmetic processing unit searches for a diagonal line candidate from an image (diagonal line search 504).
- Arithmetic processing unit selects a detected diagonal line candidate. from the image of the selected oblique line candidate of the deregulation sign is selected (diagonal line determination 505). identify the deregulation sign (identification process 506).”.
- the present invention has been made in view of the above problems, and its object is to use an image pattern similar to the design of a deregulation sign, such as a pole, a utility pole, a tree branch, etc., as a non-existent deregulation sign.
- An object of the present invention is to provide a sign recognition device and a sign recognition method that suppress erroneous detection.
- the sign recognition apparatus of the present invention includes a sign candidate recognition unit that recognizes a sign candidate of a predetermined type from an image, and a first sign candidate that recognizes a first sign candidate as the sign candidate of the predetermined type. If so, there is a paired sign searching unit that associates the first sign with the second sign installed corresponding to the first sign, and the second sign installed corresponding to the first sign. a color information extraction unit for extracting color information of the second sign associated with the first sign; and a threshold determination unit for determining a threshold when extracting color information of a candidate for a sign.
- the sign recognition device and the sign recognition method of the present invention it is possible to suppress false detection of image patterns similar to the design of deregulation signs, such as poles, utility poles, tree branches, etc., as non-existent deregulation signs. be able to.
- FIG. 1 is a functional block diagram showing a schematic configuration of a stereo camera device of one embodiment
- FIG. 4 is a flowchart of basic processing of the stereo camera device of one embodiment.
- 4 is a flowchart of sign recognition processing of the sign recognition device of one embodiment.
- 4 is a flowchart of circular object extraction processing of the sign recognition device of one embodiment.
- Speed limit sign image pattern Explanatory drawing of the center estimation process and radius estimation process with respect to the edge image of a speed limit sign.
- 4 is a flowchart of effective line search processing of the sign recognition device of one embodiment.
- FIG. 7 is an explanatory diagram of left and right edge search processing in FIG. 6; Explanatory drawing of the straight line detection process of FIG.
- FIG. 7 is an explanatory diagram of the effective straight line determination process in FIG.
- FIG. 1 is a functional block diagram showing a schematic configuration of a stereo camera device 100 according to one embodiment of the invention.
- the stereo camera device 100 is a type of in-vehicle sensing device mounted in a vehicle that executes automatic driving control and driving support control, and is based on image information of a shooting target area in front of the vehicle. It is a device that recognizes white lines, pedestrians, other vehicles, other three-dimensional objects, traffic lights, traffic signs, lighting lamps, etc.).
- the stereo camera device 100 determines a control policy (brake, steering, etc.) of the own vehicle according to the environment outside the vehicle, and provides the control policy to an ECU (Electronic Control Unit) via an in-vehicle network CAN (Controller Area Network). It is an output device. Then, the ECU controls the braking system and the steering system of the own vehicle according to the control policy of the stereo camera device 100, thereby safely decelerating the own vehicle and causing the own vehicle to avoid obstacles.
- a control policy brake, steering, etc.
- the stereo camera device 100 of this embodiment has a camera 1 and a sign recognition device 2.
- the camera 1 is composed of a left camera 1L and a right camera 1R arranged side by side, and outputs a pair of left and right images (left image P L and right image P R ) synchronously photographed in front of the vehicle.
- the sign recognition device 2 recognizes traffic signs in the image captured by the camera 1.
- the sign recognition device 2 includes a processor such as a CPU (Central Processing Unit), a memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive).
- a processor such as a CPU (Central Processing Unit)
- a memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive).
- Each function of the sign recognition device 2 is realized by the processor executing a program stored in the ROM.
- the RAM and SSD store data such as intermediate data for computation by the program and images of the camera 1 .
- the stereo camera device 100 is a device capable of recognizing the environment outside the vehicle (white lines on the road, pedestrians, other vehicles, etc.) other than traffic signs. Since the recognition function is focused on, the device 2, which should be called the environment recognition device, will be called the sign recognition device 2 in this embodiment.
- the sign recognition device 2 includes an image input interface 21, an image processing unit 22, an arithmetic processing unit 23, a storage unit 24, a CAN interface 25, a monitoring processing unit 26, and interconnecting them. It has an internal bus 27 that Each part will be outlined below.
- the image input interface 21 is an interface that controls the imaging device of the camera 1 and captures the captured image.
- the image captured by the image input interface 21 is transmitted to the image processing section 22 and the arithmetic processing section 23 via the internal bus 27 .
- the image processing unit 22 compares the left image PL captured by the imaging device of the left camera 1L and the right image PR captured by the imaging device of the right camera 1R, and determines device-specific After performing image correction such as deviation correction and noise interpolation, the left and right images after correction are stored in the storage unit 24 . Further, the image processing unit 22 extracts mutually corresponding portions between the corrected left and right images, and then calculates distance information of each pixel on the image based on the parallax between the corresponding portions. Then, the calculated distance information is stored in the storage unit 24 .
- the arithmetic processing unit 23 uses the corrected left and right images and the distance information stored in the storage unit 24 to recognize various objects necessary for perceiving the environment around the vehicle.
- Various objects recognized here include, for example, pedestrians, other vehicles, other obstacles, traffic lights, traffic signs, tail lamps and headlights of vehicles, and the like. Some of these recognition results and intermediate calculation results are recorded in the storage unit 24 . Furthermore, the arithmetic processing unit 23 determines a control strategy for the own vehicle using the recognition results of various objects.
- the storage unit 24 stores data during or after processing by the image processing unit 22 and the arithmetic processing unit 23 . Further, various information necessary for recognizing various objects is registered in advance in the storage unit 24 . Note that the storage unit 24 is specifically a memory such as the above-described RAM or SSD, and includes an image buffer 24a and a discriminator 24b, which will be described later.
- the CAN interface 25 is an interface that transmits various objects recognized by the arithmetic processing unit 23 and vehicle control policies determined by the arithmetic processing unit 23 to the in-vehicle network CAN.
- the ECU connected to the in-vehicle network CAN executes braking of the own vehicle, warning to the driver, etc., based on the object recognition result and control policy transmitted via the in-vehicle network CAN.
- the monitoring processing unit 26 monitors whether any of the above-described units are operating abnormally, or whether an error has occurred during data transfer, and is a mechanism for preventing abnormal operations.
- step S1L the image processing unit 22 causes the left camera 1L to capture the left image PL .
- step S2L the image processing unit 22 performs processing such as image correction that absorbs the unique characteristics of the imaging element on the captured left image PL , and then corrects the left image PL . is stored in the image buffer 24 a of the storage unit 24 .
- step S1R the image processing unit 22 causes the right camera 1R to capture the right image PR .
- step S2R the image processing unit 22 performs processing such as image correction that absorbs the unique characteristics of the imaging element on the captured right image PR , and then corrects the right image PR . is stored in the image buffer 24 a of the storage unit 24 .
- step S3 the image processing unit 22 compares the corrected left image PL and right image PR stored in the image buffer 24a, and calculates the parallax between the left and right images.
- the parallax between the left and right images makes it clear where a given point on the object corresponds to where on the left and right images, so the distance to the object can be calculated by the principle of triangulation. . Note that the distance information obtained in this step is also stored in the storage unit 24 .
- step S4 the arithmetic processing unit 23 recognizes various objects using the corrected left and right images and the distance information stored in the image buffer 24a.
- Objects to be recognized include pedestrians, other vehicles, other three-dimensional objects, traffic signs, traffic lights, tail lamps, and the like.
- the discriminator 24b is used as necessary for various object recognitions in this step.
- the discriminator 24b stores and records, for example, features of an object to be recognized as machine learning data. Details of the sign recognition processing (step S40) executed in this step will be described later.
- step S5 the arithmetic processing unit 23 considers the recognition results of various objects in step S4 and the state of the own vehicle (speed, steering angle, etc.) to determine a vehicle control policy. For example, if the own vehicle exceeds the speed limit, a control policy such as issuing a warning to the passengers or braking the own vehicle is determined. In this embodiment, as will be described below, the arithmetic processing unit 23 determines control of the own vehicle based on information on the sign recognized in step S4.
- step S6 the CAN interface 25 outputs the various objects recognized in step S4 and the control policy of the own vehicle determined in step S6 to the external ECU through the in-vehicle network CAN.
- the ECU can issue a warning to the occupants or brake the vehicle in accordance with the control policy determined by the stereo camera device 100 .
- step S40 which is one aspect of the various object recognition processes (step S4) in FIG. 2, will be described using the flowchart in FIG.
- step S41 the arithmetic processing unit 23 extracts an image pattern including a circular object from the post-correction image in the image buffer 24a.
- This processing is divided into edge image generation processing (step S41a), center estimation processing (step S41b), and radius estimation processing (step S41c). Each process will be described in turn.
- the arithmetic processing unit 23 generates an edge image PE1 by extracting edge components of the post-correction image in the image buffer 24a. For example, if an image pattern of a speed limit sign as shown in FIG. 5A is captured in part of the post-correction image, an edge image P E1 as shown in FIG. 5B is generated in this step.
- the arithmetic processing unit 23 estimates the center of the circular object. Specifically, as shown in FIG. 5B, the arithmetic processing unit 23 draws a line segment Ln from each edge of the edge image PE1 in the normal direction, and draws a line segment Ln at a point where a certain number or more of intersections of the line segments Ln overlap. Assume C. In the example of FIG. 5B, the center C is presumed to be between the number "11" and the number "0".
- the arithmetic processing unit 23 estimates the radius of the circular object based on the histogram of the edge image PE1 .
- the histogram of circular objects in FIG. 5B can be generated.
- the horizontal axis position corresponding to the distance from the center C to the first edge group E1 and the horizontal axis position corresponding to the distance from the center C to the second edge group E2 Therefore, it can be assumed that both the first edge group E1 and the second edge group E2 are substantially annular.
- the arithmetic processing unit 23 estimates that the second edge group E2 having the larger radius is the outer diameter of the circular object.
- the arithmetic processing unit 23 can extract a circular object and its features (center position, radius) from the corrected image in the image buffer 24a.
- step S42 the arithmetic processing unit 23 acquires color information of the circular object extracted in step S41. This is because the color information of this sign is used as a reference when judging the authenticity of the deregulation sign in step S4b, which will be described later. Note that step S42 may be processed after step S43, which will be described later, or may be processed in parallel with step S43.
- the arithmetic processing unit 23 identifies whether the circular object extracted at step S41 is a traffic sign candidate. Specifically, the arithmetic processing unit 23 performs identification processing using the classifier 24b on the image pattern of the corrected image in the image buffer 24a, which corresponds to the position of the circular object extracted in step S41. . As a result, it is recognized whether or not the circular object extracted in step S41 is a traffic sign candidate, and if the circular object is a traffic sign candidate, the content of the traffic sign (for example, speed limit) is recognized. A first identification score representing the likelihood of a traffic sign when identified as a traffic sign candidate is output in this identification process and stored in memory. The traffic sign candidates identified in this step are hereinafter referred to as "main signs". If a circular object is not extracted in step S41, this step may be omitted.
- steps S44-45 and 47-4b candidates for deregulation signs are searched.
- step S44 the arithmetic processing unit 23 searches for an effective straight line that can be a candidate for a deregulation sign from the corrected image in the image buffer 24a.
- This effective line search processing is divided into left and right edge search processing (step S44a), line detection processing (step S44b), and effective line determination processing (step S44c). Each process will be described in order with reference to 7C.
- the arithmetic processing unit 23 scans the pixels of the corrected image in the image buffer 24a row by row from left to right, as shown in FIG. 7A. If the left edge E L and the right edge E R can be found within a certain distance, they are extracted as a pair of left and right edges. By applying such processing to the entire area of the post-correction image, an edge image P E2 ( FIG. 7B is an excerpt of the edge image) can be generated by extracting a pair of left and right edges from the post-correction image.
- the arithmetic processing unit 23 detects a straight line portion from the edge image PE2 generated in step S44a.
- the edge image P E2 illustrated in FIG. 7B has a portion in which pairs of left and right edges are continuous in two or more rows and a portion in which there is only one row. In this step, a portion in which two or more rows of pairs of left and right edges continue vertically is detected as a straight line.
- step S44c the arithmetic processing unit 23 determines whether the straight line detected in step S44b is a valid candidate for the deregulation sign, and extracts a valid straight line that can be a candidate.
- Japan's deregulation sign has a conspicuous thick blue slanted line inside a circular sign (see the upper sign in Figure 11(a)).
- release signs eg Germany, Tru, Sweden, etc.
- straight portions that can be candidates for the deregulation sign are extracted from the post-correction image.
- the distance to the straight line detected in step S44b can be calculated. Therefore, assuming that the deregulation sign exists at the calculated distance, the oblique line of the deregulation sign on the corrected image
- the width and height of the part can also be approximated. Therefore, if the width of the edge pair illustrated in FIG. 7B is too narrow or too wide, the straight line formed by the edge pair is determined as an invalid straight line that cannot be a candidate for the deregulation sign, and is excluded from the valid straight lines. can do.
- the height of the straight portion is short and does not reach the lower limit of the effective range that is considered appropriate as the height of the hatched portion of the deregulation sign (FIG. 7C (b) ), or longer than the upper limit (FIG. 7C(c)), these straight lines are determined as invalid straight lines that cannot be candidates for deregulation signs, and are excluded from valid straight lines.
- the straight portion falls within the effective range (FIG. 7C(a))
- the straight portion is extracted as an effective straight line.
- step S44 the image patterns of utility poles and supports that are significantly different in form from the shaded portion of the deregulation sign are excluded from the deregulation sign candidates, and the straight portions that may be deregulation signs are excluded. can extract only the image pattern of
- step S45 the arithmetic processing unit 23 determines whether an effective straight line has been extracted by the processing in step S44. If the effective straight line is not extracted, the process proceeds to step S46, and if the effective straight line is extracted, the process proceeds to step S47.
- step S46 the arithmetic processing unit 23 tracks the main sign (traffic sign) candidate identified in step S43.
- “tracking” refers to the shape information and position information of the sign candidate obtained in step S41, the color information obtained in step S42, and the color It refers to storing information in memory in association with time series.
- the type of the traffic sign candidate identified in step S43 (for example, a speed limit sign) is registered in the tracking list L1, which is a list for registering the type of traffic sign being imaged by the stereo camera device 100. . Note that when the own vehicle passes through the main sign candidate registered in the tracking list L1 and becomes unable to take an image of the main sign candidate, the information on the main sign candidate registered in the tracking list L1 is not recognized. It is assumed that it is transcribed to list L2.
- step S47 the arithmetic processing unit 23 calculates color information of the effective straight line extracted in step S44.
- the arithmetic processing unit 23 arranges a window having a shape that accommodates an effective straight line in the image, and Calculate the blue color score.
- the blue score is calculated by, for example, the following method. That is, if the color of each pixel in the image is defined by the three primary colors red (R), green (G), and blue (B), the area of the blue pixel in the window is divided by the total area of the window. By doing so, the blue score is calculated.
- the calculated blue score is stored in memory.
- blue is used as an example here, color information other than blue may be extracted according to the sign design. It says.
- the deregulation sign in Japan is designed with a conspicuous blue diagonal line (see Fig. 11(a)), so the blue score in the window that actually includes the deregulation sign is considered to be high. Therefore, by focusing on the blue score of the window, for example, effective straight lines clearly other than blue, such as white straight lines and black straight lines, are excluded from candidates for deregulation signs. Only effective straight lines can be extracted.
- the threshold for comparison with the blue score is set low in this step.
- not only actual deregulation signs, but also effective straight lines that are not deregulation signs may be extracted as deregulation sign candidates. Candidates for deregulation signs at this step are not reliable enough.
- step S48 candidates for deregulation signs are identified. This is performed in the same manner as the identification of the main mark in S42. Specifically, the arithmetic processing unit 23 uses the discriminator 24b to discriminate the deregulation sign candidates. At this time, a second identification score representing the likeness of the deregulation sign is output in this identification process and stored in memory.
- the deregulation sign candidates identified in this step are hereinafter referred to as "deregulation signs". Note that step S48 may be processed before step S47, or may be processed in parallel with step S47.
- step S49 the arithmetic processing unit 23 tracks the actual sign identified in step S42 and the candidates for the deregulation sign extracted in step S48.
- “tracking" has the same meaning as described in step S46.
- the tracking list L1 which is a list for registering the signs being imaged by the stereo camera device 100
- the type of the main sign candidate identified in step S43 for example, a speed limit sign
- the type extracted in step S48 Register the deregulation sign (candidate). Note that if the vehicle passes through a traffic sign and cannot take an image of the traffic sign registered in the tracking list L1, the information on the traffic sign registered in the tracking list L1 is transferred to the recognized list L2. shall be transcribed.
- step S4a the arithmetic processing unit 23 searches for a real sign (for example, speed limit sign, no overtaking) paired with the deregulation sign candidate extracted in step S48.
- the method of searching for paired signs in this step differs from country to country. For example, in Japan, as shown in FIG. 8A, this sign is placed below the deregulation sign. should be associated as On the other hand, in other countries, as shown in FIG. It suffices to associate the registered deregulation signs as a pair sign. By using either search method, it is possible to search for this sign that is paired with the derestriction sign regardless of the country.
- step S4a the details of the pair sign search process in step S4a will be described using the flowchart of FIG. This flowchart is divided into processing from step S4aa to step S4ad and processing from step S4af to step S4ah.
- step S4aa to step S4ad the arithmetic processing unit 23 searches for pair signs based on the tracking list L1. This is a search method for paired signs in countries where this sign and deregulated sign are installed at the same location, as illustrated in FIG. 8A.
- step S4ab the arithmetic processing unit 23, based on the tracking information registered in the tracking list L1 in step S49 (information for specifying the sign such as the set position, size, type, and tracking number of the sign) Check if there is a main sign paired with the deregulation sign. For example, as shown in FIG. 8A, if a deregulation sign and a speed limit sign are registered in the tracking list L1, they are extracted as pair sign candidates.
- step S4ac the arithmetic processing unit 23 confirms the arrangement of the deregulation sign and the pair sign candidate, and determines that both are pair signs if their installation positions are close. For example, as shown in FIG. 10A, a speed limit sign, which is the first paired sign candidate, is installed on the same post as the deregulation sign, and a second pair sign candidate is installed on a post far from the deregulation sign. If there is an overtaking prohibition sign, the speed limit sign installed on the same pole as the deregulation sign is determined as a pair sign, and the overtaking prohibition sign in the distance is not determined as a pair sign. In addition, when a plurality of permanent signs are installed on the same post as the deregulation sign, the plurality of permanent signs may be extracted as a pair of signs.
- step S4ad the arithmetic processing unit 23 stores information on the paired signs extracted in step 48c in the storage unit 24.
- step S4ae the arithmetic processing unit 23 confirms whether the pair indicator has been registered. Then, when the pair indicator is registered, the process of step S4a is terminated and the process proceeds to step S4b. On the other hand, if the pair indicator is unregistered, the process proceeds to step S4af.
- step S4af to step S4ah the arithmetic processing unit 23 searches for pair signs based on the tracking list L1 and the recognized list L2. As illustrated in FIG. 8B, this is a search method for paired signs in countries where the main sign and deregulation sign are installed in separate locations.
- step S4ag the arithmetic processing unit 23, based on the recognized information registered in the recognized list L2 (previously recognized sign information such as the setting position, size, type, and registration number of the sign), It is checked whether or not there is a regular sign paired with the deregulation sign registered in L1. For example, as shown in FIG. 8B, if a previously imaged speed limit sign is registered in the recognized list L2, it is extracted as a pair sign candidate. Since there is a possibility that a large number of previously recognized sign information are registered in the recognized list L2, in this step, the most recently registered main sign is determined as a pair sign candidate. For example, as shown in FIG.
- a speed limit sign which is the first pair of candidate signs, is installed on the pole in front of the deregulation sign, and the second pair of sign candidates is installed on the pole in front of the speed limit sign. If a no overtaking sign is installed, the speed limit sign registered most recently is determined to be a paired sign, and the previously registered no overtaking sign is determined not to be a paired sign.
- step S4aa to step S4ad corresponding to the environment in FIG. 8A and the processing from step S4af to step S4ah corresponding to the environment in FIG. 8B are executed. If the country in which the vehicle is traveling can be identified using It is good also as a structure to carry out.
- step S4a After completing the process of step S4a shown in FIG. 9, the main sign that is paired with the deregulation sign can be extracted regardless of the installation mode of the deregulation sign (FIGS. 8A and 8B).
- step S45 as described above, there is a possibility that an effective straight line that is not actually a deregulation sign is extracted as a candidate.
- the color information of the actual sign it is confirmed whether or not the candidate extracted in step S48 is truly a deregulation sign.
- the color information of the main sign is acquired in step S48) and stored in the memory.
- step S4b the arithmetic processing unit 23 determines whether it is a time zone in which it is possible to perform the authenticity determination process (step S4c) of the candidate deregulation sign.
- step S4c the color information of the paired signs is used. reliability is low. Therefore, in this step, for example, based on the time information acquired from the built-in timer of the ECU or the GPS, it is determined whether it is daytime or nighttime. Proceeding to step S4c, it is determined that it is difficult to obtain highly reliable color information at night, and the process proceeds to step S4d.
- step S4c determines whether the candidate for this sign is extracted in step S43 or not extracted in step S43 or not extracted in step S48. If the candidate for this sign is not extracted in step S43, the process of step S4c cannot be executed, and if the candidate for the deregulation sign is not extracted in step S48, the truth of the candidate is determined. Since the process of step S4c for determining is not necessary in the first place, step S4c is avoided in these cases as well, and the process proceeds to step S4c.
- step S4c the arithmetic processing unit 23 determines whether the candidates for deregulation signs extracted in step S48 are true or false. reject.
- candidates for deregulation signs are extracted based on the blue score of the effective straight line, but deregulation signs illuminated with colored light and deregulation signs in cloudy or rainy weather are excluded from the candidates.
- the threshold for comparison with the blue score of the effective straight line was set to be low, which allowed the extraction of effective straight lines that were not actual deregulation signs as candidates. Therefore, in this step, by referring to the color score of the main sign acquired in step S42 and adjusting the threshold value used to determine the authenticity of the candidates for the deregulation sign, the deregulation sign It is now possible to accurately judge the authenticity of a candidate.
- this sign is a speed limit sign with a red ring around the sign
- the red score within the window shaped to accommodate the red ring is considered to be a high value.
- this sign is illuminated by colored light, cloudy or rainy weather, clear weather, etc.
- the red score of this sign will change. In that case, the blue score of the deregulation sign would also change to the same extent.
- the threshold of the blue score for judging the authenticity of the candidate of the deregulation sign is also set low. If it is high, the blue score threshold is also set high.
- the color information for example, blue score (that is, confidence in being blue)
- the rate of change in color information acquired when recognizing this sign against the color information of the template image of this sign to recognize it as a deregulation sign
- a threshold for color information may be determined. Together, the color information of the sign and the rate of change of the color information of the sign are referred to as the "reliability" of the sign.
- threshold value of color information for example, blue score
- step S4d the arithmetic processing unit 23 refers to the tracking list L1 or the recognized list L2 to determine the result of identifying the sign. That is, if the process proceeds from step S45 to step S46, only the main marker candidate identified in step S43 is determined as the identification result as a marker. On the other hand, if the process proceeds from step S45 to step S47, for example, the main sign candidate identified in step S43 is identified as a pair sign of the deregulation sign candidate extracted in steps S48 and S4c. Then, when the pair sign is recognized in this step, a control policy that cancels the regulation indicated by this sign is determined in the subsequent step S5.
- “determine” means outputting the information obtained from the recognized sign to the subsequent control process (S5 in FIG. 2) as the result of the recognition process.
- the stereo camera device 100 of the present embodiment described above not only can the deregulation sign shown in FIG. Since it is possible to suppress erroneous detection of the upper ends of the posts having similar thicknesses as the deregulation sign, it is possible to improve the reliability of sign detection.
- the stereo camera device 100 composed of two cameras was used, but one camera or three or more cameras may be used.
- each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them using an integrated circuit.
- each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function.
- Information such as programs, tables, and files that implement each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.
- DESCRIPTION OF SYMBOLS 100 Stereo camera apparatus, 1... Camera, 1L... Left camera, 1R... Right camera, 2... Sign recognition apparatus, 21... Image input interface, 22... Image processing part, 23... Operation process part, 24... Storage part, 24a ... image buffer, 24b ... discriminator, 25 ... CAN interface, 26 ... monitoring processing unit, 27 ... internal bus, CAN in-vehicle network
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
Provided is a processing device capable of improving the performance of recognizing an object in an image such as a restriction lifting sign. This sign recognition device is characterized by comprising: a candidate sign recognition unit that recognizes a candidate sign of a predetermined type from an image; a paired sign search unit that associates, if a candidate for a first sign is recognized as the candidate sign of the predetermined type, the first sign with a second sign installed in correspondence with the first sign; a color information extraction unit that extracts, if there is a second sign installed in correspondence with the first sign, information on the color of the second sign associated with the first sign; and a threshold value determination unit that determines a threshold value for extracting information on the color of the candidate for the first sign on the basis of the reliability of the information on the color of the second sign extracted by the color information extraction unit.
Description
本発明は、ステレオカメラ装置などの車載センシング装置で利用される、標識認識装置、および、標識認識方法に関する。
The present invention relates to a sign recognition device and a sign recognition method used in an in-vehicle sensing device such as a stereo camera device.
近年、自動運転制御や運転支援制御等の、車外環境に応じた高度な車両制御の実現に向け、車載センシング装置の各種認識機能への要求も高度化している。車載センシング装置の一種であるステレオカメラ装置は、画像に依る視覚的な情報と、画像内の物体の距離情報を同時に検出する装置であるため、自車周辺の様々な環境(歩行者、他車、立体物、路面、路面標識、看板標識など)を詳細に把握でき、自動運転制御や運転支援制御の安全性向上に大きく寄与する。
In recent years, the demand for various recognition functions of in-vehicle sensing devices has also increased toward the realization of advanced vehicle control that responds to the environment outside the vehicle, such as automatic driving control and driving support control. A stereo camera device, which is a type of in-vehicle sensing device, is a device that simultaneously detects visual information based on images and distance information of objects in the image. , three-dimensional objects, road surfaces, road signs, signage signs, etc.) can be grasped in detail, greatly contributing to the safety improvement of automatic driving control and driving support control.
ステレオカメラ装置を搭載した車両のなかには、認識した交通標識を自車の加減速制御等に使用するものがある。また、先進運転支援システムの評価指標であるEuro NCAPにおいても、速度支援システム(SAS、Speed Assistance Systems)に関する評価項目が設けられており、その重要度が増している。日本国内においても、ドライバの不注意や誤操作による速度超過を抑制する自動速度制御機能(ISA、Intelligent Speed Assistance)の重要性が高まっており、速度制限情報を取得する標識認識機能の認識精度向上が必要になる。さらに、有効区間、時間制限や車型制限などの条件付き速度制限の認識が必要になるため、認識可能な交通標識の拡大が求められている。
Some vehicles equipped with a stereo camera device use the recognized traffic signs for acceleration and deceleration control of the vehicle. In addition, Euro NCAP, which is an evaluation index for advanced driver assistance systems, also has evaluation items related to speed assistance systems (SAS), and their importance is increasing. In Japan as well, the importance of automatic speed control functions (ISA, Intelligent Speed Assistance) that suppress overspeeding due to driver carelessness or erroneous operation is increasing, and the recognition accuracy of the sign recognition function that acquires speed limit information is increasing. become necessary. Furthermore, recognition of conditional speed limits such as valid sections, time limits, and vehicle type limits is required, so there is a demand for an expansion of recognizable traffic signs.
上記した事情により、車両前方の環境を認識する車載カメラ装置に関する様々な技術が提案されてきた。例えば、規制解除標識(速度制限、追い越し禁止などの規制を解除する標識)の認識性能の向上に着目した従来技術として、特許文献1が挙げられ、同文献の要約には、「規制解除標識の検知精度を向上することができるカメラ装置を提供する。演算処理部(探索部)は、画像から斜線候補を探索する(斜線探索504)。演算処理部(選択部)は、検出された斜線候補から本標識に対応する規制の解除を示す規制解除標識の斜線候補を選択する(斜線判定505)。演算処理部(規制解除標識識別部)は、選択された規制解除標識の斜線候補の画像から規制解除標識を識別する(識別処理506)。」と記載されている。
Due to the above circumstances, various technologies related to in-vehicle camera devices that recognize the environment in front of the vehicle have been proposed. For example, Patent Document 1 is cited as a conventional technology that focuses on improving the recognition performance of deregulation signs (signs that release regulations such as speed limits and overtaking prohibitions). A camera device capable of improving detection accuracy is provided.Arithmetic processing unit (searching unit) searches for a diagonal line candidate from an image (diagonal line search 504).Arithmetic processing unit (selecting unit) selects a detected diagonal line candidate. from the image of the selected oblique line candidate of the deregulation sign is selected (diagonal line determination 505). identify the deregulation sign (identification process 506).".
特許文献1のカメラ装置は、斜線候補の画像から規制解除標識を識別するものであるため、撮像画像の一部に図11(a)のような画像パターンがある場合、速度制限標識の上方の斜線を規制解除標識として正しく識別できる。
Since the camera device of Patent Document 1 identifies the deregulation sign from the image of the diagonal candidate, if the captured image has an image pattern as shown in FIG. A diagonal line can be correctly identified as a deregulation sign.
しかしながら、特許文献1のカメラ装置では、撮像画像の一部に図11(b)のような画像パターンがある場合、速度制限標識の上方に延びる支柱の上端の長さ、太さ、角度が、規制解除標識の斜線部と類似しているため、標識上方の支柱を規制解除標識として誤検知してしまう可能性がある。
However, in the camera device of Patent Document 1, when an image pattern as shown in FIG. Since it resembles the shaded part of the deregulation sign, there is a possibility that the post above the sign will be misidentified as the deregulation sign.
本発明は、上記課題に鑑みてなされたもので、その目的とするところは、規制解除標識のデザインと類似する、支柱、電柱、木の枝などの画像パターンを、非実在の規制解除標識として誤検知することを抑制する、標識認識装置および標識認識方法を提供することである。
The present invention has been made in view of the above problems, and its object is to use an image pattern similar to the design of a deregulation sign, such as a pole, a utility pole, a tree branch, etc., as a non-existent deregulation sign. An object of the present invention is to provide a sign recognition device and a sign recognition method that suppress erroneous detection.
上記課題を解決するため、本発明の標識認識装置は、画像から所定の種別の標識の候補を認識する標識候補認識部と、前記所定の種別の標識の候補として、第1標識の候補が認識された場合、当該第1標識と、当該第1標識に対応して設置されている第2標識を関連付けるペア標識探索部と、第1標識に対応して設置されている第2標識が存在する場合、前記第1標識に関連付けた前記第2標識の色情報を抽出する色情報抽出部と、該色情報抽出部で抽出した前記第2標識の色情報の信頼度に基づいて、前記第1標識の候補の色情報を抽出する際の閾値を決定する閾値決定部と、を有する標識認識装置とした。
In order to solve the above-described problems, the sign recognition apparatus of the present invention includes a sign candidate recognition unit that recognizes a sign candidate of a predetermined type from an image, and a first sign candidate that recognizes a first sign candidate as the sign candidate of the predetermined type. If so, there is a paired sign searching unit that associates the first sign with the second sign installed corresponding to the first sign, and the second sign installed corresponding to the first sign. a color information extraction unit for extracting color information of the second sign associated with the first sign; and a threshold determination unit for determining a threshold when extracting color information of a candidate for a sign.
本発明の標識認識装置および標識認識方法によれば、規制解除標識のデザインと類似する、支柱、電柱、木の枝などの画像パターンを、非実在の規制解除標識として誤検知することを抑制することができる。
According to the sign recognition device and the sign recognition method of the present invention, it is possible to suppress false detection of image patterns similar to the design of deregulation signs, such as poles, utility poles, tree branches, etc., as non-existent deregulation signs. be able to.
上記した以外の課題、構成及び効果は以下の実施例の説明により明らかにされる。
Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.
以下、本発明の一実施例に係る標識認識装置を、図面を用いて説明する。なお、各図において同じ機能を有する部分には同じ符号を付して繰り返し説明は省略する場合がある。
A sign recognition device according to an embodiment of the present invention will be described below with reference to the drawings. Note that portions having the same function in each drawing may be denoted by the same reference numerals, and repeated description may be omitted.
<ステレオカメラ装置の概略構成>
図1は、本発明の一実施例に係るステレオカメラ装置100の概略構成を示す機能ブロック図である。このステレオカメラ装置100は、自動運転制御や運転支援制御を実行する車両に搭載される車載センシング装置の一種であり、自車前方の撮影対象領域の画像情報に基づいて車外環境(例えば、道路の白線、歩行者、他車、その他の立体物、交通信号機、交通標識、点灯ランプなど)を認識する装置である。また、ステレオカメラ装置100は、車外環境に応じた自車の制御方針(ブレーキやステアリングなど)を決定し、車載ネットワークCAN(Controller Area Network)を介して、ECU(Electronic Control Unit)に制御方針を出力する装置である。そして、ECUは、ステレオカメラ装置100の制御方針に従って、自車の制動系や操舵系を制御することで、自車を安全に減速させたり、自車に障害物を回避させたりする。 <Schematic configuration of stereo camera device>
FIG. 1 is a functional block diagram showing a schematic configuration of astereo camera device 100 according to one embodiment of the invention. The stereo camera device 100 is a type of in-vehicle sensing device mounted in a vehicle that executes automatic driving control and driving support control, and is based on image information of a shooting target area in front of the vehicle. It is a device that recognizes white lines, pedestrians, other vehicles, other three-dimensional objects, traffic lights, traffic signs, lighting lamps, etc.). Also, the stereo camera device 100 determines a control policy (brake, steering, etc.) of the own vehicle according to the environment outside the vehicle, and provides the control policy to an ECU (Electronic Control Unit) via an in-vehicle network CAN (Controller Area Network). It is an output device. Then, the ECU controls the braking system and the steering system of the own vehicle according to the control policy of the stereo camera device 100, thereby safely decelerating the own vehicle and causing the own vehicle to avoid obstacles.
図1は、本発明の一実施例に係るステレオカメラ装置100の概略構成を示す機能ブロック図である。このステレオカメラ装置100は、自動運転制御や運転支援制御を実行する車両に搭載される車載センシング装置の一種であり、自車前方の撮影対象領域の画像情報に基づいて車外環境(例えば、道路の白線、歩行者、他車、その他の立体物、交通信号機、交通標識、点灯ランプなど)を認識する装置である。また、ステレオカメラ装置100は、車外環境に応じた自車の制御方針(ブレーキやステアリングなど)を決定し、車載ネットワークCAN(Controller Area Network)を介して、ECU(Electronic Control Unit)に制御方針を出力する装置である。そして、ECUは、ステレオカメラ装置100の制御方針に従って、自車の制動系や操舵系を制御することで、自車を安全に減速させたり、自車に障害物を回避させたりする。 <Schematic configuration of stereo camera device>
FIG. 1 is a functional block diagram showing a schematic configuration of a
図1に示すように、本実施例のステレオカメラ装置100は、カメラ1と標識認識装置2を有する。カメラ1は、左右に横並びに配置した左カメラ1Lと右カメラ1Rで構成され、自車前方を同期撮像した左右一対の画像(左画像PL、右画像PR)を出力する。
As shown in FIG. 1, the stereo camera device 100 of this embodiment has a camera 1 and a sign recognition device 2. As shown in FIG. The camera 1 is composed of a left camera 1L and a right camera 1R arranged side by side, and outputs a pair of left and right images (left image P L and right image P R ) synchronously photographed in front of the vehicle.
標識認識装置2は、カメラ1で撮像した画像内の交通標識を認識する。この標識認識装置2は、具体的には、CPU(Central Processing Unit)等のプロセッサ、ROM(Read Only Memory)、RAM(Random Access Memory)、SSD(Solid State Drive)等のメモリ等を備える、単一または複数のコンピュータユニットである。標識認識装置2の各機能は、ROMに記憶されたプログラムをプロセッサが実行することによって実現される。RAMやSSDは、プログラムによる演算の中間データやカメラ1の画像等のデータを格納する。なお、上記したように、ステレオカメラ装置100は、交通標識以外の車外環境(道路の白線、歩行者、他車など)も認識可能な装置であるが、本発明はステレオカメラ装置100の交通標識認識機能に着目したものであるため、本来、環境認識装置とでも称すべき符号2の装置を、本実施例では標識認識装置2と称することとする。
The sign recognition device 2 recognizes traffic signs in the image captured by the camera 1. Specifically, the sign recognition device 2 includes a processor such as a CPU (Central Processing Unit), a memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). One or more computer units. Each function of the sign recognition device 2 is realized by the processor executing a program stored in the ROM. The RAM and SSD store data such as intermediate data for computation by the program and images of the camera 1 . As described above, the stereo camera device 100 is a device capable of recognizing the environment outside the vehicle (white lines on the road, pedestrians, other vehicles, etc.) other than traffic signs. Since the recognition function is focused on, the device 2, which should be called the environment recognition device, will be called the sign recognition device 2 in this embodiment.
この標識認識装置2は、図示するように、画像入力インタフェース21と、画像処理部22と、演算処理部23と、記憶部24と、CANインタフェース25と、監視処理部26と、それらを相互接続する内部バス27を有している。以下、各部を概説する。
As shown, the sign recognition device 2 includes an image input interface 21, an image processing unit 22, an arithmetic processing unit 23, a storage unit 24, a CAN interface 25, a monitoring processing unit 26, and interconnecting them. It has an internal bus 27 that Each part will be outlined below.
画像入力インタフェース21は、カメラ1の撮像素子を制御して、撮像した画像を取り込むインタフェースである。この画像入力インタフェース21が取り込んだ画像は、内部バス27を介して、画像処理部22や演算処理部23に送信される。
The image input interface 21 is an interface that controls the imaging device of the camera 1 and captures the captured image. The image captured by the image input interface 21 is transmitted to the image processing section 22 and the arithmetic processing section 23 via the internal bus 27 .
画像処理部22は、左カメラ1Lの撮像素子で撮像した左画像PLと、右カメラ1Rの撮像素子で撮像した右画像PRを比較し、各画像に対して撮像素子に起因するデバイス固有の偏差補正やノイズ補間などの画像補正を施した後、補正後の左右画像を記憶部24に記憶する。また、画像処理部22は、補正後の左右画像間で相互に対応する箇所を抽出した後、対応箇所同士の視差に基づいて、画像上の各画素の距離情報を計算する。そして、計算した距離情報を記憶部24に記憶する。
The image processing unit 22 compares the left image PL captured by the imaging device of the left camera 1L and the right image PR captured by the imaging device of the right camera 1R, and determines device-specific After performing image correction such as deviation correction and noise interpolation, the left and right images after correction are stored in the storage unit 24 . Further, the image processing unit 22 extracts mutually corresponding portions between the corrected left and right images, and then calculates distance information of each pixel on the image based on the parallax between the corresponding portions. Then, the calculated distance information is stored in the storage unit 24 .
演算処理部23は、記憶部24に記憶した補正後の左右画像と距離情報を使い、自車周辺の環境を知覚するために必要な、各種物体を認識する。ここで認識される各種物体とは、例えば、歩行者、他車、その他の障害物、交通信号機、交通標識、車のテールランプやヘッドライド、などである。これら認識結果や中間的な計算結果の一部が、記憶部24に記録される。さらに、演算処理部23は、各種物体の認識結果を用いて自車の制御方針を決定する。
The arithmetic processing unit 23 uses the corrected left and right images and the distance information stored in the storage unit 24 to recognize various objects necessary for perceiving the environment around the vehicle. Various objects recognized here include, for example, pedestrians, other vehicles, other obstacles, traffic lights, traffic signs, tail lamps and headlights of vehicles, and the like. Some of these recognition results and intermediate calculation results are recorded in the storage unit 24 . Furthermore, the arithmetic processing unit 23 determines a control strategy for the own vehicle using the recognition results of various objects.
記憶部24は、画像処理部22や演算処理部23による処理中または処理後のデータを記憶する。また、記憶部24には、各種物体を認識するために必要な各種情報が予め登録されている。なお、記憶部24は、具体的には、上記したRAMやSSD等のメモリであり、後述する画像バッファ24aや識別器24bを備えている。
The storage unit 24 stores data during or after processing by the image processing unit 22 and the arithmetic processing unit 23 . Further, various information necessary for recognizing various objects is registered in advance in the storage unit 24 . Note that the storage unit 24 is specifically a memory such as the above-described RAM or SSD, and includes an image buffer 24a and a discriminator 24b, which will be described later.
CANインタフェース25は、演算処理部23で認識した各種物体や、演算処理部23で決定した車両の制御方針を、車載ネットワークCANに送信するインタフェースである。車載ネットワークCANに接続されたECUは、車載ネットワークCANを経由して伝えられた物体認識結果や制御方針に基づいて、自車の制動やドライバへの警報などを実行する。
The CAN interface 25 is an interface that transmits various objects recognized by the arithmetic processing unit 23 and vehicle control policies determined by the arithmetic processing unit 23 to the in-vehicle network CAN. The ECU connected to the in-vehicle network CAN executes braking of the own vehicle, warning to the driver, etc., based on the object recognition result and control policy transmitted via the in-vehicle network CAN.
監視処理部26は、上記した各部が異常動作を起こしていないか、データ転送時にエラーが発生していないかなどを監視しており、異常動作を防ぐ仕掛けとなっている。
The monitoring processing unit 26 monitors whether any of the above-described units are operating abnormally, or whether an error has occurred during data transfer, and is a mechanism for preventing abnormal operations.
<ステレオカメラ装置の基本処理>
ここで、図2のフローチャートを用いて、本実施例のステレオカメラ装置100による基本処理を説明する。 <Basic processing of the stereo camera device>
Here, basic processing by thestereo camera device 100 of the present embodiment will be described using the flowchart of FIG.
ここで、図2のフローチャートを用いて、本実施例のステレオカメラ装置100による基本処理を説明する。 <Basic processing of the stereo camera device>
Here, basic processing by the
まず、ステップS1Lでは、画像処理部22は、左カメラ1Lに左画像PLを撮像させる。次に、ステップS2Lでは、画像処理部22は、撮像した左画像PLに対して、撮像素子が持つ固有の特性を吸収する画像補正等の処理を施した後、補正後の左画像PLを記憶部24の画像バッファ24aに記憶する。
First, in step S1L, the image processing unit 22 causes the left camera 1L to capture the left image PL . Next, in step S2L, the image processing unit 22 performs processing such as image correction that absorbs the unique characteristics of the imaging element on the captured left image PL , and then corrects the left image PL . is stored in the image buffer 24 a of the storage unit 24 .
また、ステップS1L,S2Lと同期して、ステップS1Rでは、画像処理部22は、右カメラ1Rに右画像PRを撮像させる。次に、ステップS2Rでは、画像処理部22は、撮像した右画像PRに対して、撮像素子が持つ固有の特性を吸収する画像補正等の処理を施した後、補正後の右画像PRを記憶部24の画像バッファ24aに記憶する。
Further, in synchronism with steps S1L and S2L, in step S1R, the image processing unit 22 causes the right camera 1R to capture the right image PR . Next, in step S2R, the image processing unit 22 performs processing such as image correction that absorbs the unique characteristics of the imaging element on the captured right image PR , and then corrects the right image PR . is stored in the image buffer 24 a of the storage unit 24 .
ステップS3では、画像処理部22は、画像バッファ24aに蓄えた補正後の左画像PLと右画像PRを照合し、左右画像の視差を計算する。左右画像の視差により、対象物体上のある着目点が、左右の画像上の何処と何処に対応するかが明らかとなるので、三角測量の原理によって、対象物までの距離を演算することができる。なお、本ステップで求めた距離情報も、記憶部24に記憶される。
In step S3, the image processing unit 22 compares the corrected left image PL and right image PR stored in the image buffer 24a, and calculates the parallax between the left and right images. The parallax between the left and right images makes it clear where a given point on the object corresponds to where on the left and right images, so the distance to the object can be calculated by the principle of triangulation. . Note that the distance information obtained in this step is also stored in the storage unit 24 .
ステップS4では、演算処理部23は、画像バッファ24aに記憶された補正後の左右画像と距離情報を用いて、各種物体を認識する。認識対象の物体としては、歩行者、他車、その他の立体物、交通標識、交通信号機、テールランプなどである。本ステップでの各種物体認識には必要に応じて識別器24bを利用する。識別器24bは、例えば認識対象の物体の特徴を機械学習データとして保存・記録したものである。なお、本ステップ中で実行される、標識認識処理(ステップS40)の詳細については後述する。
In step S4, the arithmetic processing unit 23 recognizes various objects using the corrected left and right images and the distance information stored in the image buffer 24a. Objects to be recognized include pedestrians, other vehicles, other three-dimensional objects, traffic signs, traffic lights, tail lamps, and the like. The discriminator 24b is used as necessary for various object recognitions in this step. The discriminator 24b stores and records, for example, features of an object to be recognized as machine learning data. Details of the sign recognition processing (step S40) executed in this step will be described later.
ステップS5では、演算処理部23は、ステップS4での各種物体の認識結果と、自車の状態(速度、舵角など)を勘案して、車両制御の方針を決定する。例えば、自車が制限速度を超過している場合であれば、乗員に警告を発したり、自車を制動したりするなどの制御方針を決める。本実施例においては、以下に説明するように、ステップS4で認識された標識の情報に基づいて、演算処理部23が自車の制御を決定する。
In step S5, the arithmetic processing unit 23 considers the recognition results of various objects in step S4 and the state of the own vehicle (speed, steering angle, etc.) to determine a vehicle control policy. For example, if the own vehicle exceeds the speed limit, a control policy such as issuing a warning to the passengers or braking the own vehicle is determined. In this embodiment, as will be described below, the arithmetic processing unit 23 determines control of the own vehicle based on information on the sign recognized in step S4.
ステップS6では、CANインタフェース25は、ステップS4で認識した各種物体や、ステップS6で決定した自車の制御方針を、車載ネットワークCANを通して外部のECUに出力する。この結果、ECUは、ステレオカメラ装置100で決定した制御方針に従い、乗員に警告を発したり、自車を制動したりすることができる。
In step S6, the CAN interface 25 outputs the various objects recognized in step S4 and the control policy of the own vehicle determined in step S6 to the external ECU through the in-vehicle network CAN. As a result, the ECU can issue a warning to the occupants or brake the vehicle in accordance with the control policy determined by the stereo camera device 100 .
<標識認識処理>
次に、図3のフローチャートを用いて、図2の各種物体認識処理(ステップS4)の一態様である、標識認識処理(ステップS40)を説明する。 <Sign recognition processing>
Next, the sign recognition process (step S40), which is one aspect of the various object recognition processes (step S4) in FIG. 2, will be described using the flowchart in FIG.
次に、図3のフローチャートを用いて、図2の各種物体認識処理(ステップS4)の一態様である、標識認識処理(ステップS40)を説明する。 <Sign recognition processing>
Next, the sign recognition process (step S40), which is one aspect of the various object recognition processes (step S4) in FIG. 2, will be described using the flowchart in FIG.
まず、ステップS41では、演算処理部23は、画像バッファ24a内の補正後画像から円形物体を含む画像パターンを抽出する。この処理は、詳細には、エッジ画像生成処理(ステップS41a)と、中心推定処理(ステップS41b)と、半径推定処理(ステップS41c)に分けられるため、以下では、図4から図5Bを参照しながら、各処理を順に説明する。
First, in step S41, the arithmetic processing unit 23 extracts an image pattern including a circular object from the post-correction image in the image buffer 24a. This processing is divided into edge image generation processing (step S41a), center estimation processing (step S41b), and radius estimation processing (step S41c). Each process will be described in turn.
まず、エッジ画像生成処理(ステップS41a)では、演算処理部23は、画像バッファ24a内の補正後画像のエッジ成分を抽出したエッジ画像PE1を生成する。例えば、補正後画像の一部に図5Aのような速度制限標識の画像パターンが撮像されていた場合、本ステップでは、図5Bのようなエッジ画像PE1が生成される。
First, in the edge image generation process (step S41a), the arithmetic processing unit 23 generates an edge image PE1 by extracting edge components of the post-correction image in the image buffer 24a. For example, if an image pattern of a speed limit sign as shown in FIG. 5A is captured in part of the post-correction image, an edge image P E1 as shown in FIG. 5B is generated in this step.
次に、中心推定処理(ステップS41b)では、演算処理部23は、円形物体の中心を推定する。具体的には、演算処理部23は、図5Bに示すように、エッジ画像PE1の各エッジから法線方向に線分Lnを引き、その線分Lnの交点が一定数以上重なる点を中心Cと推定する。図5Bの例では、数字「11」と数字「0」の間が中心Cと推定される。
Next, in the center estimation process (step S41b), the arithmetic processing unit 23 estimates the center of the circular object. Specifically, as shown in FIG. 5B, the arithmetic processing unit 23 draws a line segment Ln from each edge of the edge image PE1 in the normal direction, and draws a line segment Ln at a point where a certain number or more of intersections of the line segments Ln overlap. Assume C. In the example of FIG. 5B, the center C is presumed to be between the number "11" and the number "0".
半径推定処理(ステップS41c)では、演算処理部23は、エッジ画像PE1のヒストグラムに基づいて円形物体の半径を推定する。例えば、図5Bの円形物体の半径を推定する場合であれば、横軸を中心Cからの距離とし、縦軸をエッジ数としたヒストグラムを作成する。具体的には、まず、中心Cに半径r=0の円を配置し、この円の半径rを徐々に大きくする。そして、この円とエッジが重なった場合、重なったエッジの数を、その距離でのエッジ数としてカウントする。この処理を、半径rが所定値(交通標識として妥当な半径)になるまで継続することで、図5Bの円形物体のヒストグラムを生成することができる。このようにして生成した、図5Bに基づくヒストグラムでは、中心Cから第1エッジ群E1までの距離に相当する横軸位置と、中心Cから第2エッジ群E2までの距離に相当する横軸位置の2箇所に、多数のエッジが存在するため、第1エッジ群E1と第2エッジ群E2の双方が略環状であると推測できる。この場合、演算処理部23は、半径のより大きい、第2エッジ群E2を円形物体の外径であると推定する。
In the radius estimation process (step S41c), the arithmetic processing unit 23 estimates the radius of the circular object based on the histogram of the edge image PE1 . For example, when estimating the radius of the circular object in FIG. 5B, a histogram is created with the horizontal axis representing the distance from the center C and the vertical axis representing the number of edges. Specifically, first, a circle having a radius of r=0 is arranged at the center C, and the radius r of this circle is gradually increased. Then, when this circle overlaps an edge, the number of overlapped edges is counted as the number of edges at that distance. By continuing this process until the radius r reaches a predetermined value (a radius appropriate for a traffic sign), the histogram of circular objects in FIG. 5B can be generated. In the histogram based on FIG. 5B generated in this way, the horizontal axis position corresponding to the distance from the center C to the first edge group E1 and the horizontal axis position corresponding to the distance from the center C to the second edge group E2 Therefore, it can be assumed that both the first edge group E1 and the second edge group E2 are substantially annular. In this case, the arithmetic processing unit 23 estimates that the second edge group E2 having the larger radius is the outer diameter of the circular object.
以上のような処理により、演算処理部23は、画像バッファ24a内の補正後画像から、円形物体とその特徴(中心位置、半径)を抽出することができる。
Through the above-described processing, the arithmetic processing unit 23 can extract a circular object and its features (center position, radius) from the corrected image in the image buffer 24a.
ステップS42では、演算処理部23は、ステップS41で抽出した円形物体の色情報を取得する。これは、後述するステップS4bで、規制解除標識の真偽を判定するときに、本標識の色情報を基準にして比較するためである。なお、ステップS42は、後述するステップS43の次に処理しても良いし、ステップS43と並列して処理しても良い。
In step S42, the arithmetic processing unit 23 acquires color information of the circular object extracted in step S41. This is because the color information of this sign is used as a reference when judging the authenticity of the deregulation sign in step S4b, which will be described later. Note that step S42 may be processed after step S43, which will be described later, or may be processed in parallel with step S43.
ステップS43では、演算処理部23は、ステップS41で抽出した円形物体が交通標識候補であるかを識別する。具体的には、演算処理部23は、ステップS41で抽出した円形物体の位置に相当する、画像バッファ24a内の補正後画像の画像パターンに対して、識別器24bを用いた識別処理を実行する。その結果、ステップS41で抽出した円形物体が交通標識候補か否かが認識され、また、円形物体が交通標識候補であれば、その交通標識の内容(例えば、制限速度)などが認識される。交通標識候補として識別された時の交通標識らしさを表す第一の識別スコアが、この識別処理において出力され、メモリに蓄えられる。本ステップで識別した交通標識候補を、以下では「本標識」と称する。なお、ステップS41で円形物体が抽出されなかった場合は、本ステップを省略すれば良い。
At step S43, the arithmetic processing unit 23 identifies whether the circular object extracted at step S41 is a traffic sign candidate. Specifically, the arithmetic processing unit 23 performs identification processing using the classifier 24b on the image pattern of the corrected image in the image buffer 24a, which corresponds to the position of the circular object extracted in step S41. . As a result, it is recognized whether or not the circular object extracted in step S41 is a traffic sign candidate, and if the circular object is a traffic sign candidate, the content of the traffic sign (for example, speed limit) is recognized. A first identification score representing the likelihood of a traffic sign when identified as a traffic sign candidate is output in this identification process and stored in memory. The traffic sign candidates identified in this step are hereinafter referred to as "main signs". If a circular object is not extracted in step S41, this step may be omitted.
ステップS44-45、47-4bでは、規制解除標識の候補を探索する。
In steps S44-45 and 47-4b, candidates for deregulation signs are searched.
まず、ステップS44では、演算処理部23は、画像バッファ24a内の補正後画像から、規制解除標識の候補となりうる有効直線を探索する。この有効直線探索処理は、詳細には、左右エッジ探索処理(ステップS44a)と、直線検知処理(ステップS44b)と、有効直線判定処理(ステップS44c)に分けられるため、以下では、図6から図7Cを参照しながら、各処理を順に説明する。
First, in step S44, the arithmetic processing unit 23 searches for an effective straight line that can be a candidate for a deregulation sign from the corrected image in the image buffer 24a. This effective line search processing is divided into left and right edge search processing (step S44a), line detection processing (step S44b), and effective line determination processing (step S44c). Each process will be described in order with reference to 7C.
まず、左右エッジ探索処理(ステップS44a)では、演算処理部23は、図7Aに示すように、画像バッファ24a内の補正後画像の画素を1行毎に左から右に走査する。そして、一定距離内に左エッジELと右エッジERを探索できた場合は、それらを左右エッジのペアとして抽出する。このような処理を、補正後画像の全域に施すことで、補正後画像から左右エッジのペアを抽出したエッジ画像PE2(図7Bはエッジ画像の抜粋)を生成することができる。
First, in the left and right edge search processing (step S44a), the arithmetic processing unit 23 scans the pixels of the corrected image in the image buffer 24a row by row from left to right, as shown in FIG. 7A. If the left edge E L and the right edge E R can be found within a certain distance, they are extracted as a pair of left and right edges. By applying such processing to the entire area of the post-correction image, an edge image P E2 ( FIG. 7B is an excerpt of the edge image) can be generated by extracting a pair of left and right edges from the post-correction image.
次に、直線検知処理(ステップS44b)では、演算処理部23は、ステップS44aで生成したエッジ画像PE2から直線部分を検知する。図7Bに例示するエッジ画像PE2には、左右エッジのペアが上下に2行以上連続する部分と、1行しか存在しない部分がある。本ステップでは、左右エッジのペアが上下に2行以上連続する部分を直線として検知する。
Next, in the straight line detection process (step S44b), the arithmetic processing unit 23 detects a straight line portion from the edge image PE2 generated in step S44a. The edge image P E2 illustrated in FIG. 7B has a portion in which pairs of left and right edges are continuous in two or more rows and a portion in which there is only one row. In this step, a portion in which two or more rows of pairs of left and right edges continue vertically is detected as a straight line.
有効直線判定処理(ステップS44c)では、演算処理部23は、ステップS44bで検知した直線が規制解除標識の候補となり得る有効なものかを判定し、候補となり得る有効直線を抽出する。日本国の規制解除標識は、円形標識内の太い青色斜線が目際立つデザインであり(図11(a)の上標識を参照)、諸外国においても、円形標識内の太い斜線が目立つデザインの規制解除標識が多い(例えば、ドイツ、ポーランド、スウェーデン等)。従って、本ステップでは、規制解除標識の斜線部のデザイン上の特徴を考慮して、補正後画像から規制解除標識の候補となり得る直線部を抽出する。
In the effective straight line determination process (step S44c), the arithmetic processing unit 23 determines whether the straight line detected in step S44b is a valid candidate for the deregulation sign, and extracts a valid straight line that can be a candidate. Japan's deregulation sign has a conspicuous thick blue slanted line inside a circular sign (see the upper sign in Figure 11(a)). There are many release signs (eg Germany, Poland, Sweden, etc.). Therefore, in this step, considering the design features of the hatched portion of the deregulation sign, straight portions that can be candidates for the deregulation sign are extracted from the post-correction image.
ステレオカメラ装置100を使用する場合、ステップS44bで検知した直線部までの距離を演算できるので、演算した距離に規制解除標識が存在すると仮定した場合の、補正後画像上での規制解除標識の斜線部の幅や高さも概算可能である。従って、図7Bに例示するエッジペアの幅が狭すぎる場合や広すぎる場合は、そのエッジペアにより形成された直線部を、規制解除標識の候補にはなり得ない無効直線と判定して有効直線から除外することができる。
When the stereo camera device 100 is used, the distance to the straight line detected in step S44b can be calculated. Therefore, assuming that the deregulation sign exists at the calculated distance, the oblique line of the deregulation sign on the corrected image The width and height of the part can also be approximated. Therefore, if the width of the edge pair illustrated in FIG. 7B is too narrow or too wide, the straight line formed by the edge pair is determined as an invalid straight line that cannot be a candidate for the deregulation sign, and is excluded from the valid straight lines. can do.
同様に、図7Cに例示するように、直線部の高さが、規制解除標識の斜線部の高さとして妥当と考えられる有効範囲の下限に達しない短いものであったり(図7C(b))、上限を超える長いものであったりした場合には(図7C(c))、それらの直線部を、規制解除標識の候補にはなり得ない無効直線と判定して有効直線から除外する。一方、直線部の高さが、有効範囲に収まる場合は(図7C(a))、その直線部を有効直線として抽出する。
Similarly, as exemplified in FIG. 7C, the height of the straight portion is short and does not reach the lower limit of the effective range that is considered appropriate as the height of the hatched portion of the deregulation sign (FIG. 7C (b) ), or longer than the upper limit (FIG. 7C(c)), these straight lines are determined as invalid straight lines that cannot be candidates for deregulation signs, and are excluded from valid straight lines. On the other hand, when the height of the straight portion falls within the effective range (FIG. 7C(a)), the straight portion is extracted as an effective straight line.
以上で説明したステップS44の処理により、規制解除標識の斜線部とは大きく形態の異なる電柱や支柱の画像パターンを、規制解除標識の候補から除外し、規制解除標識である可能性のある直線部の画像パターンのみを抽出することができる。
By the processing in step S44 described above, the image patterns of utility poles and supports that are significantly different in form from the shaded portion of the deregulation sign are excluded from the deregulation sign candidates, and the straight portions that may be deregulation signs are excluded. can extract only the image pattern of
次に、ステップS45では、演算処理部23は、ステップS44の処理により、有効直線が抽出されたかを判定する。そして、有効直線が抽出されなかった場合は、ステップS46に進み、有効直線が抽出された場合は、ステップS47に進む。
Next, in step S45, the arithmetic processing unit 23 determines whether an effective straight line has been extracted by the processing in step S44. If the effective straight line is not extracted, the process proceeds to step S46, and if the effective straight line is extracted, the process proceeds to step S47.
ステップS46では、演算処理部23は、ステップS43で識別した本標識(交通標識)候補を追跡する。ここで、「追跡」とは、フレームごとまたは所定の時間間隔で得られる画像ごとに取得される、ステップS41で得られるような標識候補の形状情報や位置情報、ステップS42で得られるような色情報を時系列で関連付けてメモリに記憶しておくことをいう。具体的には、ステレオカメラ装置100が撮像中の交通標識の種別を登録するためのリストである追跡リストL1に、ステップS43で識別した本標識候補の種別(例えば、速度制限標識)を登録する。なお、自車が、追跡リストL1に登録された本標識候補を通過し、その本標識候補を撮像できなくなった場合には、追跡リストL1に登録されていた本標識候補の情報は、認識済リストL2に転写されるものとする。
In step S46, the arithmetic processing unit 23 tracks the main sign (traffic sign) candidate identified in step S43. Here, “tracking” refers to the shape information and position information of the sign candidate obtained in step S41, the color information obtained in step S42, and the color It refers to storing information in memory in association with time series. Specifically, the type of the traffic sign candidate identified in step S43 (for example, a speed limit sign) is registered in the tracking list L1, which is a list for registering the type of traffic sign being imaged by the stereo camera device 100. . Note that when the own vehicle passes through the main sign candidate registered in the tracking list L1 and becomes unable to take an image of the main sign candidate, the information on the main sign candidate registered in the tracking list L1 is not recognized. It is assumed that it is transcribed to list L2.
一方、ステップS47では、演算処理部23は、ステップS44で抽出した有効直線の色情報を算出する。例えば、日本国の規制解除標識(図11(a)参照)の候補を探索する場合であれば、演算処理部23は、有効直線が収まる形状のウィンドウを画像内に配置し、このウィンドウ内の青色スコアを算出する。ここで、青色スコアは、例えば、次の方法で算出する。すなわち、画像の各画素の色が、赤色(R)、緑色(G)、青色(B)の三原色で定義される場合であれば、ウィンドウ内の青色画素の面積を、ウィンドウの全面積で除算することで、青色スコアを算出する。算出された青色スコアはメモリに記憶される。なお、ここでは青色を例にしているが、標識デザインに応じて青以外の色情報を抽出してもよく、総称してこれらの色のスコアを算出することを「色情報を取得」する、という。
On the other hand, in step S47, the arithmetic processing unit 23 calculates color information of the effective straight line extracted in step S44. For example, in the case of searching for a candidate for a deregulation sign in Japan (see FIG. 11(a)), the arithmetic processing unit 23 arranges a window having a shape that accommodates an effective straight line in the image, and Calculate the blue color score. Here, the blue score is calculated by, for example, the following method. That is, if the color of each pixel in the image is defined by the three primary colors red (R), green (G), and blue (B), the area of the blue pixel in the window is divided by the total area of the window. By doing so, the blue score is calculated. The calculated blue score is stored in memory. Although blue is used as an example here, color information other than blue may be extracted according to the sign design. It says.
上記したように、例えば、日本国の規制解除標識は青色斜線が目際立つデザインであるため(図11(a)参照)、規制解除標識を実際に含むウィンドウ内の青色スコアは高くなると考えられる。従って、ウィンドウの青色スコアの高低に着目することで、例えば、白色直線や黒色直線などの明らかに青色以外の有効直線を規制解除標識の候補から除外し、規制解除標識の候補となり得る青色系の有効直線のみを抽出することができる。
As mentioned above, for example, the deregulation sign in Japan is designed with a conspicuous blue diagonal line (see Fig. 11(a)), so the blue score in the window that actually includes the deregulation sign is considered to be high. Therefore, by focusing on the blue score of the window, for example, effective straight lines clearly other than blue, such as white straight lines and black straight lines, are excluded from candidates for deregulation signs. Only effective straight lines can be extracted.
但し、朝日や夕日に照らされた規制解除標識を撮像した場合や、曇天時や雨天時に規制解除標識を撮像した場合などには、画像内の規制解除標識の色味が弱まることがあるが、そのような規制解除標識を候補から除外しないように、本ステップで青色スコアと比較する閾値は低めに設定している。その結果、本ステップでは、実際の規制解除標識だけでなく、規制解除標識ではない有効直線(例えば支柱、電柱、木の枝など)も、規制解除標識の候補として抽出される可能性があり、本ステップ時点の規制解除標識の候補の信頼性は十分ではない。
However, when capturing an image of a deregulation sign illuminated by the morning sun or sunset, or when capturing an image of a deregulation sign in cloudy or rainy weather, the color of the deregulation sign may be weakened in the image. In order not to exclude such deregulation signs from the candidates, the threshold for comparison with the blue score is set low in this step. As a result, in this step, not only actual deregulation signs, but also effective straight lines that are not deregulation signs (for example, poles, utility poles, tree branches, etc.) may be extracted as deregulation sign candidates. Candidates for deregulation signs at this step are not reliable enough.
ステップS48では、規制解除標識候補の識別を行う。これはS42の本標識の識別と同様に行う。具体的には、演算処理部23は、識別器24bを用いて規制解除標識候補の識別を行う。このとき、規制解除標識らしさを表す第二の識別スコアが、この識別処理において出力され、メモリに蓄えられる。本ステップで識別した規制解除標識候補を、以下では「規制解除標識」と称する。なお、ステップS48は、ステップS47の前に処理しても良いし、ステップS47と並列して処理しても良い。
In step S48, candidates for deregulation signs are identified. This is performed in the same manner as the identification of the main mark in S42. Specifically, the arithmetic processing unit 23 uses the discriminator 24b to discriminate the deregulation sign candidates. At this time, a second identification score representing the likeness of the deregulation sign is output in this identification process and stored in memory. The deregulation sign candidates identified in this step are hereinafter referred to as "deregulation signs". Note that step S48 may be processed before step S47, or may be processed in parallel with step S47.
ステップS49では、演算処理部23は、ステップS42で識別した本標識と、ステップS48で抽出した規制解除標識の候補を追跡する。ここでも、「追跡」はステップS46で述べた通りの意味である。具体的には、ステレオカメラ装置100で撮像中の標識を登録するためのリストである追跡リストL1に、ステップS43で識別した本標識候補の種別(例えば、速度制限標識)と、ステップS48で抽出した規制解除標識(候補)を登録する。なお、自車が標識を通過するなどして、追跡リストL1に登録された本標識を撮像できなくなった場合には、追跡リストL1に登録されていた本標識の情報は、認識済リストL2に転写されるものとする。
In step S49, the arithmetic processing unit 23 tracks the actual sign identified in step S42 and the candidates for the deregulation sign extracted in step S48. Again, "tracking" has the same meaning as described in step S46. Specifically, in the tracking list L1, which is a list for registering the signs being imaged by the stereo camera device 100, the type of the main sign candidate identified in step S43 (for example, a speed limit sign) and the type extracted in step S48. Register the deregulation sign (candidate). Note that if the vehicle passes through a traffic sign and cannot take an image of the traffic sign registered in the tracking list L1, the information on the traffic sign registered in the tracking list L1 is transferred to the recognized list L2. shall be transcribed.
ステップS4aでは、演算処理部23は、ステップS48で抽出した規制解除標識の候補とペアになる本標識(例えば、速度制限標識、追い越し禁止)を探索する。本ステップでのペア標識の探索方法は国毎に異なる。例えば、日本国内においては、図8Aに示すように、規制解除標識の下方に本標識が配置されているため、追跡リストL1に同時期に登録されている、規制解除標識と本標識をペア標識として関連付ければ良い。一方、その他の国においては、図8Bに示すように、本標識より先に規制解除標識が設置されていることもあるため、認識済リストL2に登録された本標識と、追跡リストL1に新規登録された規制解除標識をペア標識として関連付ければ良い。何れかの探索方法を利用することにより、国に関わらず規制解除標識とペアになる本標識を探索することができる。
In step S4a, the arithmetic processing unit 23 searches for a real sign (for example, speed limit sign, no overtaking) paired with the deregulation sign candidate extracted in step S48. The method of searching for paired signs in this step differs from country to country. For example, in Japan, as shown in FIG. 8A, this sign is placed below the deregulation sign. should be associated as On the other hand, in other countries, as shown in FIG. It suffices to associate the registered deregulation signs as a pair sign. By using either search method, it is possible to search for this sign that is paired with the derestriction sign regardless of the country.
ここで、図9のフローチャートを用いて、ステップS4aでのペア標識の探索処理の詳細を説明する。このフローチャートは、ステップS4aaからステップS4adの処理と、ステップS4afからステップS4ahの処理に区分される。
Here, the details of the pair sign search process in step S4a will be described using the flowchart of FIG. This flowchart is divided into processing from step S4aa to step S4ad and processing from step S4af to step S4ah.
ステップS4aaからステップS4adでは、演算処理部23は、追跡リストL1に基づいてペア標識を探索する。これは、図8Aに例示したように、本標識と規制解除標識が同じ場所に設置されている国における、ペア標識の探索方法である。
From step S4aa to step S4ad, the arithmetic processing unit 23 searches for pair signs based on the tracking list L1. This is a search method for paired signs in countries where this sign and deregulated sign are installed at the same location, as illustrated in FIG. 8A.
まず、ステップS4abでは、演算処理部23は、ステップS49で追跡リストL1に登録した追跡情報(標識の設定位置、大きさ、種別、追跡番号等の標識を特定するための情報)に基づいて、規制解除標識とペアになる本標識が存在するか否かを確認する。例えば、図8Aのように、追跡リストL1に規制解除標識と速度制限標識が登録されていれば、これらをペア標識の候補として抽出する。
First, in step S4ab, the arithmetic processing unit 23, based on the tracking information registered in the tracking list L1 in step S49 (information for specifying the sign such as the set position, size, type, and tracking number of the sign) Check if there is a main sign paired with the deregulation sign. For example, as shown in FIG. 8A, if a deregulation sign and a speed limit sign are registered in the tracking list L1, they are extracted as pair sign candidates.
次に、ステップS4acでは、演算処理部23は、規制解除標識とペア標識候補の配置を確認し、両者の設置位置が近ければ、両者をペア標識と判定する。例えば、図10Aに示すように、規制解除標識と同じ支柱に一つ目のペア標識候補である速度制限標識が設置されており、規制解除標識から遠い支柱に二つ目のペア標識候補である追い越し禁止標識されている場合であれば、規制解除標識と同じ支柱に設置された速度制限標識をペア標識と判断し、遠方の追い越し禁止標識はペア標識ではないと判断する。なお、規制解除標識と同じ支柱に、複数の本標識が設置されている場合は、それら複数の本標識をペア標識として抽出しても良い。
Next, in step S4ac, the arithmetic processing unit 23 confirms the arrangement of the deregulation sign and the pair sign candidate, and determines that both are pair signs if their installation positions are close. For example, as shown in FIG. 10A, a speed limit sign, which is the first paired sign candidate, is installed on the same post as the deregulation sign, and a second pair sign candidate is installed on a post far from the deregulation sign. If there is an overtaking prohibition sign, the speed limit sign installed on the same pole as the deregulation sign is determined as a pair sign, and the overtaking prohibition sign in the distance is not determined as a pair sign. In addition, when a plurality of permanent signs are installed on the same post as the deregulation sign, the plurality of permanent signs may be extracted as a pair of signs.
ステップS4adでは、演算処理部23は、ステップ48cで抽出したペア標識に関する情報を記憶部24に記憶する。
In step S4ad, the arithmetic processing unit 23 stores information on the paired signs extracted in step 48c in the storage unit 24.
ステップS4aeでは、演算処理部23は、ペア標識が登録されたかを確認する。そして、ペア標識が登録された場合は、ステップS4aの処理を終了しステップS4bに進む。一方、ペア標識が未登録の場合は、ステップS4afに進む。
At step S4ae, the arithmetic processing unit 23 confirms whether the pair indicator has been registered. Then, when the pair indicator is registered, the process of step S4a is terminated and the process proceeds to step S4b. On the other hand, if the pair indicator is unregistered, the process proceeds to step S4af.
ステップS4afからステップS4ahでは、演算処理部23は、追跡リストL1と認識済リストL2に基づいてペア標識を探索する。これは、図8Bに例示したように、本標識と規制解除標識が離れた場所に設置されている国における、ペア標識の探索方法である。
From step S4af to step S4ah, the arithmetic processing unit 23 searches for pair signs based on the tracking list L1 and the recognized list L2. As illustrated in FIG. 8B, this is a search method for paired signs in countries where the main sign and deregulation sign are installed in separate locations.
まず、ステップS4agでは、演算処理部23は、認識済リストL2に登録した認識済情報(標識の設定位置、大きさ、種別、登録番号等の過去に認識した標識情報)に基づいて、追跡リストL1に登録された規制解除標識とペアになる本標識が存在するか否か確認する。例えば、図8Bのように、認識済リストL2に過去に撮像した速度制限標識が登録されていれば、これをペア標識の候補として抽出する。認識済リストL2には過去に認識した標識情報が多数登録されている可能性があるため、本ステップでは、直近に登録した本標識をペア標識の候補と判断する。例えば、図10Bに示すように、規制解除標識の手前の支柱に一つ目のペア標識候補である速度制限標識が設置されており、速度制限標識の手前の支柱に二つ目のペア標識候補である追い越し禁止標識が設置されている場合であれば、直近に登録された速度制限標識をペア標識と判断し、それ以前に登録された追い越し禁止標識はペア標識ではないと判断する。
First, in step S4ag, the arithmetic processing unit 23, based on the recognized information registered in the recognized list L2 (previously recognized sign information such as the setting position, size, type, and registration number of the sign), It is checked whether or not there is a regular sign paired with the deregulation sign registered in L1. For example, as shown in FIG. 8B, if a previously imaged speed limit sign is registered in the recognized list L2, it is extracted as a pair sign candidate. Since there is a possibility that a large number of previously recognized sign information are registered in the recognized list L2, in this step, the most recently registered main sign is determined as a pair sign candidate. For example, as shown in FIG. 10B, a speed limit sign, which is the first pair of candidate signs, is installed on the pole in front of the deregulation sign, and the second pair of sign candidates is installed on the pole in front of the speed limit sign. If a no overtaking sign is installed, the speed limit sign registered most recently is determined to be a paired sign, and the previously registered no overtaking sign is determined not to be a paired sign.
なお、図9では、図8Aの環境に対応したステップS4aaからステップS4adの処理と、図8Bの環境に対応したステップS4afからステップS4ahの処理の双方を実行したが、GPSから取得した位置情報等を利用して自車が走行中の国を特定できる場合は、その国の交通法規に応じて、ステップS4aaからステップS4adの処理か、ステップS4afからステップS4ahの処理の一方のみを選択して実施する構成としても良い。
In FIG. 9, both the processing from step S4aa to step S4ad corresponding to the environment in FIG. 8A and the processing from step S4af to step S4ah corresponding to the environment in FIG. 8B are executed. If the country in which the vehicle is traveling can be identified using It is good also as a structure to carry out.
図9に示したステップS4aの処理を終えると、規制解除標識の設置態様(図8A、図8B)によらず、規制解除標識のペアとなる本標識を抽出できる。但し、ステップS45では、上記したように、実際には規制解除標識でない有効直線を候補として抽出している可能性もあるので、以降の処理では、ステップS4aで抽出したペア標識(つまりペアとなる本標識)の色情報も参照して、ステップS48で抽出した候補が真に規制解除標識であるかを確認する。当該本標識の色情報はステップS48)にて取得され、メモリに記憶されている。
After completing the process of step S4a shown in FIG. 9, the main sign that is paired with the deregulation sign can be extracted regardless of the installation mode of the deregulation sign (FIGS. 8A and 8B). However, in step S45, as described above, there is a possibility that an effective straight line that is not actually a deregulation sign is extracted as a candidate. By also referring to the color information of the actual sign), it is confirmed whether or not the candidate extracted in step S48 is truly a deregulation sign. The color information of the main sign is acquired in step S48) and stored in the memory.
ステップS4bでは、演算処理部23は、規制解除標識の候補の真偽判定処理(ステップS4c)を実施可能な時間帯かを判定する。後述するように、ステップS4cでは、ペア標識の色情報を使用するが、ペア標識の色情報は、昼間の撮像画像に基づくものであれば信頼度が高く、夜間の撮像画像に基づくものであれば信頼度が低いものである。そこで、本ステップでは、例えば、ECUの内蔵タイマーやGPSから取得した時刻情報に基づいて、昼夜の別を判定し、昼間であれば信頼度の高い色情報を取得可能な環境と判断して、ステップS4cに進み、夜間であれば信頼度の高い色情報の取得が困難な環境と判断して、ステップS4dに進む。なお、ステップS43で本標識候補が抽出されなかった場合は、ステップS4cの処理を実行することができないし、また、ステップS48で規制解除標識の候補が抽出されなかった場合は、候補の真偽を判定するステップS4cの処理がそもそも不要なので、これらの場合も、ステップS4cを回避してステップS4cに進む。
In step S4b, the arithmetic processing unit 23 determines whether it is a time zone in which it is possible to perform the authenticity determination process (step S4c) of the candidate deregulation sign. As will be described later, in step S4c, the color information of the paired signs is used. reliability is low. Therefore, in this step, for example, based on the time information acquired from the built-in timer of the ECU or the GPS, it is determined whether it is daytime or nighttime. Proceeding to step S4c, it is determined that it is difficult to obtain highly reliable color information at night, and the process proceeds to step S4d. It should be noted that if the candidate for this sign is not extracted in step S43, the process of step S4c cannot be executed, and if the candidate for the deregulation sign is not extracted in step S48, the truth of the candidate is determined. Since the process of step S4c for determining is not necessary in the first place, step S4c is avoided in these cases as well, and the process proceeds to step S4c.
ステップS4cでは、演算処理部23は、ステップS48で抽出した規制解除標識の候補の真偽を判断し、その候補が偽の規制解除標識であれば(例えば、支柱であったなら)、その候補を棄却する。上記したように、ステップS48では、有効直線の青色スコアに基づいて規制解除標識の候補を抽出するが、有色光で照らされた規制解除標識や、曇天や雨天時の規制解除標識を候補から除外しないよう、有効直線の青色スコアと比較する閾値を低めに設定しており、実際には規制解除標識ではない有効直線が候補として抽出されることを許容していた。そこで、本ステップでは、ステップS42で取得した本標識の色スコアを参照して、規制解除標識の候補の真偽判定に利用する閾値を調整することで、外部環境によらず、規制解除標識の候補の真偽を正確に判定できるようにした。
In step S4c, the arithmetic processing unit 23 determines whether the candidates for deregulation signs extracted in step S48 are true or false. reject. As described above, in step S48, candidates for deregulation signs are extracted based on the blue score of the effective straight line, but deregulation signs illuminated with colored light and deregulation signs in cloudy or rainy weather are excluded from the candidates. In order to avoid this, the threshold for comparison with the blue score of the effective straight line was set to be low, which allowed the extraction of effective straight lines that were not actual deregulation signs as candidates. Therefore, in this step, by referring to the color score of the main sign acquired in step S42 and adjusting the threshold value used to determine the authenticity of the candidates for the deregulation sign, the deregulation sign It is now possible to accurately judge the authenticity of a candidate.
例えば、図11のように、本標識が、標識外周の赤色環の速度制限標識であれば、その赤色環が収まる形状のウィンドウ内の赤色スコアは高い値になると考えられる。しかしながら、この本標識が有色光に照らされている場合、曇天や雨天の場合、晴天の場合、等であれば、本標識(速度制限標識)の赤色スコアが変化すると考えられる。また、その場合、規制解除標識の青色スコアも同程度に変化すると考えられる。
For example, as shown in Fig. 11, if this sign is a speed limit sign with a red ring around the sign, the red score within the window shaped to accommodate the red ring is considered to be a high value. However, if this sign is illuminated by colored light, cloudy or rainy weather, clear weather, etc., the red score of this sign (speed limit sign) will change. In that case, the blue score of the deregulation sign would also change to the same extent.
従って、本ステップでは、本標識(速度制限標識)の赤色スコアが低い場合には、規制解除標識の候補の真偽を判定するための青色スコアの閾値も低く設定し、反対に、赤色スコアが高い場合には、青色スコアの閾値も高く設定する。つまり、本標識の認識時に取得された色情報(例えば青色スコア(つまり青色である信頼度))を用いて、規制解除標識として認識するための色情報の閾値を決定する。また、本標識の認識時に取得された色情報自体ではなく、本標識のテンプレート画像の色情報に対する本標識の認識時に取得された色情報の変化率に基づいて、規制解除標識として認識するための色情報の閾値を決定してもよい。本標識の色情報、および本標識の色情報の変化率を合わせて、本標識の「信頼度」と称する。
Therefore, in this step, when the red score of this sign (speed limit sign) is low, the threshold of the blue score for judging the authenticity of the candidate of the deregulation sign is also set low. If it is high, the blue score threshold is also set high. In other words, the color information (for example, blue score (that is, confidence in being blue)) acquired when recognizing this sign is used to determine the threshold of color information for recognition as a deregulation sign. In addition, not the color information itself acquired when recognizing this sign, but the rate of change in color information acquired when recognizing this sign against the color information of the template image of this sign, to recognize it as a deregulation sign A threshold for color information may be determined. Together, the color information of the sign and the rate of change of the color information of the sign are referred to as the "reliability" of the sign.
このように調整した色情報(例えば青色スコア)の閾値を利用することで、環境によらず、規制解除標識の候補の真偽を正確に判定することができる。
By using the threshold value of color information (for example, blue score) adjusted in this way, it is possible to accurately determine the authenticity of candidates for deregulation signs regardless of the environment.
最後に、ステップS4dでは、演算処理部23は、上記の追跡リストL1または認識済リストL2を参照して、標識を識別した結果を確定する。すなわち、ステップS45からステップS46に進んだ場合であれば、ステップS43で識別した本標識候補のみを標識として識別結果として確定する。一方、ステップS45からステップS47に進んだ場合であれば、例えば、ステップS43で識別した本標識候補を、ステップS48、S4cで抽出した規制解除標識候補のペア標識として識別する。そして、本ステップで、ペア標識が認識された場合には、続くステップS5では、本標識が示す規制をキャンセルするような制御方針が決定される。ここで、「確定する」とは、認識処理結果として、認識された標識から得られる情報を後段の制御処理(図2のS5)に出力することを意味する。
Finally, in step S4d, the arithmetic processing unit 23 refers to the tracking list L1 or the recognized list L2 to determine the result of identifying the sign. That is, if the process proceeds from step S45 to step S46, only the main marker candidate identified in step S43 is determined as the identification result as a marker. On the other hand, if the process proceeds from step S45 to step S47, for example, the main sign candidate identified in step S43 is identified as a pair sign of the deregulation sign candidate extracted in steps S48 and S4c. Then, when the pair sign is recognized in this step, a control policy that cancels the regulation indicated by this sign is determined in the subsequent step S5. Here, "determine" means outputting the information obtained from the recognized sign to the subsequent control process (S5 in FIG. 2) as the result of the recognition process.
以上で説明した本実施例のステレオカメラ装置100によれば、図11(a)に示す規制解除標識を正しく認識できるだけでなく、図11(b)に示す、規制解除標識の斜線部と角度や太さが類似している支柱の上端部を規制解除標識と誤検知することを抑制できるので、標識検知の信頼性を向上させることができる。
According to the stereo camera device 100 of the present embodiment described above, not only can the deregulation sign shown in FIG. Since it is possible to suppress erroneous detection of the upper ends of the posts having similar thicknesses as the deregulation sign, it is possible to improve the reliability of sign detection.
なお、前述した実施例では、2つのカメラから構成されるステレオカメラ装置100を用いて説明したが、カメラは1台でもよいし、3台以上使用してもよい。また、上記の各構成、機能、処理部、処理手段等は、それらの一部又は全部を、例えば集積回路で設計する等によりハードウェアで実現してもよい。また、上記の各構成、機能等は、プロセッサがそれぞれの機能を実現するプログラムを解釈し、実行することによりソフトウェアで実現してもよい。各機能を実現するプログラム、テーブル、ファイル等の情報は、メモリや、ハードディスク、SSD(Solid State Drive)等の記憶装置、または、ICカード、SDカード、DVD等の記録媒体に置くことができる。
In the above-described embodiment, the stereo camera device 100 composed of two cameras was used, but one camera or three or more cameras may be used. Further, each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.
100…ステレオカメラ装置、1…カメラ、1L…左カメラ、1R…右カメラ、2…標識認識装置、21…画像入力インタフェース、22…画像処理部、23…演算処理部、24…記憶部、24a…画像バッファ、24b…識別器、25… CANインタフェース、26…監視処理部、27…内部バス、CAN 車載ネットワーク
DESCRIPTION OF SYMBOLS 100... Stereo camera apparatus, 1... Camera, 1L... Left camera, 1R... Right camera, 2... Sign recognition apparatus, 21... Image input interface, 22... Image processing part, 23... Operation process part, 24... Storage part, 24a ... image buffer, 24b ... discriminator, 25 ... CAN interface, 26 ... monitoring processing unit, 27 ... internal bus, CAN in-vehicle network
Claims (12)
- 画像から所定の種別の標識の候補を認識する標識候補認識部と、
前記所定の種別の標識の候補として、第1標識の候補が認識された場合、当該第1標識と、当該第1標識に対応して設置されている第2標識を関連付けるペア標識探索部と、
第1標識に対応して設置されている第2標識が存在する場合、前記第1標識に関連付けた前記第2標識の色情報を抽出する色情報抽出部と、
該色情報抽出部で抽出した前記第2標識の色情報の信頼度に基づいて、前記第1標識の候補の色情報を抽出する際の閾値を決定する閾値決定部と、
を有することを特徴とする標識認識装置。 a sign candidate recognition unit that recognizes sign candidates of a predetermined type from an image;
a paired sign searching unit that, when a first sign candidate is recognized as a sign candidate of the predetermined type, associates the first sign with a second sign installed corresponding to the first sign;
a color information extraction unit for extracting color information of the second sign associated with the first sign when there is a second sign installed corresponding to the first sign;
a threshold determination unit configured to determine a threshold for extracting the color information of the first marker candidate based on the reliability of the color information of the second marker extracted by the color information extraction unit;
A sign recognition device comprising: - 請求項1に記載の標識認識装置において、
前記ペア標識探索部で、前記第1標識に対応して設置されている前記第2標識が存在するか否か判断し、前記第1標識に対して前記第2標識が関連付けられなかった場合には、前記閾値とは異なる閾値を用いることを特徴とする標識認識装置。 The sign recognition device according to claim 1,
The paired sign searching unit determines whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, uses a threshold value different from the threshold value. - 請求項1に記載の標識認識装置において、
さらに、前記標識候補認識部の認識結果を時系列に記憶する標識候補記憶部を備え、
前記ペア標識探索部で、前記第1標識に対応して設置されている前記第2標識が存在するか否か判断し、前記第1標識に対して前記第2標識が関連付けなかった場合には、前記標識候補記憶部に記憶された認識結果の中から、前記第1標識に関連付ける第2標識を抽出することを特徴とする標識認識装置。 The sign recognition device according to claim 1,
further comprising a sign candidate storage unit that stores the recognition results of the sign candidate recognition unit in time series,
The paired sign searching unit determines whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, and extracting a second sign associated with the first sign from recognition results stored in the sign candidate storage unit. - 請求項1に記載の標識認識装置において、
追跡リストに複数の第1標識の情報が登録されている場合は、第2標識に最も近い第1標識をペア標識として関連付けることを特徴とする標識認識装置。 The sign recognition device according to claim 1,
A sign recognition device, characterized in that when information on a plurality of first signs is registered in a tracking list, a first sign closest to a second sign is associated as a pair sign. - 請求項1に記載の標識認識装置において、
車両が前記第1標識を通過したときに、追跡リストに登録された第1標識の情報を認識済リストに転写し、
前記追跡リストに登録された第2標識の候補と前記認識済リストに登録された第1標識をペア標識として関連付けることを特徴とする標識認識装置。 The sign recognition device according to claim 1,
transferring the information of the first sign registered in the tracking list to the recognized list when the vehicle passes the first sign;
A sign recognition device, wherein a second sign candidate registered in the tracking list and a first sign registered in the recognized list are associated as a pair sign. - 請求項5に記載の標識認識装置において、
前記認識済リストに複数の第1標識の情報が登録されている場合は、直近に登録された第1標識をペア標識として関連付けることを特徴とする標識認識装置。 In the sign recognition device according to claim 5,
A sign recognition device, wherein when information on a plurality of first signs is registered in the recognized list, the most recently registered first sign is associated as a pair sign. - 車載カメラが撮像した画像から所定の種別の標識の候補を認識するステップと、
前記所定の種別の標識の候補として、第1標識の候補が認識された場合、当該第1標識と、当該第1標識に対応して設置されている第2標識を関連付けるステップと、
第1標識に対応して設置されている第2標識が存在する場合、前記第1標識に関連付けた前記第2標識の色情報を抽出するステップと、
抽出した前記第2標識の色情報の信頼度に基づいて、前記第1標識の候補の色情報を抽出する際の閾値を決定するステップと、
を有することを特徴とする標識認識方法。 a step of recognizing candidate signs of a predetermined type from an image captured by an in-vehicle camera;
when a first sign candidate is recognized as a sign candidate of the predetermined type, associating the first sign with a second sign installed corresponding to the first sign;
if there is a second sign installed corresponding to the first sign, extracting color information of the second sign associated with the first sign;
determining a threshold for extracting the color information of the candidate for the first marker based on the reliability of the extracted color information for the second marker;
A sign recognition method, comprising: - 請求項7に記載の標識認識方法において、
前記第1標識に対応して設置されている前記第2標識が存在するか否か判断し、前記第1標識に対して前記第2標識が関連付けられなかった場合には、前記閾値とは異なる閾値を用いることを特徴とする標識認識方法。 In the sign recognition method according to claim 7,
It is determined whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, it is different from the threshold A sign recognition method characterized by using a threshold. - 請求項7に記載の標識認識方法において、
さらに、認識結果を時系列に記憶するステップを備え、
前記第1標識に対応して設置されている前記第2標識が存在するか否か判断し、前記第1標識に対して前記第2標識が関連付けなかった場合には、記憶された認識結果の中から、前記第1標識に関連付ける第2標識を抽出することを特徴とする標識認識方法。 In the sign recognition method according to claim 7,
further comprising a step of storing recognition results in chronological order;
It is determined whether or not the second sign installed corresponding to the first sign exists, and if the second sign is not associated with the first sign, the stored recognition result is determined. A method of recognizing a sign, comprising extracting a second sign associated with the first sign from among them. - 請求項7に記載の標識認識方法において、
追跡リストに複数の第1標識の情報が登録されている場合は、第2標識に最も近い第1標識をペア標識として関連付けることを特徴とする標識認識方法。 In the sign recognition method according to claim 7,
A sign recognition method characterized by associating a first sign closest to a second sign as a pair sign when information on a plurality of first signs is registered in a tracking list. - 請求項7に記載の標識認識方法において、
車両が前記第1標識を通過したときに、追跡リストに登録された第1標識の情報を認識済リストに転写するステップと、
追跡リストに登録された第2標識の候補と前記認識済リストに登録された第1標識をペア標識として関連付けるステップと、
を有することを特徴とする標識認識方法。 In the sign recognition method according to claim 7,
transferring the information of the first sign registered in the tracking list to the recognized list when the vehicle passes the first sign;
a step of associating a second marker candidate registered in a tracking list and a first marker registered in the recognized list as a paired marker;
A sign recognition method, comprising: - 請求項11に記載の標識認識方法において、
前記認識済リストに複数の第1標識の情報が登録されている場合は、直近に登録された第1標識をペア標識として関連付けることを特徴とする標識認識方法。 The sign recognition method according to claim 11,
A sign recognition method, wherein when information on a plurality of first signs is registered in the recognized list, the most recently registered first sign is associated as a pair sign.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE112022002739.8T DE112022002739T5 (en) | 2021-08-18 | 2022-02-07 | SIGN RECOGNITION DEVICE AND SIGN RECOGNITION METHOD |
JP2023542187A JPWO2023021726A1 (en) | 2021-08-18 | 2022-02-07 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2021-133360 | 2021-08-18 | ||
JP2021133360 | 2021-08-18 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023021726A1 true WO2023021726A1 (en) | 2023-02-23 |
Family
ID=85240273
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2022/004760 WO2023021726A1 (en) | 2021-08-18 | 2022-02-07 | Sign recognition device and sign recognition method |
Country Status (3)
Country | Link |
---|---|
JP (1) | JPWO2023021726A1 (en) |
DE (1) | DE112022002739T5 (en) |
WO (1) | WO2023021726A1 (en) |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015191619A (en) * | 2014-03-28 | 2015-11-02 | 富士重工業株式会社 | Outside-vehicle environment recognition device |
JP2017146711A (en) * | 2016-02-16 | 2017-08-24 | 株式会社日立製作所 | Image processing device, warning apparatus, image processing system, and image processing method |
JP2021056575A (en) * | 2019-09-27 | 2021-04-08 | スズキ株式会社 | Vehicle drive support system |
-
2022
- 2022-02-07 JP JP2023542187A patent/JPWO2023021726A1/ja active Pending
- 2022-02-07 DE DE112022002739.8T patent/DE112022002739T5/en active Pending
- 2022-02-07 WO PCT/JP2022/004760 patent/WO2023021726A1/en active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015191619A (en) * | 2014-03-28 | 2015-11-02 | 富士重工業株式会社 | Outside-vehicle environment recognition device |
JP2017146711A (en) * | 2016-02-16 | 2017-08-24 | 株式会社日立製作所 | Image processing device, warning apparatus, image processing system, and image processing method |
JP2021056575A (en) * | 2019-09-27 | 2021-04-08 | スズキ株式会社 | Vehicle drive support system |
Non-Patent Citations (1)
Title |
---|
KAGEYAMA, YOICHI; NISHIDA, MAKOTO: "Extraction of Circular Road Sign Considering Scene Image Features, Utilization of Brightness Information and Automatic Setting of Threshold", IMAGE LAB, vol. 22, no. 5, 10 May 2011 (2011-05-10), JP , pages 8 - 14, XP009543528, ISSN: 0915-6755 * |
Also Published As
Publication number | Publication date |
---|---|
DE112022002739T5 (en) | 2024-04-11 |
JPWO2023021726A1 (en) | 2023-02-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10262216B2 (en) | Hazard detection from a camera in a scene with moving shadows | |
CN106647776B (en) | Method and device for judging lane changing trend of vehicle and computer storage medium | |
CN110197589B (en) | Deep learning-based red light violation detection method | |
WO2020000251A1 (en) | Method for identifying video involving violation at intersection based on coordinated relay of video cameras | |
US9558412B2 (en) | Vehicle exterior environment recognition device | |
CN106169244A (en) | The guidance information utilizing crossing recognition result provides device and method | |
US20150161796A1 (en) | Method and device for recognizing pedestrian and vehicle supporting the same | |
US20050102070A1 (en) | Vehicle image processing device | |
US20140002656A1 (en) | Lane departure warning system and lane departure warning method | |
CN104036279A (en) | Intelligent vehicle running control method and system | |
CN105426864A (en) | Multiple lane line detecting method based on isometric peripheral point matching | |
US11373417B2 (en) | Section line recognition device | |
JP2007179386A (en) | Method and apparatus for recognizing white line | |
CN111222441A (en) | Point cloud target detection and blind area target detection method and system based on vehicle-road cooperation | |
CN108944926B (en) | Vehicle exterior environment recognition device | |
JP2020077293A (en) | Lane line detection device and lane line detection method | |
US11679769B2 (en) | Traffic signal recognition method and traffic signal recognition device | |
WO2023021726A1 (en) | Sign recognition device and sign recognition method | |
CN112270258A (en) | Violation information acquisition method and device for non-motor vehicle | |
EP3287940A1 (en) | Intersection detection system for a vehicle | |
KR20210002893A (en) | License plate recognition method using hybrid approach and system therefore | |
CN115565363A (en) | Signal recognition device | |
JP7058753B2 (en) | Camera device | |
CN112334944B (en) | Mark recognition method and mark recognition device for camera device | |
KR101622051B1 (en) | Distinguishing system and method for vehicle |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22858051 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2023542187 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 112022002739 Country of ref document: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 22858051 Country of ref document: EP Kind code of ref document: A1 |