WO2022230283A1 - Sign recognition device and sign recognition method - Google Patents

Sign recognition device and sign recognition method Download PDF

Info

Publication number
WO2022230283A1
WO2022230283A1 PCT/JP2022/004107 JP2022004107W WO2022230283A1 WO 2022230283 A1 WO2022230283 A1 WO 2022230283A1 JP 2022004107 W JP2022004107 W JP 2022004107W WO 2022230283 A1 WO2022230283 A1 WO 2022230283A1
Authority
WO
WIPO (PCT)
Prior art keywords
sign
candidate
road
unit
vehicle
Prior art date
Application number
PCT/JP2022/004107
Other languages
French (fr)
Japanese (ja)
Inventor
雄飛 椎名
ユイビン ツーン
健 永崎
Original Assignee
日立Astemo株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立Astemo株式会社 filed Critical 日立Astemo株式会社
Priority to DE112022000874.1T priority Critical patent/DE112022000874T5/en
Priority to JP2023517058A priority patent/JP7564946B2/en
Publication of WO2022230283A1 publication Critical patent/WO2022230283A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/09623—Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle

Definitions

  • the present invention relates to a sign recognition device that recognizes road signs based on captured images in front of a vehicle.
  • An in-vehicle camera device is a device that extracts visual information from captured images and at the same time measures distance information to objects in the captured images. and the position and shape of white lines, road surface conditions, types of road signs, etc.) in detail, contributing to improved safety when assisting driving.
  • Road signs are one of the objects recognized by the in-vehicle camera device.
  • the sign recognition function is used to accelerate or decelerate a vehicle that performs automatic driving in cooperation with map information.
  • Euro NCAP (updated from 2016 to 2023), which is an evaluation index for advanced driver assistance systems, also has evaluation items for speed assistance systems (SAS: Speed Assistance Systems), and its importance is increasing.
  • SAS Speed Assistance Systems
  • ISA Intelligent Speed Adaptation
  • the speed limit display device of Patent Document 1 As a device for recognizing the speed limit from road signs, the speed limit display device of Patent Document 1 is known.
  • the speed limit displayed on the traffic sign in the image is detected based on the image data acquired by the image acquisition device, which is equipped with a display that displays the speed limit.
  • the detected speed limit is displayed on the display. If the distance from the vehicle is large, the detected speed limit is not displayed on the display.
  • the speed limit display device of Patent Document 1 does not work when a plurality of speed limit signs with a lateral distance between the vehicle and the traffic sign being less than a threshold value are installed (for example, on an expressway, speed limit signs for large vehicles (80) and the speed limit sign (100) for small vehicles are installed adjacently), if the detection timing of each speed limit sign is off, the speed limit information for the own vehicle There is a possibility that the speed limit information that is not displayed may be displayed for a short period of time or for a certain period of time after passing the sign.
  • the present invention has been made to solve the above-mentioned problems, and when a plurality of speed limit signs are installed side by side, the speed limit information for the own vehicle can be obtained by recognizing the side signs at the same timing. It is an object of the present invention to provide a sign recognition device capable of outputting without omission.
  • the sign recognition device of the present invention includes a sign detection unit that detects a sign candidate from an image captured by a camera device mounted on a vehicle, and a sign identification unit that identifies the type of the sign candidate.
  • a sign tracking unit that tracks identified sign candidates in a plurality of captured images in association with each other; and when the travel direction distance to the sign candidate being tracked is within a predetermined distance, the sign candidate corresponds to a road sign.
  • a sign determination unit that determines whether to If the sign candidate has already been determined to be a road sign, it is determined whether the second sign candidate corresponds to the road sign even if the second sign candidate is not within the predetermined distance.
  • the sign recognition device of the present invention when multiple speed limit signs are installed side by side, the side signs can be recognized at the same timing and the speed limit information for the own vehicle can be output without omission.
  • FIG. 1 Schematic diagram showing the hardware configuration of the sign recognition device of one embodiment.
  • FIG. 1 is a functional block diagram of a camera device according to one embodiment;
  • FIG. FIG. 3 is a flow chart showing processing contents of sign recognition in FIG. 2 ;
  • FIG. A captured image of scene 1.
  • FIG. 4B is a top view corresponding to FIG. 4A;
  • FIG. 5B is a top view corresponding to FIG. 5A;
  • a sign recognition device 100 mounted on a vehicle V will be described below with reference to the drawings.
  • the sign recognition device 100 is an in-vehicle device that performs image processing on a shooting target area in front of the vehicle V to recognize the environment outside the vehicle. It has the function of recognizing lighting lamps, etc., and also has the function of judging the authenticity of road signs.
  • the sign recognition device 100 may have a function of adjusting the brakes and steering of the vehicle V, but this function may be omitted.
  • FIG. 1 is a block diagram showing a schematic configuration of the sign recognition device 100.
  • the sign recognition device 100 is a device in which the camera device 10 and the output device 20 are connected via a CAN (Controller Area Network). It may be connected to an ECU (Electronic Control Unit), etc.) that constitutes a system or an automatic driving system. Each unit will be described below in order.
  • ECU Electronic Control Unit
  • the camera device 10 includes a camera 11 (a left camera 11L and a right camera 11R), an image input interface 12, an image processing section 13, an arithmetic processing section 14, a storage section 15, a CAN interface 16, and a control processing section 17.
  • the image input interface 12 to the control processing unit 17 other than the camera 11 are interconnected via an internal bus, and can be configured by a single or a plurality of computer units.
  • FIG. 1 exemplifies a case where these are configured by a single computer unit.
  • the left camera 11L and the right camera 11R are arranged on the left and right with a predetermined interval in order to stereoscopically capture an area in front of the vehicle V.
  • the imaging element of the left camera 11L captures the left image PL, A right image PR is captured by the imaging device.
  • the image input interface 12 controls the camera 11 and takes in captured images P (left image P L , right image P R ). An image P captured through the image input interface 12 is sent to the image processing section 13, the arithmetic processing section 14, etc. through the internal bus.
  • the image processing unit 13 compares the left image P L and the right image P R , corrects device-specific deviations caused by the image sensor and performs image correction such as noise interpolation on each image, and stores the images in the storage unit. 15. Furthermore, between the left image P L and the right image P R , mutually corresponding parts are calculated and parallax information (the difference in the direction seen when the same point is viewed from two different points, that is, the difference between the two directions) is calculated. angle information) is calculated and stored in the storage unit 15 .
  • the arithmetic processing unit 14 is an arithmetic unit such as a CPU that performs various kinds of arithmetic processing.
  • the image P that the sign recognition unit 1 uses as a basis for road sign recognition may be either the left image P L or the right image P R , or both images may be used.
  • the method for recognizing road signs here is to compare the shape recognized by image processing with, for example, a database of road signs stored in the storage unit 15, and match the shape with any one of the road signs above a certain degree. is recognized as a road sign, and if there are multiple road signs, the sign information to be observed by the own vehicle is output. This processing will be described in detail later.
  • the storage unit 15 is, for example, a semiconductor memory, and stores information and the like obtained by the image processing unit 13 and the arithmetic processing unit 14.
  • the CAN interface 16 is an input/output unit for the CAN, and the calculation information of the camera device 10 is output to the CAN via the CAN interface 16 and output to the control system (ECU, etc.) of the vehicle V via the CAN. . Specifically, the CAN interface 16 outputs a sign recognition result to the output device 20 when the sign recognition unit 1 acquires sign information to be observed by the vehicle V via the CAN.
  • the control processing unit 17 has the role of preventing abnormal operations by monitoring whether each processing unit is operating abnormally or whether an error has occurred during data transfer. Fulfill.
  • ⁇ Output device 20> The road sign judgment results and the like calculated by the camera device 10 are transmitted to the output device 20 via the CAN.
  • the output device 20 is, for example, a liquid crystal monitor (see FIGS. 4B and 5B), a lamp, a buzzer, a speaker, etc. mounted in the driver's cab of the vehicle V for display output and audio output. Based on the received information, the driver is visually or audibly notified of road signs. As for the method of notification, it is possible to simply notify the presence or absence of a road sign, or it is also possible to notify the type of road sign as well. Also, it is possible to adopt a configuration in which at least one of the visual output mode and the auditory output mode can be selected.
  • the output device 20 is a safe driving support system or an automatic driving system and controls the brakes and steering of the vehicle V
  • information such as road sign information calculated by the camera device 10 is added to the basic information for control by the system. be done.
  • FIG. 2 schematically shows the functions of the camera device 10 in FIG. 1.
  • Each part in the sign recognition part 1 in FIG. corresponds to the function realized by executing it.
  • the left camera 11L and the right camera 11R capture an image in front of the vehicle V to obtain a left image PL and a right image PR .
  • the sign recognition unit 1 performs sign recognition processing based on the monocular image (either the left image PL or the right image PR) captured by the camera 11 . Each processing performed by the sign recognition unit 1 will be sequentially described below.
  • the sign detection unit 1a detects sign candidates from the image P.
  • the outer shape of the road sign is circular. Therefore, the sign detection unit 1a selects a shape in the image P that is estimated to have a certain degree of coincidence with a perfect circle by image processing as the sign candidate 30. Extract.
  • the sign identification unit 1 b identifies the type of the sign candidate 30 .
  • the sign candidate 30 detected by the sign detection unit 1a is collated with, for example, a sign database (recognition dictionary) stored in the storage unit 15, and the type of the sign candidate 30 (which road sign candidate the sign candidate 30 is a candidate for) is determined. ).
  • a sign database recognition dictionary
  • the marker tracking unit 1c associates a plurality of images P captured continuously. Specifically, the type-identified sign candidates 30 whose relative positions with respect to the vehicle V are displaced as the vehicle V travels or the like are associated with each other by image processing of the images P of a plurality of continuous frames and tracked. Then, all tracked candidate signs 30 are registered in the tracking list L1.
  • the sign determination unit 1d makes a comprehensive judgment on the sign candidate 30 in a plurality of frames. Specifically, image processing is performed on images P of a plurality of frames, and it is determined whether or not the sign candidate 30 being tracked is a genuine road sign. Then, the sign candidate 30 determined to be a genuine road sign is registered in the notification list L2. Note that this processing is performed for all the sign candidates 30 registered in the tracking list L1, so when a plurality of road signs are placed side by side, a plurality of road signs are registered in the notification list L2. The details of this sign determination process will be described with reference to FIG.
  • the sign selection unit 1e selects road signs related to the own vehicle. Specifically, it determines whether or not the road sign determined by the sign determination unit 1d is a sign that the vehicle should follow, and generates a sign recognition result R in a form suitable for the application to be applied. For example, in the case of an application that displays the speed limit information for the vehicle on the LCD monitor, if the speed limit signs for large and small vehicles are adjacent to each other, if the vehicle is a small vehicle, at least the small vehicle Elect the sign recognition result R to contain speed limit information for .
  • the sign recognition result R obtained by the sign recognition unit 1 is saved in the storage unit 15.
  • the sign recognition result R by the sign recognition unit 1 is stored in the storage unit 15 and simultaneously output to another computer unit or the output device 20 via CAN. Therefore, in this embodiment, the information on the signs to be observed by the own vehicle selected by the sign selection unit 1e is output to the output device 20, and the occupants of the vehicle V are notified of the information.
  • step S1 the sign determination unit 1d confirms whether or not a determination completion flag is set for the target sign candidate 30 at present. If the determination completion flag is set, it has already been determined whether the target sign candidate 30 is a road sign. On the other hand, if the determination completion flag is not set, the process proceeds to step S2.
  • step S ⁇ b>2 the sign determination unit 1 d determines whether or not there is a possibility that the current target sign candidate 30 is placed side by side with another sign candidate 30 .
  • the conditions for judging the possibility of co-locating referred to here are, for example, (1) that the number of time-series tracking results is two or more, (2) that the current road is an expressway, and (3) a traffic sign.
  • the candidate 30 is an electronic sign, and (4) the candidate 30 is a 50 kph to 120 kph speed limit sign. If there is a possibility of co-locating, the process proceeds to step S3, and if there is no possibility of co-locating, the process proceeds to step S4.
  • step S2 may be changed as appropriate according to the road conditions of the country or region in which they are applied.
  • the determination of the road on which the vehicle is traveling may be obtained from external information such as map data, or may be determined from the speed of the vehicle, the type of sign recognized immediately before, or the like. Whether or not an electric sign is determined may be determined from the edge gradient in the sign detection unit 1a. Since there is no restriction information, it may be determined that the sign is an electronic sign.
  • step S3 the sign determination unit 1d sets a joint possibility flag for the currently targeted sign candidate 30. After that, the process proceeds to step S4.
  • step S4 the sign determination unit 1d confirms whether or not the target sign candidate 30 is under conditions that allow it to be determined as a road sign.
  • the judgment possible condition is when both of (1) the distance to the sign candidate 30 is 12.45 m or less and (2) the number of time-series tracking results is two or more. If the current target sign candidate 30 is under the condition that can be determined as a road sign, the process proceeds to step S7. Otherwise (for example, if the distance from the vehicle V to the sign candidate 30 is 13 m or more), step Move to S5.
  • step S5 the sign determination unit 1d confirms whether or not another sign candidate 30 with the joint possibility flag set is registered in the notification list L2. Then, if another sign candidate 30 with the joint possibility flag set is registered in the notification list L2, the process proceeds to step S6, and if not, the sign determination processing for that sign candidate 30 ends.
  • step S6 the sign determination unit 1d sets a simultaneous determination flag for the current target sign candidate 30, indicating that simultaneous determination with other sign candidates 30 in the notification list L2 is required. After that, the process moves to step S7.
  • step S7 the sign determination unit 1d determines whether the currently targeted sign candidate 30 corresponds to a road sign. Specifically, the time-series sign identification processing results obtained by sign tracking are used to determine whether the sign corresponds to a road sign. Then, if the current target sign candidate 30 corresponds to a road sign, the process proceeds to step S8, and if not, the process proceeds to step S9.
  • step S8 the sign determination unit 1d registers the current target sign candidate 30, which was determined to be a road sign in step S7, in the notification list L2. After that, the process proceeds to step S9.
  • step S9 the sign determination unit 1d determines whether or not the target sign candidate 30 needs to undergo sign determination again. For example, in steps S5 and S6, there is a possibility that a sign candidate 30 that is not actually an additional sign is determined at the same timing. to implement.
  • step S7 (1) the sign candidate 30 determined to be a road sign in step S7, or (2) the sign candidate 30 determined not to be a road sign in step S7 and for which the simultaneous determination flag is not set. , since re-determination is not necessary, the process moves to step S10 to set a determination completion flag. On the other hand, for the sign candidate 30 that does not meet the above conditions, for example, the simultaneous determination flag is set and the sign candidate 30 is determined not to be a road sign in step S7, the determination is made so that it can be re-determined later. End the traffic sign determination process without setting the flag.
  • step S10 the sign determination unit 1d sets a determination completion flag for the sign candidate 30 determined as not requiring re-determination in step S9.
  • Scene 1 includes image P (left image P L or right image P R ) in FIG. 4A and, as shown in the top view of FIG. An environment in which candidate signs 32 indicating the limit of 100 kph are arranged laterally adjacent to each other. In this environment, the flowchart of FIG. 3 will be described assuming that both of the candidate markers 31 and 32 have been registered in the tracking list L1 by the marker tracking unit 1c.
  • the adjacently arranged sign candidates 31 and 32 are judged at the same timing to be juxtaposed signs, and both are registered in the notification list L2 as road signs.
  • a sign recognition result R is expected to be generated.
  • Table 1 shows the features of the candidate markers 31 and 32 in FIGS. 4A and 4B and the candidate marker 33 in FIGS. 5A and 5B, which will be described later.
  • step S1 the sign determination unit 1d checks whether or not there is a determined flag for the candidate sign 31 in the tracking list L1. is not set, and the process proceeds to step S2.
  • step S2 the sign determination unit 1d determines whether the sign candidate 31 is a sign that can be installed together.
  • the sign candidate 31 has three chronological tracking results, the road type is a highway, the sign system is an electronic sign, and the speed limit sign is 80 kph. In light of the above, it is determined to be a potential sign candidate.
  • step S3 the sign determination unit 1d sets the joint possibility flag to the sign candidate 31.
  • step S4 the sign determination unit 1d determines whether the sign candidate 31 can be determined. Since the calculated distance from the vehicle V for the sign candidate 31 in scene 1 is 12 m, and the number of time-series tracking results is 3, it is determined that the determination is possible in light of the above-described two conditions of step S4, and the process proceeds to step S7. .
  • step S7 the sign determination unit 1d determines whether the sign candidate 31 corresponds to a road sign.
  • the sign candidate 31 is a road sign indicating the speed limit of 80 kph, and the process proceeds to step S8.
  • step S8 the sign determination unit 1d registers the sign candidates 31 determined as road signs in the notification list L2. After that, the process proceeds to step S9.
  • step S ⁇ b>9 the sign determination unit 1 d determines whether or not the candidate sign 31 needs to be re-determined.
  • the sign candidate 31 has already been judged to be a road sign, and re-judgment is unnecessary, so the process proceeds to step S10.
  • step S10 the sign determination unit 1d sets a judged flag for the sign candidate 31.
  • the processing of the sign determination unit 1d for the sign candidates 31 in the tracking list L1 is completed.
  • step S1 the sign determination unit 1d confirms whether or not there is a determined flag for the sign candidate 32, and determines whether or not there is a possibility that the sign candidate 32 may be placed together. 1, a joint possibility flag is set for the sign candidate 32, which indicates that the sign can be jointly arranged. After that, the process proceeds to step S4.
  • step S4 the sign determination unit 1d determines whether the sign candidate 32 can be determined. As shown in Table 1, the actual distance from the vehicle V to the sign candidate 32 is 12 m. However, in Table 1, the calculated distance to the sign candidate 32 is erroneously 32 m. is determined to be undeterminable, and the process proceeds to step S5.
  • step S5 the sign determination unit 1d confirms whether or not another sign candidate with a joint possibility flag set is registered in the notification list L2.
  • the possibility flag is set for the candidate sign 31 in the notification list L2, so the process proceeds to step S6.
  • step S6 the sign determination unit 1d sets a simultaneous determination flag for the sign candidate 32, and proceeds to step S7.
  • the sign determination unit 1d determines whether the sign candidate 32 corresponds to a road sign.
  • the sign candidate 32 is determined to be a road sign indicating a speed limit of 100 kph, so the sign candidate 32 is registered in the notification list L2. After that, the process proceeds to step S9.
  • the sign determination unit 1d sets a determination flag for the sign candidate 32 because the sign candidate 32 has already been determined to be a road sign. This completes the sign determination processing for the sign candidates 32 in the tracking list L1.
  • both the sign candidate 31 and the sign candidate 32 can be registered in the notification list L2.
  • the sign candidate 31 even if the sign candidate 32 is determined not to satisfy the judgment condition in step S4 due to an error in the calculated distance, the sign candidate 31, and by judging whether or not notification can be made at the same timing as the sign candidate 31, registration timing in the notification list L2 can be aligned. This is the action of the sign recognition device 100 of this embodiment. As a result, the sign information of both of the sign candidates 31 and 32 can be output to the sign selection unit 1e at the same timing.
  • the sign selection unit 1e selects sign information to be notified to the driver according to the sign information reporting form of the output device 20.
  • FIG. For example, if there is a liquid crystal monitor of the vehicle V as the output device 20, and if it is a notification form that can display a plurality of signs, as shown in FIG.
  • Both of the recognized speed limit 100 kph sign information may be selected, or if only one sign information is in the form of sign information that can be displayed on a meter, the sign candidate 31 and the sign candidate 32 according to a predetermined rule. Either label information may be selected.
  • the vehicle V is a small vehicle, as shown in FIG. 4B(b)
  • only the fast speed limit of 100 kph may be displayed, and if the vehicle V is a large vehicle, as shown in FIG. may display only the slow speed limit of 80 kph.
  • the sign determination unit 1d and the sign recognition result R are stored only for the sign candidate 31, and the driver is notified by the output device 20 via CAN. do. After that, the same process is performed on the candidate sign 33, and the operation is expected to be notified to the driver by the output device 20.
  • the sign determination unit 1d confirms whether or not a flag candidate 33 in the tracking list L1 has a determined flag, and determines whether or not the candidate sign 33 in the tracking list L1 has a flag, and determines whether the candidate sign 33 can be placed together. After that, based on the feature of the candidate sign 33 shown in Table 1, a joint possibility flag is set in the candidate sign 33 to indicate that the sign has a possibility of being jointly arranged. After that, the process proceeds to step S4.
  • step S4 the sign determination unit 1d determines whether the sign candidate 33 can be determined. As shown in Table 1, the calculated distance from the vehicle V to the sign candidate 33 is 35 m, which does not satisfy the condition of 12.45 m or less, so the process proceeds to step S5.
  • step S5 the sign determination unit 1d confirms whether or not another sign candidate with a joint possibility flag set is registered in the notification list L2.
  • the possibility flag is set for the candidate sign 31 in the notification list L2, so the process proceeds to step S6. Note that if there is no sign candidate with the flag of possibility of being added together in the notification list L2, the sign determination processing for the sign candidate 33 ends.
  • step S6 the sign determination unit 1d sets a simultaneous determination flag for the sign candidate 33, and proceeds to step S7.
  • step S7 the sign determination unit 1d determines whether the sign candidate 33 corresponds to a road sign.
  • the sign candidate 33 is a road sign indicating a speed limit of 100 kph, but since the identification result is not stable for the sign candidate 33 farther from the sign candidate 31, it is not determined to be a road sign, and the process proceeds to step S9. Transition.
  • step S9 the sign determination unit 1d confirms whether or not the sign candidate 33 needs to be re-determined.
  • the sign candidate 33 of this example is a sign candidate for which it was not determined to be a road sign in step S7 and the simultaneous determination flag was set in step S6. Therefore, the sign candidate 33 is judged to be a sign candidate that needs to be re-determined in light of the determination criteria of step S9 described above, and the sign determination processing for the sign candidate 33 is terminated without executing step S10.
  • the sign candidate 31 is registered as a road sign in the notification list L2, but at this point the sign candidate 33 is not registered in the notification list L2. Therefore, only the sign information (speed limit of 80 kph) of the sign candidate 31 is input to the sign selection unit 1e, and the sign information of the sign candidate 31 is first notified to the driver (see FIG. 5B(a)).
  • step S4 the processing is repeated for the sign candidate 33 for which the judged flag is not set, and when the calculated traveling direction distance satisfies the requirements of step S4, the sign candidate 33 is judged to be a road sign.
  • the sign information (speed limit of 100 kph) of the sign candidate 33 is passed to the sign selection unit 1e and notified to the driver (see FIG. 5B(b)).
  • the sign candidates 31 and 33 that should be judged at different timings may be judged at the same time.
  • the determination flag it is possible to determine the traffic sign candidates 31 and 33 at different timings even in an environment such as scene 2. It is possible to report in order.
  • the traffic sign recognition apparatus of this embodiment when a plurality of speed limit traffic signs are installed side by side, the speed limit traffic signs are recognized at the same timing, and the speed limit information for the own vehicle is output without omission. can do.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Traffic Control Systems (AREA)

Abstract

The purpose of the present invention is to provide a sign recognition device with which, in a situation in which multiple speed limit signs are installed together, it is possible to output complete speed limit information for a vehicle by recognizing the speed limit signs at the same time. This sign recognition device comprises: a sign detection unit that detects a candidate sign from an image photographed by a camera device mounted in a vehicle; a sign identification unit that identifies the type of the candidate sign; a sign tracking unit that tracks identified candidate signs in a plurality of continuously photographed images in association with one another; and a sign determination unit that determines, if a travel-direction distance to one of the identified candidate signs being tracked is equal to or smaller than a predetermined distance, whether or not the identified candidate sign being tracked is a road sign, wherein if the sign tracking unit is tracking a first candidate sign and a second candidate sign and the sign determination unit has already determined that the first candidate sign is a road sign, the sign determination unit determines whether or not the second candidate sign is a road sign even when the second candidate sign is not located within the predetermined distance.

Description

標識認識装置、および、標識認識方法Sign recognition device and sign recognition method
　本発明は、車両前方の撮像画像に基づき道路標識を認識する標識認識装置に関する。 The present invention relates to a sign recognition device that recognizes road signs based on captured images in front of a vehicle.
　近年、自動車に搭載する安全運転支援システムや自動運転システムの更なる性能向上のため、車載カメラ装置による物体認識や距離測定への要求が高度化している。なお、車載カメラ装置は、撮像画像から視覚情報を抽出すると同時に、撮像画像中の対象物までの距離情報を計測する装置であり、自車の周辺環境（歩行者や他車の挙動、立体物や白線の位置や形状、路面の状態、道路標識の種別など）を詳細に把握し、運転支援時等の安全性の向上に寄与するものである。 In recent years, in order to further improve the performance of safe driving support systems and autonomous driving systems installed in automobiles, the demand for object recognition and distance measurement using in-vehicle camera devices has increased. An in-vehicle camera device is a device that extracts visual information from captured images and at the same time measures distance information to objects in the captured images. and the position and shape of white lines, road surface conditions, types of road signs, etc.) in detail, contributing to improved safety when assisting driving.
　車載カメラ装置による認識対象物の一つとして道路標識がある。一般に、標識認識機能は、地図情報と連携して自動運転を行う車の加速又は減速に使われる。また、先進運転支援システムの評価指標であるEuro NCAP（２０１６年～２０２３年アップデート）においても、速度支援システム（ＳＡＳ：Speed Assistance Systems）に関する評価項目が設けられており、その重要度は増している。さらに日本国内では、２０１９年１２月に国土交通省にて加速抑制機能（ＩＳＡ：Intelligent Speed Adaptation）に関する検討会がなされるなど、国内外において標識認識機能の重要性が増している。 　Road signs are one of the objects recognized by the in-vehicle camera device. In general, the sign recognition function is used to accelerate or decelerate a vehicle that performs automatic driving in cooperation with map information. In addition, Euro NCAP (updated from 2016 to 2023), which is an evaluation index for advanced driver assistance systems, also has evaluation items for speed assistance systems (SAS: Speed Assistance Systems), and its importance is increasing. . Furthermore, in Japan, in December 2019, the Ministry of Land, Infrastructure, Transport and Tourism held a study meeting on the acceleration control function (ISA: Intelligent Speed Adaptation), and the importance of sign recognition functions is increasing both domestically and internationally.
　道路標識から規制速度を認識する装置としては、特許文献１の規制速度表示装置が知られている。例えば、同文献の要約書には、解決手段として「規制速度を表示する表示器を備え、画像取得装置によって取得された画像データに基づいて画像中の交通標識に表示されている規制速度を検出し、画像データに基づいて交通標識を検出したときの車両と交通標識との間の横方向の距離ΔDst又はΔDsubが小さい場合、検出された規制速度を表示器に表示する。一方、交通標識と車両との間の距離が大きい場合、検出された規制速度を表示器に表示しない。」が開示されており、これにより、課題として挙げられた「適用されるべき規制速度ではない速度が表示される可能性を低減した車両の規制速度表示装置を提供する。」を実現している。 As a device for recognizing the speed limit from road signs, the speed limit display device of Patent Document 1 is known. For example, in the abstract of the same document, as a solution, it is stated that "the speed limit displayed on the traffic sign in the image is detected based on the image data acquired by the image acquisition device, which is equipped with a display that displays the speed limit. However, if the lateral distance ΔDst or ΔDsub between the vehicle and the traffic sign is small when the traffic sign is detected based on the image data, the detected speed limit is displayed on the display. If the distance from the vehicle is large, the detected speed limit is not displayed on the display. To provide a vehicle speed limit display device that reduces the possibility of
特開２０１６－２１２６９３号公報JP 2016-212693 A
　しかしながら、特許文献１の規制速度表示装置は、車両と交通標識との横方向距離が閾値未満の複数の速度制限標識が併設されている場合（例えば、高速道路で、大型車用の速度制限標識（８０）と、小型車用の速度制限標識（１００）が隣接設置されている場合）には、各々の速度制限標識の検出タイミングがずれると、自車用の規制速度情報ではなく、自車用ではない規制速度情報を、短時間もしくは標識通過後の一定時間表示してしまう可能性があった。 However, the speed limit display device of Patent Document 1 does not work when a plurality of speed limit signs with a lateral distance between the vehicle and the traffic sign being less than a threshold value are installed (for example, on an expressway, speed limit signs for large vehicles (80) and the speed limit sign (100) for small vehicles are installed adjacently), if the detection timing of each speed limit sign is off, the speed limit information for the own vehicle There is a possibility that the speed limit information that is not displayed may be displayed for a short period of time or for a certain period of time after passing the sign.
　本発明は、上記の問題点を解決するためになされたものであり、複数の速度制限標識が併設されている場合、併設標識を同一タイミングで認識することで、自車用の規制速度情報を漏らさず出力できる標識認識装置を提供することを目的とする。 The present invention has been made to solve the above-mentioned problems, and when a plurality of speed limit signs are installed side by side, the speed limit information for the own vehicle can be obtained by recognizing the side signs at the same timing. It is an object of the present invention to provide a sign recognition device capable of outputting without omission.
　上記課題を解決するため、本発明の標識認識装置は、車両に搭載したカメラ装置が撮像した画像から標識候補を検知する標識検知部と、前記標識候補の種別を識別する標識識別部と、連続撮像した複数の画像内の識別済み標識候補を対応付けて追跡する標識追跡部と、追跡中の前記標識候補までの進行方向距離が所定距離内である場合に、前記標識候補が道路標識に該当するかを判定する標識判定部と、を有した標識認識装置であって、前記標識追跡部が第一標識候補と第二標識候補を追跡している場合、前記標識判定部は、前記第一標識候補が道路標識と判定済みであれば、前記第二標識候補が前記所定距離内にない場合であっても、前記第二標識候補が道路標識に該当するかを判定するものとした。 In order to solve the above problems, the sign recognition device of the present invention includes a sign detection unit that detects a sign candidate from an image captured by a camera device mounted on a vehicle, and a sign identification unit that identifies the type of the sign candidate. A sign tracking unit that tracks identified sign candidates in a plurality of captured images in association with each other; and when the travel direction distance to the sign candidate being tracked is within a predetermined distance, the sign candidate corresponds to a road sign. and a sign determination unit that determines whether to If the sign candidate has already been determined to be a road sign, it is determined whether the second sign candidate corresponds to the road sign even if the second sign candidate is not within the predetermined distance.
　本発明の標識認識装置によれば、複数の速度制限標識が併設されている場合、併設標識を同一タイミングで認識し、自車用の規制速度情報を漏らさず出力することができる。 According to the sign recognition device of the present invention, when multiple speed limit signs are installed side by side, the side signs can be recognized at the same timing and the speed limit information for the own vehicle can be output without omission.
一実施例の標識認識装置のハードウェア構成を示す概略図。Schematic diagram showing the hardware configuration of the sign recognition device of one embodiment. 一実施例のカメラ装置の機能ブロック図。1 is a functional block diagram of a camera device according to one embodiment; FIG. 図２の標識認識の処理内容を表すフローチャート。FIG. 3 is a flow chart showing processing contents of sign recognition in FIG. 2 ; FIG. シーン１の撮像画像。A captured image of scene 1. 図４Ａに対応する上面図。FIG. 4B is a top view corresponding to FIG. 4A; シーン２の撮像画像。A captured image of scene 2. 図５Ａに対応する上面図。FIG. 5B is a top view corresponding to FIG. 5A;
　以下、図面を用いて、車両Ｖに搭載した、本発明の一実施例に係る標識認識装置１００を説明する。標識認識装置１００は、車両Ｖの前方の撮影対象領域を画像処理して車外環境を認識する車載装置であり、例えば、道路の白線、歩行者、他車、その他の立体物、信号機、道路標識、点灯ランプなどを認識する機能を持ち、また、道路標識の真偽を判定する機能を持つ。なお、標識認識装置１００は、車両Ｖのブレーキ及びステアリングなどの調整を行う機能を備えても良いが、この機能は省略しても良い。 A sign recognition device 100 according to an embodiment of the present invention mounted on a vehicle V will be described below with reference to the drawings. The sign recognition device 100 is an in-vehicle device that performs image processing on a shooting target area in front of the vehicle V to recognize the environment outside the vehicle. It has the function of recognizing lighting lamps, etc., and also has the function of judging the authenticity of road signs. The sign recognition device 100 may have a function of adjusting the brakes and steering of the vehicle V, but this function may be omitted.
　図１は、標識認識装置１００の概略構成を示すブロック図である。ここに示すように、標識認識装置１００は、カメラ装置１０と出力装置２０をＣＡＮ（Controller Area Network）で接続した装置であり、ＣＡＮを介して他のコンピュータユニット（例えば、車両Ｖの安全運転支援システムや自動運転システムを構成するＥＣＵ（Electronic Control Unit）等）と接続しても良い。以下、各部について順次説明する。 FIG. 1 is a block diagram showing a schematic configuration of the sign recognition device 100. FIG. As shown here, the sign recognition device 100 is a device in which the camera device 10 and the output device 20 are connected via a CAN (Controller Area Network). It may be connected to an ECU (Electronic Control Unit), etc.) that constitutes a system or an automatic driving system. Each unit will be described below in order.
　＜カメラ装置１０＞
　カメラ装置１０は、カメラ１１（左カメラ１１Ｌ、右カメラ１１Ｒ）、画像入力インタフェース１２、画像処理部１３、演算処理部１４、記憶部１５、ＣＡＮインタフェース１６、制御処理部１７を備えている。カメラ１１以外の、画像入力インタフェース１２から制御処理部１７は、内部バスを介して相互に接続されており、単一又は複数のコンピュータユニットにより構成することができる。図１では、これらを単一のコンピュータユニットで構成した場合を例示している。 <Camera device 10>
The camera device 10 includes a camera 11 (a left camera 11L and a right camera 11R), an image input interface 12, an image processing section 13, an arithmetic processing section 14, a storage section 15, a CAN interface 16, and a control processing section 17. The image input interface 12 to the control processing unit 17 other than the camera 11 are interconnected via an internal bus, and can be configured by a single or a plurality of computer units. FIG. 1 exemplifies a case where these are configured by a single computer unit.
　左カメラ１１Ｌと右カメラ１１Ｒは、車両Ｖの前方をステレオ撮像するために所定間隔を空けて左右に配置されており、左カメラ１１Ｌの撮像素子で左画像ＰＬを撮像し、右カメラ１１Ｒの撮像素子で右画像ＰＲを撮像する。 The left camera 11L and the right camera 11R are arranged on the left and right with a predetermined interval in order to stereoscopically capture an area in front of the vehicle V. The imaging element of the left camera 11L captures the left image PL, A right image PR is captured by the imaging device.
　画像入力インタフェース１２は、カメラ１１を制御し、撮像した画像Ｐ（左画像ＰＬ、右画像ＰＲ）を取り込む。画像入力インタフェース１２を通して取り込まれた画像Ｐは、内部バスを通して、画像処理部１３や演算処理部１４等に送られる。 The image input interface 12 controls the camera 11 and takes in captured images P (left image P L , right image P R ). An image P captured through the image input interface 12 is sent to the image processing section 13, the arithmetic processing section 14, etc. through the internal bus.
　画像処理部１３は、左画像ＰＬと右画像ＰＲを比較し、各画像に対して撮像素子に起因するデバイス固有の偏差の補正、及びノイズ補間などの画像補正を行い、これを記憶部１５に記憶する。さらに、左画像ＰＬ及び右画像ＰＲの間で、相互に対応する箇所を計算して、視差情報（同一点を異なった二点から眺めた時に見える方向の違い、すなわち二つの方向の間の角度の情報）を計算し、これを記憶部１５に記憶する。 The image processing unit 13 compares the left image P L and the right image P R , corrects device-specific deviations caused by the image sensor and performs image correction such as noise interpolation on each image, and stores the images in the storage unit. 15. Furthermore, between the left image P L and the right image P R , mutually corresponding parts are calculated and parallax information (the difference in the direction seen when the same point is viewed from two different points, that is, the difference between the two directions) is calculated. angle information) is calculated and stored in the storage unit 15 .
　演算処理部１４は、様々な演算処理を実施するＣＰＵ等の演算装置であり、所定のプログラムを実行することで、記憶部１５に蓄えた画像Ｐから道路標識を認識する機能に相当する標識認識部１を実現する。標識認識部１が道路標識認識の基礎とする画像Ｐは、左画像ＰＬ、右画像ＰＲのいずれでも良く、また双方の画像を用いても良い。ここでの道路標識の認識方法は、例えば画像処理により認識された形状を、例えば記憶部１５に格納された道路標識のデータベースと照合し、いずれかの道路標識との合致度が一定以上のものを道路標識として認識し、複数の道路標識が存在する場合は、自車が遵守すべき標識情報を出力するというものである。本処理については後ほど詳述する。 The arithmetic processing unit 14 is an arithmetic unit such as a CPU that performs various kinds of arithmetic processing. Implement Part 1. The image P that the sign recognition unit 1 uses as a basis for road sign recognition may be either the left image P L or the right image P R , or both images may be used. The method for recognizing road signs here is to compare the shape recognized by image processing with, for example, a database of road signs stored in the storage unit 15, and match the shape with any one of the road signs above a certain degree. is recognized as a road sign, and if there are multiple road signs, the sign information to be observed by the own vehicle is output. This processing will be described in detail later.
　記憶部１５は、例えば半導体メモリであり、画像処理部１３や演算処理部１４で得られた情報等を記憶する。 The storage unit 15 is, for example, a semiconductor memory, and stores information and the like obtained by the image processing unit 13 and the arithmetic processing unit 14.
　ＣＡＮインタフェース１６は、ＣＡＮに対する入出力部であり、カメラ装置１０の演算情報は、このＣＡＮインタフェース１６を介してＣＡＮに出力され、ＣＡＮを介して車両Ｖの制御システム（ＥＣＵ等）に出力される。具体的には、ＣＡＮインタフェース１６は、ＣＡＮを介して、車両Ｖが遵守するべき標識情報を、標識認識部１が取得した場合に出力装置２０に標識認識結果を出力する。 The CAN interface 16 is an input/output unit for the CAN, and the calculation information of the camera device 10 is output to the CAN via the CAN interface 16 and output to the control system (ECU, etc.) of the vehicle V via the CAN. . Specifically, the CAN interface 16 outputs a sign recognition result to the output device 20 when the sign recognition unit 1 acquires sign information to be observed by the vehicle V via the CAN.
　制御処理部１７は、以上のカメラ装置１０の動作について、各処理部が異常動作を起こしていないか、又はデータ転送時にエラーが発生していないかどうかなどを監視して異常動作を防ぐ役割を果たす。 The control processing unit 17 has the role of preventing abnormal operations by monitoring whether each processing unit is operating abnormally or whether an error has occurred during data transfer. Fulfill.
　＜出力装置２０＞
　カメラ装置１０で演算された道路標識の判定結果等は、ＣＡＮを介して、出力装置２０に送信される。 <Output device 20>
The road sign judgment results and the like calculated by the camera device 10 are transmitted to the output device 20 via the CAN.
　出力装置２０は、例えば、車両Ｖの運転室に搭載された液晶モニタ（図４Ｂ、図５Ｂ参照）やランプ、ブザー、スピーカ等の表示出力や音声出力をする装置であり、カメラ装置１０から入力された情報を基に道路標識を視覚的又は聴覚的に乗員に報知するものである。報知の方法は単に道路標識の有無のみを知らせるようにすることもできるし、道路標識の種類も併せて知らせるようにすることもできる。また、視覚的出力態様と聴覚的出力態様の少なくとも一方を選択できる構成とすることもできる。 The output device 20 is, for example, a liquid crystal monitor (see FIGS. 4B and 5B), a lamp, a buzzer, a speaker, etc. mounted in the driver's cab of the vehicle V for display output and audio output. Based on the received information, the driver is visually or audibly notified of road signs. As for the method of notification, it is possible to simply notify the presence or absence of a road sign, or it is also possible to notify the type of road sign as well. Also, it is possible to adopt a configuration in which at least one of the visual output mode and the auditory output mode can be selected.
　また、出力装置２０が安全運転支援システムや自動運転システムであり、車両Ｖのブレーキやステアリングを制御する場合には、システムによる制御の基礎情報にカメラ装置１０で演算した道路標識の情報等が付加される。 In addition, when the output device 20 is a safe driving support system or an automatic driving system and controls the brakes and steering of the vehicle V, information such as road sign information calculated by the camera device 10 is added to the basic information for control by the system. be done.
　＜カメラ装置１０の処理フロー＞
　次に、図２の機能ブロック図を用いて、カメラ装置１０による標識認識の処理フローを概説する。図２は、図１のカメラ装置１０の機能を模式的に表したものであり、図２の標識認識部１内の各部は、図１の演算処理部１４（ＣＰＵ等）が所定のプログラムを実行することで実現した機能に相当する。 <Processing Flow of Camera Device 10>
Next, the processing flow of sign recognition by the camera device 10 will be outlined using the functional block diagram of FIG. FIG. 2 schematically shows the functions of the camera device 10 in FIG. 1. Each part in the sign recognition part 1 in FIG. It corresponds to the function realized by executing it.
　図２のカメラ装置１０では、まず、左カメラ１１Ｌと右カメラ１１Ｒによる、車両Ｖの前方の撮像が行われ、左画像ＰＬと右画像ＰＲが取得される。そして、標識認識部１では、カメラ１１で撮像した単眼画像（左画像ＰＬまたは右画像ＰＲの一方）に基づいて、標識認識処理を実施する。以下、標識認識部１で実施される各処理を順次説明する。 In the camera device 10 of FIG. 2, first, the left camera 11L and the right camera 11R capture an image in front of the vehicle V to obtain a left image PL and a right image PR . Then, the sign recognition unit 1 performs sign recognition processing based on the monocular image (either the left image PL or the right image PR) captured by the camera 11 . Each processing performed by the sign recognition unit 1 will be sequentially described below.
　まず、標識検知部１ａでは、画像Ｐから標識候補を検知する。本実施例では、道路標識の外形を円形と想定しているため、標識検知部１ａは、画像処理により真円との一致度が一定以上と推定される画像Ｐ中の形状を標識候補３０として抽出する。 First, the sign detection unit 1a detects sign candidates from the image P. In this embodiment, it is assumed that the outer shape of the road sign is circular. Therefore, the sign detection unit 1a selects a shape in the image P that is estimated to have a certain degree of coincidence with a perfect circle by image processing as the sign candidate 30. Extract.
　次に、標識識別部１ｂでは、標識候補３０の種別を識別する。具体的には、標識検知部１ａで検出した標識候補３０を、例えば記憶部１５に格納された標識データベース（認識辞書）と照合し、標識候補３０の種別（標識候補３０がどの道路標識の候補であるのか）を識別する。 Next, the sign identification unit 1 b identifies the type of the sign candidate 30 . Specifically, the sign candidate 30 detected by the sign detection unit 1a is collated with, for example, a sign database (recognition dictionary) stored in the storage unit 15, and the type of the sign candidate 30 (which road sign candidate the sign candidate 30 is a candidate for) is determined. ).
　標識追跡部１ｃでは、連続的に撮像した複数の画像Ｐの対応付けを行う。具体的には、車両Ｖの走行等に伴って車両Ｖとの相対位置が変位する、種別識別済みの標識候補３０を、連続する複数フレームの画像Ｐに対する画像処理により対応付けて追跡する。そして、追跡中の全ての標識候補３０を追跡リストＬ１に登録する。 The marker tracking unit 1c associates a plurality of images P captured continuously. Specifically, the type-identified sign candidates 30 whose relative positions with respect to the vehicle V are displaced as the vehicle V travels or the like are associated with each other by image processing of the images P of a plurality of continuous frames and tracked. Then, all tracked candidate signs 30 are registered in the tracking list L1.
　標識判定部１ｄでは、標識候補３０に対し複数フレームでの総合的な判断を下す。具体的には、複数フレームの画像Ｐを画像処理し、追跡中の標識候補３０が本物の道路標識であるかが判定される。そして、本物の道路標識と判定された標識候補３０は、通知リストＬ２に登録される。なお、この処理は、追跡リストＬ１に登録された全ての標識候補３０に対し実施されるため、複数の道路標識の併設時には、通知リストＬ２に複数の道路標識が登録される。この標識判定処理の詳細については、図３を用いて説明する。 The sign determination unit 1d makes a comprehensive judgment on the sign candidate 30 in a plurality of frames. Specifically, image processing is performed on images P of a plurality of frames, and it is determined whether or not the sign candidate 30 being tracked is a genuine road sign. Then, the sign candidate 30 determined to be a genuine road sign is registered in the notification list L2. Note that this processing is performed for all the sign candidates 30 registered in the tracking list L1, so when a plurality of road signs are placed side by side, a plurality of road signs are registered in the notification list L2. The details of this sign determination process will be described with reference to FIG.
　標識選出部１ｅでは、自車に関連する道路標識を選出する。具体的には、標識判定部１ｄで判定した道路標識が、自車の遵守すべき標識であるかどうかの判断を行い、適用するアプリケーションに適した形態で標識認識結果Ｒを生成する。例えば、自車用の速度制限情報を液晶モニタに表示するというアプリケーションであれば、大型車用と小型車用の速度制限標識が隣接している場合には、自車が小型車であれば、少なくとも小型車用の速度制限情報を含むように標識認識結果Ｒを選出する。 The sign selection unit 1e selects road signs related to the own vehicle. Specifically, it determines whether or not the road sign determined by the sign determination unit 1d is a sign that the vehicle should follow, and generates a sign recognition result R in a form suitable for the application to be applied. For example, in the case of an application that displays the speed limit information for the vehicle on the LCD monitor, if the speed limit signs for large and small vehicles are adjacent to each other, if the vehicle is a small vehicle, at least the small vehicle Elect the sign recognition result R to contain speed limit information for .
　最後に、標識認識部１で得られた標識認識結果Ｒを記憶部１５に保存する。標識認識部１による標識認識結果Ｒは記憶部１５に保存されると同時に、ＣＡＮを介して他のコンピュータユニットや出力装置２０に出力される。従って、本実施例では、標識選出部１ｅで選出した、自車の遵守すべき標識情報が出力装置２０に出力され、車両Ｖの乗員に報知される。 Finally, the sign recognition result R obtained by the sign recognition unit 1 is saved in the storage unit 15. The sign recognition result R by the sign recognition unit 1 is stored in the storage unit 15 and simultaneously output to another computer unit or the output device 20 via CAN. Therefore, in this embodiment, the information on the signs to be observed by the own vehicle selected by the sign selection unit 1e is output to the output device 20, and the occupants of the vehicle V are notified of the information.
　＜標識判定部１ｄの処理＞
　以下、図３のフローチャートに基づいて、標識判定部１ｄの処理フローを詳細に説明する。なお、図３の処理は、追跡リストＬ１に登録された全ての標識候補３０に対して順次実施されるが、追跡リストＬ１に複数の標識候補３０が登録されている場合の具体的な処理は後述することとし、まずは、追跡リストＬ１から選択した任意の一つの標識候補３０（以下「現在対象の標識候補３０」と称する）に対する基本的な判定処理の流れを説明する。 <Processing of the sign determination unit 1d>
Hereinafter, based on the flowchart of FIG. 3, the process flow of the sign determination part 1d is demonstrated in detail. The processing of FIG. 3 is sequentially performed for all the sign candidates 30 registered in the tracking list L1. As will be described later, first, the flow of basic determination processing for an arbitrary one of the sign candidates 30 selected from the tracking list L1 (hereinafter referred to as "current target sign candidate 30") will be described.
　ステップＳ１では、標識判定部１ｄは、現在対象の標識候補３０に判定済フラグが設定されているか確認する。判定済フラグが設定されている場合は、現在対象の標識候補３０が道路標識であるか判定済みであるため、その標識候補３０には再判定処理を実施せず処理を終了する。一方、判定済フラグが設定されていない場合は、ステップＳ２に移行する。 In step S1, the sign determination unit 1d confirms whether or not a determination completion flag is set for the target sign candidate 30 at present. If the determination completion flag is set, it has already been determined whether the target sign candidate 30 is a road sign. On the other hand, if the determination completion flag is not set, the process proceeds to step S2.
　ステップＳ２では、標識判定部１ｄは、現在対象の標識候補３０に対して、他の標識候補３０との併設可能性があるか判定する。ここで参照する併設可能性の判定条件は、例えば、（１）時系列追跡結果数が２個以上であること、（２）現在走行している道路が高速道路であること、（３）標識候補３０が電光式標識であること、（４）標識候補３０が５０ｋｐｈから１２０ｋｐｈの速度制限標識であること、の全てを満たす場合などである。
そして、併設可能性があれば、ステップＳ３に移行し、併設可能性がなければ、ステップＳ４に移行する。 In step S<b>2 , the sign determination unit 1 d determines whether or not there is a possibility that the current target sign candidate 30 is placed side by side with another sign candidate 30 . The conditions for judging the possibility of co-locating referred to here are, for example, (1) that the number of time-series tracking results is two or more, (2) that the current road is an expressway, and (3) a traffic sign. For example, the candidate 30 is an electronic sign, and (4) the candidate 30 is a 50 kph to 120 kph speed limit sign.
If there is a possibility of co-locating, the process proceeds to step S3, and if there is no possibility of co-locating, the process proceeds to step S4.
　なお、ステップＳ２で参照する条件は、適用する国や地域の道路事情に応じて適宜変更しても良い。走行中の道路の判断については、地図データ等の外部情報から取得しても良いし、自車速度や直前に認識した標識種別などから判断しても良い。また、電光式標識であるかどうかの判定については、標識検知部１ａでのエッジ勾配から判断しても良いし、電光式標識が提供する情報が可変であることに着目し、地図情報に速度制限の情報が無いということから電光式標識であると判断しても良い。 It should be noted that the conditions referred to in step S2 may be changed as appropriate according to the road conditions of the country or region in which they are applied. The determination of the road on which the vehicle is traveling may be obtained from external information such as map data, or may be determined from the speed of the vehicle, the type of sign recognized immediately before, or the like. Whether or not an electric sign is determined may be determined from the edge gradient in the sign detection unit 1a. Since there is no restriction information, it may be determined that the sign is an electronic sign.
　ステップＳ３では、標識判定部１ｄは、現在対象の標識候補３０に対して、併設可能性フラグを設定する。その後、ステップＳ４へと移行する。 In step S3, the sign determination unit 1d sets a joint possibility flag for the currently targeted sign candidate 30. After that, the process proceeds to step S4.
　ステップＳ４では、標識判定部１ｄは、現在対象の標識候補３０が道路標識として判定可能な条件下にあるか確認する。判定可能条件は、例えば、（１）標識候補３０までの距離が１２．４５ｍ以下であることと、（２）時系列追跡結果が２個以上であること、の双方を満たす場合などである。そして、現在対象の標識候補３０が道路標識として判定可能な条件下にある場合はステップＳ７に移行し、そうでない場合（例えば、車両Ｖから標識候補３０までの距離が１３ｍ以上ある場合）はステップＳ５に移行する。 In step S4, the sign determination unit 1d confirms whether or not the target sign candidate 30 is under conditions that allow it to be determined as a road sign. For example, the judgment possible condition is when both of (1) the distance to the sign candidate 30 is 12.45 m or less and (2) the number of time-series tracking results is two or more. If the current target sign candidate 30 is under the condition that can be determined as a road sign, the process proceeds to step S7. Otherwise (for example, if the distance from the vehicle V to the sign candidate 30 is 13 m or more), step Move to S5.
　ステップＳ５では、標識判定部１ｄは、通知リストＬ２中に併設可能性フラグが立った他の標識候補３０が登録されているかを確認する。そして、通知リストＬ２に併設可能性フラグが立った他の標識候補３０が登録されている場合は、ステップＳ６へ移行し、無かった場合は、その標識候補３０への標識判定処理は終了する。 In step S5, the sign determination unit 1d confirms whether or not another sign candidate 30 with the joint possibility flag set is registered in the notification list L2. Then, if another sign candidate 30 with the joint possibility flag set is registered in the notification list L2, the process proceeds to step S6, and if not, the sign determination processing for that sign candidate 30 ends.
　ステップＳ６では、標識判定部１ｄは、現在対象の標識候補３０に対して、通知リストＬ２中の他の標識候補３０との同時判定が必要であることを示す同時判定フラグを設定する。その後、ステップＳ７に移行する。 In step S6, the sign determination unit 1d sets a simultaneous determination flag for the current target sign candidate 30, indicating that simultaneous determination with other sign candidates 30 in the notification list L2 is required. After that, the process moves to step S7.
　ステップＳ７では、標識判定部１ｄは、現在対象の標識候補３０が道路標識に該当するか判定する。具体的には、標識追跡により得られた時系列の標識識別処理結果等を利用し、道路標識に該当するかを判定する。そして、現在対象の標識候補３０が道路標識に該当する場合はステップＳ８に移行し、該当しない場合はステップＳ９に移行する。 In step S7, the sign determination unit 1d determines whether the currently targeted sign candidate 30 corresponds to a road sign. Specifically, the time-series sign identification processing results obtained by sign tracking are used to determine whether the sign corresponds to a road sign. Then, if the current target sign candidate 30 corresponds to a road sign, the process proceeds to step S8, and if not, the process proceeds to step S9.
　ステップＳ８では、標識判定部１ｄは、ステップＳ７にて道路標識であると判定された現在対象の標識候補３０を通知リストＬ２に登録する。その後、ステップＳ９に移行する。 In step S8, the sign determination unit 1d registers the current target sign candidate 30, which was determined to be a road sign in step S7, in the notification list L2. After that, the process proceeds to step S9.
　ステップＳ９では、標識判定部１ｄは、現在対象の標識候補３０に対して、標識判定を再実施する必要があるかの判定を行う。例えば、ステップＳ５、Ｓ６によって、実際には併設標識ではない標識候補３０を同じタイミングで判定している可能性もあるため、そのような標識候補３０を適切なタイミングで再判定するため、本処理を実施する。 In step S9, the sign determination unit 1d determines whether or not the target sign candidate 30 needs to undergo sign determination again. For example, in steps S5 and S6, there is a possibility that a sign candidate 30 that is not actually an additional sign is determined at the same timing. to implement.
　すなわち、（１）ステップＳ７で道路標識であると判定された標識候補３０、もしくは、（２）ステップＳ７で道路標識でないと判定され、かつ、同時判定フラグが設定されていない標識候補３０については、再判定が不要であるため、判定済フラグを設定すべく、ステップＳ１０に移行する。一方、上記条件に該当しない標識候補３０、例えば、同時判定フラグが設定されており、かつ、ステップＳ７で道路標識でないと判定された標識候補３０については、後で再判定できるように、判定済フラグを設定せずに標識判定処理を終了する。 That is, (1) the sign candidate 30 determined to be a road sign in step S7, or (2) the sign candidate 30 determined not to be a road sign in step S7 and for which the simultaneous determination flag is not set. , since re-determination is not necessary, the process moves to step S10 to set a determination completion flag. On the other hand, for the sign candidate 30 that does not meet the above conditions, for example, the simultaneous determination flag is set and the sign candidate 30 is determined not to be a road sign in step S7, the determination is made so that it can be re-determined later. End the traffic sign determination process without setting the flag.
　ステップＳ１０では、標識判定部１ｄは、ステップＳ９で再判定不要と判定された標識候補３０に対し、判定済フラグを設定する。 In step S10, the sign determination unit 1d sets a determination completion flag for the sign candidate 30 determined as not requiring re-determination in step S9.
　以上で追跡リスト中の一つの標識候補３０に対する標識判定の処理が終了する。続いて、複数の標識候補３０が併存する環境下での、標識判定部１ｄの処理について、高速道路におけるシーン１とシーン２を例に説明する。 This completes the sign determination processing for one sign candidate 30 in the tracking list. Next, the processing of the sign determination unit 1d in an environment where a plurality of sign candidates 30 coexist will be described using scene 1 and scene 2 on an expressway as an example.
　＜シーン１＞
　シーン１は、図４Ａの画像Ｐ（左画像ＰＬまたは右画像ＰＲ）と、図４Ｂの上面図に示すように、大型車用の速度制限８０ｋｐｈを示す標識候補３１と、小型車用の速度制限１００ｋｐｈを示す標識候補３２が、横方向に隣接配置された環境である。この環境において、標識追跡部１ｃにより、追跡リストＬ１には、標識候補３１、３２の双方が登録済みであるものとして、図３のフローチャートの説明を行う。 <Scene 1>
Scene 1 includes image P (left image P L or right image P R ) in FIG. 4A and, as shown in the top view of FIG. An environment in which candidate signs 32 indicating the limit of 100 kph are arranged laterally adjacent to each other. In this environment, the flowchart of FIG. 3 will be described assuming that both of the candidate markers 31 and 32 have been registered in the tracking list L1 by the marker tracking unit 1c.
　この環境下では、隣接配置された標識候補３１、３２は、併設標識として同じタイミングで標識判定され、双方が道路標識として通知リストＬ２に登録された後、その通知リストＬ２に基づいて、所望の標識認識結果Ｒが生成されることが期待される。 Under this environment, the adjacently arranged sign candidates 31 and 32 are judged at the same timing to be juxtaposed signs, and both are registered in the notification list L2 as road signs. A sign recognition result R is expected to be generated.
　但し、本シーンでは、標識検知部１ａでの検知ズレなどにより標識候補３２までの距離を誤算しているものと想定して、図３のフローチャートを説明する。なお、図４Ａ、図４Ｂの標識候補３１、３２、および、後述する図５Ａ、図５Ｂの標識候補３３の素性は表１に示すものとする。 However, in this scene, the flowchart in FIG. 3 will be described on the assumption that the distance to the candidate sign 32 is miscalculated due to a detection deviation in the sign detection unit 1a. Table 1 shows the features of the candidate markers 31 and 32 in FIGS. 4A and 4B and the candidate marker 33 in FIGS. 5A and 5B, which will be described later.
Figure JPOXMLDOC01-appb-T000001
Figure JPOXMLDOC01-appb-T000001
　＜＜正しい算出距離の標識候補３１に対する処理＞＞
　まず、ステップＳ１では、標識判定部１ｄは、追跡リストＬ１中の標識候補３１について、判定済フラグの有無を確認するが、標識候補３１に対する標識判定部１ｄの処理は未実施のため判定済フラグは設定されておらず、ステップＳ２に移行する。 <<Process for Sign Candidate 31 with Correctly Calculated Distance>>
First, in step S1, the sign determination unit 1d checks whether or not there is a determined flag for the candidate sign 31 in the tracking list L1. is not set, and the process proceeds to step S2.
　次に、ステップＳ２では、標識判定部１ｄは、標識候補３１に対して、併設可能性のある標識であるかを判定する。標識候補３１は、表１に示すように、時系列追跡結果が３個、道路種別が高速道路、標識方式は電光式標識、８０ｋｐｈの速度制限標識であることから、上記したステップＳ２の４条件に照らし、併設可能性のある標識候補と判定される。 Next, in step S2, the sign determination unit 1d determines whether the sign candidate 31 is a sign that can be installed together. As shown in Table 1, the sign candidate 31 has three chronological tracking results, the road type is a highway, the sign system is an electronic sign, and the speed limit sign is 80 kph. In light of the above, it is determined to be a potential sign candidate.
　そのため、ステップＳ３では、標識判定部１ｄは、標識候補３１に併設可能性フラグを設定する。 Therefore, in step S3, the sign determination unit 1d sets the joint possibility flag to the sign candidate 31.
　ステップＳ４では、標識判定部１ｄは、標識候補３１の判定可否を判定する。シーン１の標識候補３１は車両Ｖからの算出距離が１２ｍ、時系列追跡結果が３個であるため、上記したステップＳ４の２条件に照らし、判定可能であると判定し、ステップＳ７に移行する。 In step S4, the sign determination unit 1d determines whether the sign candidate 31 can be determined. Since the calculated distance from the vehicle V for the sign candidate 31 in scene 1 is 12 m, and the number of time-series tracking results is 3, it is determined that the determination is possible in light of the above-described two conditions of step S4, and the process proceeds to step S7. .
　ステップＳ７では、標識判定部１ｄは、標識候補３１が道路標識に該当するか判定する。ここでは、標識候補３１が速度制限８０ｋｐｈを示す道路標識と判定されたものとして、ステップＳ８に移行する。 In step S7, the sign determination unit 1d determines whether the sign candidate 31 corresponds to a road sign. Here, it is determined that the sign candidate 31 is a road sign indicating the speed limit of 80 kph, and the process proceeds to step S8.
　ステップＳ８では、標識判定部１ｄは、道路標識と判定された標識候補３１を通知リストＬ２に登録する。その後、ステップＳ９に移行する。 In step S8, the sign determination unit 1d registers the sign candidates 31 determined as road signs in the notification list L2. After that, the process proceeds to step S9.
　ステップＳ９では、標識判定部１ｄは、標識候補３１に対する再判定要否を判定する。
ここでは、標識候補３１は道路標識と判定済であり、再判定は不要なので、ステップＳ１０に移行する。 In step S<b>9 , the sign determination unit 1 d determines whether or not the candidate sign 31 needs to be re-determined.
Here, the sign candidate 31 has already been judged to be a road sign, and re-judgment is unnecessary, so the process proceeds to step S10.
　ステップＳ１０では、標識判定部１ｄは、標識候補３１に対して判定済フラグを設定する。以上で、追跡リストＬ１中の標識候補３１に対する標識判定部１ｄの処理は終了する。 In step S10, the sign determination unit 1d sets a judged flag for the sign candidate 31. Thus, the processing of the sign determination unit 1d for the sign candidates 31 in the tracking list L1 is completed.
　＜＜誤った算出距離の標識候補３２に対する処理＞＞
　次に、追跡リストＬ１中の標識候補３２に対する標識判定部１ｄの処理を実施する。 <<Processing for Sign Candidate 32 with Wrong Calculated Distance>>
Next, the processing of the sign determination unit 1d for the sign candidates 32 in the tracking list L1 is performed.
　ステップＳ１、Ｓ２、Ｓ３では、標識判定部１ｄは、標識候補３１に対する処理と同様に、標識候補３２について、判定済フラグの有無の確認と、併設可能性の有無の判定を行った後、表１に示す標識候補３２の素性に基づき、併設可能性のある標識であることを示す併設可能性フラグを標識候補３２に設定する。その後、ステップＳ４へと移行する。 In steps S1, S2, and S3, the sign determination unit 1d confirms whether or not there is a determined flag for the sign candidate 32, and determines whether or not there is a possibility that the sign candidate 32 may be placed together. 1, a joint possibility flag is set for the sign candidate 32, which indicates that the sign can be jointly arranged. After that, the process proceeds to step S4.
　ステップＳ４では、標識判定部１ｄは、標識候補３２の判定可否を判定する。表１に示すように、車両Ｖから標識候補３２までの実際の距離は１２ｍであるため、本来であれば、併設標識である標識候補３１と標識候補３２は、ステップＳ４の距離条件を同じタイミングで満たすはずであるが、表１では標識候補３２までの算出距離が３２ｍと誤っており、１２．４５ｍ以下というステップＳ４の距離条件を満たしていないため、標識判定部１ｄは、未だ標識候補３２の判定不可と判定して、ステップＳ５に移行する。 In step S4, the sign determination unit 1d determines whether the sign candidate 32 can be determined. As shown in Table 1, the actual distance from the vehicle V to the sign candidate 32 is 12 m. However, in Table 1, the calculated distance to the sign candidate 32 is erroneously 32 m. is determined to be undeterminable, and the process proceeds to step S5.
　ステップＳ５では、標識判定部１ｄは、通知リストＬ２中に併設可能性フラグが立った他の標識候補が登録されているかを確認する。本例では、標識候補３１に対するステップＳ７，Ｓ８の処理により、通知リストＬ２中の標識候補３１には併設可能性フラグが立っているため、ステップＳ６へと移行する。 In step S5, the sign determination unit 1d confirms whether or not another sign candidate with a joint possibility flag set is registered in the notification list L2. In this example, as a result of the processing of steps S7 and S8 for the candidate sign 31, the possibility flag is set for the candidate sign 31 in the notification list L2, so the process proceeds to step S6.
　ステップＳ６では、標識判定部１ｄは、標識候補３２に対して、同時判定フラグを設定し、ステップＳ７に移行する。 In step S6, the sign determination unit 1d sets a simultaneous determination flag for the sign candidate 32, and proceeds to step S7.
　ステップＳ７、Ｓ８では、標識判定部１ｄは、標識候補３２が道路標識に該当するかを判定する。本例では、標識候補３２が速度制限１００ｋｐｈを示す道路標識と判定されるので、標識候補３２を通知リストＬ２に登録する。その後、ステップＳ９に移行する。 In steps S7 and S8, the sign determination unit 1d determines whether the sign candidate 32 corresponds to a road sign. In this example, the sign candidate 32 is determined to be a road sign indicating a speed limit of 100 kph, so the sign candidate 32 is registered in the notification list L2. After that, the process proceeds to step S9.
　ステップＳ９、Ｓ１０では、標識判定部１ｄは、標識候補３２が道路標識と判定済であるため、標識候補３２に対し、判定済フラグを設定する。これで、追跡リストＬ１中の標識候補３２に対する標識判定処理は終了となる。 In steps S9 and S10, the sign determination unit 1d sets a determination flag for the sign candidate 32 because the sign candidate 32 has already been determined to be a road sign. This completes the sign determination processing for the sign candidates 32 in the tracking list L1.
　上記した標識判定部１ｄの処理フローにより、標識候補３２までの算出距離が誤っていた場合であっても、標識候補３１と標識候補３２を共に通知リストＬ２に登録できることになる。 According to the processing flow of the sign determination unit 1d described above, even if the calculated distance to the sign candidate 32 is incorrect, both the sign candidate 31 and the sign candidate 32 can be registered in the notification list L2.
　以上の処理によれば、算出距離の誤りにより、ステップＳ４では、判定可能条件を満たしていないと判定される標識候補３２であっても、ステップＳ２、Ｓ３、Ｓ５、Ｓ６の処理によって、標識候補３１の併設標識とみなされ、標識候補３１と同じタイミングで通知可否を判定することにより、通知リストＬ２への登録タイミングを揃えることができた。
これが本実施例の標識認識装置１００の作用となる。これにより、標識選出部１ｅには、標識候補３１、３２の両方の標識情報を同じタイミングで出力することができる。 According to the above process, even if the sign candidate 32 is determined not to satisfy the judgment condition in step S4 due to an error in the calculated distance, the sign candidate 31, and by judging whether or not notification can be made at the same timing as the sign candidate 31, registration timing in the notification list L2 can be aligned.
This is the action of the sign recognition device 100 of this embodiment. As a result, the sign information of both of the sign candidates 31 and 32 can be output to the sign selection unit 1e at the same timing.
　＜標識選出部１ｅ＞
　標識選出部１ｅでは、出力装置２０の標識情報報知形態に合わせて、ドライバーに報知する標識情報を選出する。例えば、出力装置２０として車両Ｖの液晶モニタがあり、複数標識を表示可能な報知形態であれば、図４Ｂ（ａ）に示すように、標識候補３１から認識した速度制限８０ｋｐｈと標識候補３２から認識した速度制限１００ｋｐｈの両方の標識情報を選出しても良いし、一つの標識情報のみがメータ表示可能な標識情報報知形態であれば、あらかじめ定められた規則に従って標識候補３１と標識候補３２のどちらかの標識情報のみを選出しても良い。例えば、車両Ｖが小型車であれば、図４Ｂ（ｂ）に示すように、速い速度制限１００ｋｐｈのみを表示しても良いし、車両Ｖが大型車であれば、図４Ｂ（ｃ）に示すように、遅い速度制限８０ｋｐｈのみを表示しても良い。 <Marker selection unit 1e>
The sign selection unit 1e selects sign information to be notified to the driver according to the sign information reporting form of the output device 20. FIG. For example, if there is a liquid crystal monitor of the vehicle V as the output device 20, and if it is a notification form that can display a plurality of signs, as shown in FIG. Both of the recognized speed limit 100 kph sign information may be selected, or if only one sign information is in the form of sign information that can be displayed on a meter, the sign candidate 31 and the sign candidate 32 according to a predetermined rule. Either label information may be selected. For example, if the vehicle V is a small vehicle, as shown in FIG. 4B(b), only the fast speed limit of 100 kph may be displayed, and if the vehicle V is a large vehicle, as shown in FIG. may display only the slow speed limit of 80 kph.
　以上のように、シーン１では、併設標識に対して判定タイミングを揃えることで、出力装置２０の標識情報報知形態に合わせた標識情報の報知が可能となった。 As described above, in scene 1, it is possible to notify the sign information in accordance with the sign information notification form of the output device 20 by aligning the judgment timings for the attached signs.
　＜シーン２＞
　シーン２は、図５Ａの画像Ｐ（左画像ＰＬまたは右画像ＰＲ）と、図５Ｂの上面図に示すように、車種を問わない速度制限標識である標識候補３１、３３が進行方向に一定距離離れて設置された環境である。この環境において、標識追跡部１ｃにより、追跡リストＬ１には、標識候補３１、３３の双方が登録済みであるものとして、図３のフローチャートの説明を行う。 <Scene 2>
In scene 2, as shown in the image P (left image P L or right image P R ) of FIG. 5A and the top view of FIG. It is an environment placed at a certain distance. In this environment, the flowchart of FIG. 3 will be described assuming that both of the candidate markers 31 and 33 have been registered in the tracking list L1 by the marker tracking unit 1c.
　この環境下では、標識候補３１、３３は併設標識ではないため、まず、標識候補３１のみに対して標識判定部１ｄ、標識認識結果Ｒを保存し、ＣＡＮを介して出力装置２０によってドライバーに報知する。その後、標識候補３３に対しても同様に処理を行い出力装置２０によってドライバーに報知されることが期待する動作となる。標識候補３１の処理はシーン１と同一のため割愛し、以下では、標識候補３３の処理を詳細に説明する。 Under this environment, since the sign candidates 31 and 33 are not concurrent signs, the sign determination unit 1d and the sign recognition result R are stored only for the sign candidate 31, and the driver is notified by the output device 20 via CAN. do. After that, the same process is performed on the candidate sign 33, and the operation is expected to be notified to the driver by the output device 20. FIG. Since the processing of the sign candidate 31 is the same as that of the scene 1, the processing of the sign candidate 33 will be described in detail below.
　＜＜正しい算出距離の標識候補３３に対する処理＞＞
　ステップＳ１、Ｓ２、Ｓ３では、標識判定部１ｄは、標識候補３１に対する処理と同様に、追跡リストＬ１中の標識候補３３について、判定済フラグの有無の確認と、併設可能性の有無の判定を行った後、表１に示す標識候補３３の素性に基づき、併設可能性のある標識であることを示す併設可能性フラグを標識候補３３に設定する。その後、ステップＳ４へと移行する。 <<Process for Sign Candidate 33 with Correct Calculated Distance>>
In steps S1, S2, and S3, the sign determination unit 1d confirms whether or not a flag candidate 33 in the tracking list L1 has a determined flag, and determines whether or not the candidate sign 33 in the tracking list L1 has a flag, and determines whether the candidate sign 33 can be placed together. After that, based on the feature of the candidate sign 33 shown in Table 1, a joint possibility flag is set in the candidate sign 33 to indicate that the sign has a possibility of being jointly arranged. After that, the process proceeds to step S4.
　ステップＳ４では、標識判定部１ｄは、標識候補３３の判定可否を判定する。表１に示すように、車両Ｖから標識候補３３までの算出距離が３５ｍとなり、１２．４５ｍ以下である条件を満たしていないため、ステップＳ５の処理に移行する。 In step S4, the sign determination unit 1d determines whether the sign candidate 33 can be determined. As shown in Table 1, the calculated distance from the vehicle V to the sign candidate 33 is 35 m, which does not satisfy the condition of 12.45 m or less, so the process proceeds to step S5.
　ステップＳ５では、標識判定部１ｄは、通知リストＬ２中に併設可能性フラグが立った他の標識候補が登録されているかを確認する。本例では、標識候補３１に対するステップＳ７，Ｓ８の処理により、通知リストＬ２中の標識候補３１には併設可能性フラグが立っているため、ステップＳ６へと移行する。なお、通知リストＬ２に併設可能性フラグを伴う標識候補が無かった場合は、標識候補３３への標識判定処理は終了する。 In step S5, the sign determination unit 1d confirms whether or not another sign candidate with a joint possibility flag set is registered in the notification list L2. In this example, as a result of the processing of steps S7 and S8 for the candidate sign 31, the possibility flag is set for the candidate sign 31 in the notification list L2, so the process proceeds to step S6. Note that if there is no sign candidate with the flag of possibility of being added together in the notification list L2, the sign determination processing for the sign candidate 33 ends.
　ステップＳ６では、標識判定部１ｄは、標識候補３３に対して、同時判定フラグを設定し、ステップＳ７に移行する。 In step S6, the sign determination unit 1d sets a simultaneous determination flag for the sign candidate 33, and proceeds to step S7.
　ステップＳ７では、標識判定部１ｄは、標識候補３３が道路標識に該当するかを判定する。本例では、標識候補３３が速度制限１００ｋｐｈを示す道路標識であるが、標識候補３１より遠方の標識候補３３では、識別結果が安定しないため、道路標識であるとは判定されず、ステップＳ９に移行する。 In step S7, the sign determination unit 1d determines whether the sign candidate 33 corresponds to a road sign. In this example, the sign candidate 33 is a road sign indicating a speed limit of 100 kph, but since the identification result is not stable for the sign candidate 33 farther from the sign candidate 31, it is not determined to be a road sign, and the process proceeds to step S9. Transition.
　ステップＳ９では、標識判定部１ｄは、標識候補３３に対する再判定の要否を確認する。本例の標識候補３３は、ステップＳ７で道路標識に該当すると判定されず、また、ステップＳ６で同時判定フラグが設定された標識候補である。そのため、上記したステップＳ９の判定基準に照らし、標識候補３３は再判定が必要な標識候補と判断され、ステップＳ１０を実行せずに、標識候補３３に対する標識判定処理を終了する。 In step S9, the sign determination unit 1d confirms whether or not the sign candidate 33 needs to be re-determined. The sign candidate 33 of this example is a sign candidate for which it was not determined to be a road sign in step S7 and the simultaneous determination flag was set in step S6. Therefore, the sign candidate 33 is judged to be a sign candidate that needs to be re-determined in light of the determination criteria of step S9 described above, and the sign determination processing for the sign candidate 33 is terminated without executing step S10.
　上記した処理フローにより、標識候補３１は道路標識として通知リストＬ２に登録されたが、この時点では、標識候補３３は通知リストＬ２には登録されていない。そのため、標識選出部１ｅには、標識候補３１の標識情報（速度制限８０ｋｐｈ）のみが入力され、まずはドライバーに標識候補３１の標識情報が報知される（図５Ｂ（ａ）参照）。 According to the processing flow described above, the sign candidate 31 is registered as a road sign in the notification list L2, but at this point the sign candidate 33 is not registered in the notification list L2. Therefore, only the sign information (speed limit of 80 kph) of the sign candidate 31 is input to the sign selection unit 1e, and the sign information of the sign candidate 31 is first notified to the driver (see FIG. 5B(a)).
　その後も、判定済フラグが未設定の標識候補３３に対する処理が繰り返され、算出した進行方向距離がステップＳ４の要件を満たした時点で、標識候補３３が道路標識と判定されることになる。これにより、その後は、標識候補３３の標識情報（速度制限１００ｋｐｈ）のみが標識選出部１ｅに渡され、ドライバーに報知されることになる（図５Ｂ（ｂ）参照）。 After that, the processing is repeated for the sign candidate 33 for which the judged flag is not set, and when the calculated traveling direction distance satisfies the requirements of step S4, the sign candidate 33 is judged to be a road sign. As a result, thereafter, only the sign information (speed limit of 100 kph) of the sign candidate 33 is passed to the sign selection unit 1e and notified to the driver (see FIG. 5B(b)).
　車両Ｖから標識候補３０までの算出距離条件を無視して複数標識を同時に判定する場合、シーン２では、本来別々のタイミングで判定するべき標識候補３１，３３を同時に判定してしまう可能性がある。しかし、本実施例では、判定済フラグを考慮することで、シーン２のような環境下においても、標識候補３１，３３を異なるタイミングで判定することができるため、ドライバーに適切な標識情報を正しい順序で報知することが可能である。 If a plurality of signs are judged at the same time ignoring the calculated distance condition from the vehicle V to the sign candidate 30, in the scene 2, the sign candidates 31 and 33 that should be judged at different timings may be judged at the same time. . However, in the present embodiment, by taking into account the determination flag, it is possible to determine the traffic sign candidates 31 and 33 at different timings even in an environment such as scene 2. It is possible to report in order.
　以上で説明したように、本実施例の標識認識装置によれば、複数の速度制限標識が併設されている場合、併設標識を同一タイミングで認識し、自車用の規制速度情報を漏らさず出力することができる。 As described above, according to the traffic sign recognition apparatus of this embodiment, when a plurality of speed limit traffic signs are installed side by side, the speed limit traffic signs are recognized at the same timing, and the speed limit information for the own vehicle is output without omission. can do.
１００…標識認識装置、１０…　カメラ装置、１１…カメラ、１１Ｌ…左カメラ、１１Ｒ…右カメラ、１２…画像入力インタフェース、１３…画像処理部、１４…演算処理部、　１…標識認識部、１ａ…標識検知部、１ｂ…標識識別部、１ｃ…標識追跡部、１ｄ…標識判定部、１ｅ…標識選出部、１５…記憶部、１６…ＣＡＮインタフェース、１７…制御処理部、２０…出力装置、３０～３３…標識候補、Ｐ…画像、ＰＬ…左画像、ＰＲ…右画像、Ｌ１…追跡リスト、Ｌ２…通知リスト DESCRIPTION OF SYMBOLS 100... Sign recognition apparatus, 10... Camera apparatus, 11... Camera, 11L... Left camera, 11R... Right camera, 12... Image input interface, 13... Image processing part, 14... Operation process part, 1... Sign recognition part, 1a Sign detection unit 1b Sign identification unit 1c Sign tracking unit 1d Sign determination unit 1e Sign selection unit 15 Storage unit 16 CAN interface 17 Control processing unit 20 Output device 30 to 33...sign candidate, P...image, PL...left image, PR ...right image, L1 ...tracking list, L2...notification list

Claims (8)

  1. 　車両に搭載したカメラ装置が撮像した画像から標識候補を検知する標識検知部と、
    　前記標識候補の種別を識別する標識識別部と、
    　連続撮像した複数の画像内の識別済み標識候補を対応付けて追跡する標識追跡部と、
    　追跡中の前記標識候補までの進行方向距離が所定距離内である場合に、前記標識候補が道路標識に該当するかを判定する標識判定部と、
    　を有した標識認識装置であって、
    　前記標識追跡部が第一標識候補と第二標識候補を追跡している場合、
    　前記標識判定部は、前記第一標識候補が道路標識と判定済みであれば、前記第二標識候補が前記所定距離内にない場合であっても、前記第二標識候補が道路標識に該当するかを判定することを特徴とする標識認識装置。     a sign detection unit that detects a sign candidate from an image captured by a camera device mounted on a vehicle;
    a sign identification unit that identifies the type of the sign candidate;
    a sign tracking unit that tracks identified sign candidates in a plurality of continuously captured images in association with each other;
    a sign determination unit that determines whether the sign candidate corresponds to a road sign when the traveling direction distance to the sign candidate being tracked is within a predetermined distance;
    A sign recognition device having
    When the marker tracking unit is tracking the first marker candidate and the second marker candidate,
    The sign determination unit determines that the second sign candidate corresponds to a road sign even if the second sign candidate is not within the predetermined distance if the first sign candidate has already been judged to be a road sign. A sign recognition device characterized by determining whether
  2. 　請求項１に記載の標識認識装置において、
    　前記標識判定部が前記第二標識候補を道路標識でないと判定した後、前記第二標識候補が前記所定距離内となったときに、前記第二標識候補が道路標識に該当するかを再判定することを特徴とする標識認識装置。     The sign recognition device according to claim 1,
    After the sign determination unit determines that the second sign candidate is not a road sign, it is re-determined whether the second sign candidate corresponds to a road sign when the second sign candidate is within the predetermined distance. A sign recognition device characterized by:
  3. 　請求項１に記載の標識認識装置において、
    　さらに、前記標識判定部が複数の道路標識を判定した場合に、少なくとも、自車が遵守すべき標識情報を選出する標識選出部を有することを特徴とする標識認識装置。     The sign recognition device according to claim 1,
    The sign recognition apparatus further comprises a sign selection section that selects at least sign information to be observed by the vehicle when the sign determination section determines a plurality of road signs.
  4. 　請求項３に記載の標識認識装置において、
    　前記標識選出部は、出力装置の報知形態に合わせて標識情報を選出することを特徴とする標識認識装置。     In the sign recognition device according to claim 3,
    The sign recognition device, wherein the sign selection unit selects the sign information in accordance with the reporting form of the output device.
  5. 　請求項４に記載の標識認識装置において、
    　前記出力装置は、前記標識判定部が判定した全ての標識情報を表示するモニタであることを特徴とする標識識別装置。     In the sign recognition device according to claim 4,
    A sign identification device, wherein the output device is a monitor that displays all the sign information determined by the sign determination unit.
  6. 　請求項４に記載の標識認識装置において、
    　前記出力装置は、前記標識判定部が判定した標識情報のうち自車が遵守すべき標識情報を表示するモニタであることを特徴とする標識識別装置。     In the sign recognition device according to claim 4,
    A traffic sign identification device, wherein the output device is a monitor that displays traffic sign information to be observed by the own vehicle among the traffic sign information determined by the traffic sign determination unit.
  7. 　請求項４に記載の標識認識装置において、
    　前記出力装置は、安全運転支援システムまたは自動運転システムであることを特徴とする標識識別装置。     In the sign recognition device according to claim 4,
    The sign identification device, wherein the output device is a safe driving support system or an automatic driving system.
  8. 　車両に搭載したカメラ装置が撮像した画像から標識候補を検知するステップと、
    　前記標識候補の種別を識別するステップと、
    　連続撮像した複数の画像内の識別済み標識候補を対応付けて追跡するステップと、
    　追跡中の前記標識候補までの進行方向距離が所定距離内である場合に、前記標識候補が道路標識に該当するかを判定するステップと、を有し、
    　第一標識候補と第二標識候補を追跡している場合、前記第一標識候補が道路標識と判定済みであれば、前記第二標識候補が前記所定距離内にない場合であっても、前記第二標識候補が道路標識に該当するかを判定することを特徴とする標識認識方法。     a step of detecting a sign candidate from an image captured by a camera device mounted on a vehicle;
    identifying a type of the candidate sign;
    correlating and tracking identified sign candidates in a plurality of consecutively captured images;
    determining whether the candidate sign corresponds to a road sign when the traveling direction distance to the candidate sign being tracked is within a predetermined distance;
    When the first sign candidate and the second sign candidate are tracked, if the first sign candidate has already been determined to be a road sign, even if the second sign candidate is not within the predetermined distance, the A sign recognition method, comprising determining whether a second sign candidate corresponds to a road sign.
PCT/JP2022/004107 2021-04-26 2022-02-02 Sign recognition device and sign recognition method WO2022230283A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
DE112022000874.1T DE112022000874T5 (en) 2021-04-26 2022-02-02 CHARACTER RECOGNITION DEVICE AND CHARACTER RECOGNITION METHOD
JP2023517058A JP7564946B2 (en) 2021-04-26 2022-02-02 Sign recognition device and sign recognition method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021073975 2021-04-26
JP2021-073975 2021-04-26

Publications (1)

Publication Number Publication Date
WO2022230283A1 true WO2022230283A1 (en) 2022-11-03

Family

ID=83848247

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/004107 WO2022230283A1 (en) 2021-04-26 2022-02-02 Sign recognition device and sign recognition method

Country Status (3)

Country Link
JP (1) JP7564946B2 (en)
DE (1) DE112022000874T5 (en)
WO (1) WO2022230283A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014522049A (en) * 2011-08-04 2014-08-28 コンティ テミック マイクロエレクトロニック ゲゼルシャフト ミット ベシュレンクテル ハフツング Traffic sign recognition method
JP2017117041A (en) * 2015-12-22 2017-06-29 本田技研工業株式会社 Device and method of displaying signs for vehicle
JP2019203750A (en) * 2018-05-22 2019-11-28 三菱電機株式会社 Vehicle position correcting device, navigation system, and vehicle position correcting program
JP2021056575A (en) * 2019-09-27 2021-04-08 スズキ株式会社 Vehicle drive support system

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6292167B2 (en) 2015-05-11 2018-03-14 トヨタ自動車株式会社 Vehicle regulated speed display device, vehicle regulated speed display method, and vehicle equipped with regulated speed display device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014522049A (en) * 2011-08-04 2014-08-28 コンティ テミック マイクロエレクトロニック ゲゼルシャフト ミット ベシュレンクテル ハフツング Traffic sign recognition method
JP2017117041A (en) * 2015-12-22 2017-06-29 本田技研工業株式会社 Device and method of displaying signs for vehicle
JP2019203750A (en) * 2018-05-22 2019-11-28 三菱電機株式会社 Vehicle position correcting device, navigation system, and vehicle position correcting program
JP2021056575A (en) * 2019-09-27 2021-04-08 スズキ株式会社 Vehicle drive support system

Also Published As

Publication number Publication date
JPWO2022230283A1 (en) 2022-11-03
DE112022000874T5 (en) 2023-11-30
JP7564946B2 (en) 2024-10-09

Similar Documents

Publication Publication Date Title
US20170253177A1 (en) Vehicle lighting system
US20160350606A1 (en) Driving assistance apparatus and driving assistance method
US11074818B2 (en) Drive assist device and drive assist method
EP2402923A1 (en) Vehicle-mounted information processing apparatus and information processing method
EP1596322A2 (en) Driving lane recognizer and driving lane recognizing method
US9352746B2 (en) Lane relative position estimation method and system for driver assistance systems
US20140365109A1 (en) Apparatus and method for recognizing driving lane
JP2007178271A (en) Own position recognition system
JP2013097714A (en) Lane recognition device
EP3451311A1 (en) Method and device for parking assistance
US9835468B2 (en) Entry detection apparatus and entry detection method
CN106184232A (en) A kind of lane departure warning control method based on driver visual angle
JP2016024777A (en) Traffic lane detector and traffic lane detection method
JP4951481B2 (en) Road marking recognition device
CN115123218B (en) Vehicle detection method and device and electronic equipment thereof
US9177040B2 (en) Information processing device for vehicle and database
US11908205B2 (en) Method and system for increasing safety of partially or fully automated driving functions
JP5997962B2 (en) In-vehicle lane marker recognition device
CN112817303B (en) Self-propelled triangular warning frame and travelling control method thereof
WO2022230283A1 (en) Sign recognition device and sign recognition method
JP4899899B2 (en) Obstacle recognition apparatus and method
US10339803B2 (en) Method for operating an assistance system of a motor vehicle and assistance system
CN114407882B (en) Safety driving assisting device for vehicle
US11885640B2 (en) Map generation device and map generation method
JP2017102830A (en) Driving assist device and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22795206

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023517058

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 112022000874

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22795206

Country of ref document: EP

Kind code of ref document: A1