WO2020071132A1

WO2020071132A1 - Camera device

Info

Publication number: WO2020071132A1
Application number: PCT/JP2019/036680
Authority: WO
Inventors: 啓佑岩崎; 永崎　健; 遠藤　健
Original assignee: 日立オートモティブシステムズ株式会社
Priority date: 2018-10-04
Filing date: 2019-09-19
Publication date: 2020-04-09
Also published as: JPWO2020071132A1; CN112789654A; JP7058753B2

Abstract

The present invention provides a camera device with which it is possible to improve the accuracy of detecting a restriction removal sign. An arithmetic processing unit (search unit) searches for a diagonal line candidate from an image (diagonal line search 504). The arithmetic processing unit (selection unit) selects the diagonal line candidate of a restriction removal sign indicating the removal of restriction, the diagonal line candidate corresponding to the actual sign, from the detected diagonal line candidates (diagonal line determination 505). The arithmetic processing unit (restriction removal sign identification unit) identifies a restriction removal sign from the images of selected diagonal line candidate of the restriction removal sign (identification process 506).

Description

Camera device

The present invention relates to a camera device.

In recent years, the demand for various recognition functions for safe driving and automatic driving has been increasing due to the spread of in-vehicle camera devices. In particular, a stereo camera device measures various objects (people, cars, three-dimensional objects, road surfaces, road signs, road signs, signboard signs, etc.) around an automobile in order to simultaneously measure visual information by images and distance information to the objects. Etc.) can be understood in detail, which also contributes to improving safety during driving assistance.

標識 There is an increasing demand for sign recognition functions in Japan and overseas, and there is a need for improved recognition accuracy. One of the causes of accuracy deterioration is the difficulty in detecting and recognizing the regulation release sign. This is because it is difficult to detect, because the design of the regulation release sign is different from the speed sign etc., because the size is small and the surrounding ring is so narrow that it is difficult to obtain a sufficient contrast difference with the background. It is. As a result of not being able to detect / recognize the regulation release sign, the previous display is continued even in the situation where the regulation speed display should be terminated, thus causing a problem in providing correct speed information to the driver. .

Conventionally, various technologies and devices related to an in-vehicle camera device mounted on a vehicle and recognizing a situation in front of the vehicle have been proposed. For example, with respect to improving the performance of sign recognition, Patent Literature 1 discloses a technique that focuses on improving recognition accuracy.

JP-A-2015-191621

However, the technology disclosed in Patent Document 1 has a problem that one of the factors of accuracy deterioration is that a sign (regulation release sign) for instructing cancellation of regulation such as speed regulation is not detected.

目的 An object of the present invention has been made in view of the above points, and an object of the present invention is to provide a camera device capable of improving the detection accuracy of a regulation release sign.

In order to achieve the above object, the present invention provides a search unit that searches for a diagonal line candidate from an image, and a selection unit that selects, from the detected diagonal line candidates, a diagonal line candidate of a restriction release marker indicating release of restriction corresponding to the real sign. And a restriction release sign identifying unit that identifies the restriction release sign from the image of the hatched candidate of the selected restriction release sign.

According to the present invention, the detection accuracy of the regulation release sign can be improved. Problems, configurations, and effects other than those described above will be apparent from the following description of the embodiments.

FIG. 1 is a block diagram illustrating an overall configuration of an on-vehicle stereo camera device according to an embodiment of the present invention. It is a figure showing the basic processing flow of an in-vehicle stereo camera device. 5 is a timing chart of various processes. This is a part of the image stored in the image buffer. 4A is a processing result obtained by performing image processing on the image of FIG. 4A, and is an image having luminance in a portion where a change in shading is large. It is a figure showing the basic processing flow of sign recognition. It is a figure showing the processing flow of regulation release sign detection in an embodiment of the present invention. FIG. 6 is a diagram illustrating an outline of a process of diagonal line search illustrated in FIG. 5. It is a figure showing a continuous edge pair (left-right pair). FIG. 6 is a diagram for describing narrowing down (selection) of diagonal line candidates in the diagonal line determination shown in FIG. 5. It is a figure for explaining cooperation with other systems giving an opportunity of processing of regulation release sign detection.

実施 An embodiment of the present invention will be described below with reference to the drawings. Although partially overlapping with the object of the invention described above, the object of the present embodiment is to perform detection in accordance with the design of the regulation release in the sign detection processing, and to erroneously determine a pattern such as a tree or a telephone pole. It is an object of the present invention to provide a camera device capable of suppressing the above.

FIG. 1 is a block diagram showing an overall configuration of a vehicle-mounted stereo camera device 100 (camera device) according to the present embodiment. The in-vehicle stereo camera device 100 of the present embodiment is a device mounted on a vehicle and recognizing an environment outside the vehicle based on image information of a shooting target area in front of the vehicle. The in-vehicle stereo camera device 100 recognizes, for example, white lines on roads, pedestrians, vehicles, other three-dimensional objects, signals, signs, lighting lamps, and the like, and brakes a vehicle (own vehicle) equipped with the stereo camera device. Perform steering and other adjustments.

The in-vehicle stereo camera device 100 has two

cameras

101 and 102 arranged on the left and right for acquiring image information, and an image input interface 103 for controlling the imaging of the

cameras

101 and 102 and capturing the captured image. The image captured through this interface is transmitted through a bus 109 and processed by the image processing unit 104 and the arithmetic processing unit 105. The results during the processing and the image data as the final result are stored in the storage unit 106. Note that control lines for the

cameras

101 and 102 are not shown in FIG.

The image processing unit 104 compares the first image obtained from the image sensor of the camera 101 with the second image obtained from the image sensor of the camera 102, and determines each image based on the image sensor. Correction of a device-specific deviation and image correction such as noise interpolation are performed and stored in the storage unit 106. Further, the image processing unit 104 calculates a mutually corresponding portion between the first and second images, calculates disparity information, and stores the same in the storage unit 106 as in the above.

The arithmetic processing unit 105 uses the image and the parallax information (distance information for each point on the image) stored in the storage unit 106 to recognize various objects necessary for perceiving the environment around the vehicle. The various objects include a person, a car, other obstacles, a traffic light, a sign, a tail lamp and a head ride of a car, and the like. Some of these recognition results and intermediate calculation results are recorded in the storage unit 106 as in the previous case. After performing various object recognition processes on the captured image, the arithmetic processing unit 105 calculates the control of the vehicle using the recognition results.

The control policy of the vehicle obtained as a result of the calculation and a part of the object recognition result are transmitted to the in-vehicle network CAN (Controller Area Network) through the CAN interface 107, whereby the vehicle is braked. In addition, regarding these operations, the control processing unit 108 monitors whether or not each processing unit has performed an abnormal operation, whether or not an error has occurred during data transfer, and the like. ).

The image processing unit 104 includes, via a bus 109 (internal bus), a control processing unit 108, a storage unit 106, an arithmetic processing unit 105, and an image input interface 103 (input and output between the imaging devices of the cameras 101 and 102). ), And a CAN interface 107 (input / output unit with an external vehicle-mounted network).

The image processing unit 104, the arithmetic processing unit 105, the storage unit 106, the image input interface 103, the CAN interface 107, and the control processing unit 108 are configured by a single or a plurality of computer units. The storage unit 106 includes, for example, a memory that stores image information obtained by the image processing unit 104, image information created as a result of scanning by the arithmetic processing unit 105, and the like. The CAN interface 107 outputs information output from the in-vehicle stereo camera device 100 to another control system of the vehicle via the CAN 110.

FIG. 2 shows a processing flow of the vehicle-mounted stereo camera device 100.

The left and

right cameras

101 and 102 capture images. For each of the

image data

203 and 204 captured by each of the left and

right cameras

101 and 102, the image processing unit 104 absorbs a peculiar habit (image distortion or the like) of the image sensor of the

cameras

101 and 102. Image processing 205 such as correction is performed. The processing result is stored in the image buffer 206. The image buffer 206 is provided in the storage unit 106 in FIG.

{Circle around (2)} Using the two corrected images, the image processing unit 104 performs parallax processing (matching) between the images, thereby performing parallax processing 207 for obtaining parallax information of the images obtained by the left and right cameras. From the parallax between the left and right images, it becomes clear where and how the point of interest on the target object corresponds to the image on the left and right cameras, and the distance to the target can be obtained by the principle of triangulation. The image finally obtained by the image processing 205 and the parallax processing 207 and the parallax information are stored in the storage unit 106.

Furthermore, the arithmetic processing unit 105 performs various object recognition processes 209 using the stored image and the parallax information. The object to be recognized includes a person, a car, other three-dimensional objects, a sign, a traffic light, a tail lamp, and the like. At the time of recognition, the arithmetic processing unit 105 uses the recognition dictionary 210 as necessary.

Further, the arithmetic processing unit 105 performs the vehicle control processing 211 in consideration of the result of the object recognition and the state of the host vehicle (speed, steering angle, etc.). For example, the vehicle control process 211 issues a warning to an occupant, determines a policy of performing braking such as braking or steering angle adjustment of the own vehicle, or determines a policy of performing avoidance control of an object by that. This is the process of outputting through

The various object recognition processing 209 and the vehicle control processing 211 are performed by the arithmetic processing unit 105 of FIG. 1 as described above, and the output to the CAN is performed by the CAN interface 107. Each of these processing means is composed of, for example, a single or a plurality of computer units, and is configured to be able to exchange data with each other.

FIG. 3 shows a timing chart of the on-vehicle stereo camera device 100.

{Circle around (2)} in the timing chart of FIG. The flow of 301 indicates the processing timing in the image processing unit 104 of FIG. 1, and the flow of 302 indicates the processing timing of the arithmetic processing unit 105 in FIG. First, a right image input 303 is performed. This corresponds to a process in which the camera 102 (right camera) in FIG. 2 captures an image, passes through image processing 205, and then stores the right image in the image buffer 206 by the image processing unit 104.

Next, the left image input 304 is performed. This corresponds to the processing from capturing the image by the camera 101 (left camera) in FIG. 2, through the image processing 205, and storing the left image in the image buffer 206 by the image processing unit 104. Next, parallax processing 207 is performed. In FIG. 2, the image processing unit 104 reads out two images on the left and right from the image buffer 206, calculates the disparity by comparing the two images, and stores the disparity information obtained by the calculation in the storage unit 106. This is equivalent to the processing up to the storage in. At this point, the image and the parallax information are in the storage unit 106. Next, the arithmetic processing unit 105 performs various object recognition processing 209, performs the vehicle control processing 211, and outputs the result to the CAN.

FIGS. 4A to 4C are diagrams for explaining the processing flow of the sign recognition function.

The various object recognition processing 209 includes a sign recognition processing 209a. The sign recognition process 209a includes a circle detection process 406 and an identification process 407. FIG. 4A shows a part of the image stored in the image buffer 206 after the image processing 205. FIG. 4B is a processing result obtained by performing image processing on the image of FIG. 4A for use in the sign recognition processing 209a, and is an image having luminance in a portion where the density change is large in the vicinity. FIG. 4C is an example of a basic processing flow of sign recognition.

In the marker recognition process 209a, first, the circle detection process 406 is performed using the image of FIG. 4B. In the circle detection processing 406, the center of the circle is obtained by the center calculation. The circle detection processing 406 is divided into a center estimation step 408 and a radius estimation step 409.

First, in step 408, the center is estimated. A line segment 404 is drawn from each edge in the normal direction, and a point where intersections of the line segments overlap by a certain number or more is estimated as the center 400 (FIG. 4A). The counting of the number of edges may be omitted from the center 400 to a predetermined radius. In general, in consideration of a sign having a diameter of 30 cm or more and a sign that cannot be processed by the identification processing 407 even if a sign that is very far is detected, the calculation omission area can reduce the calculation load. .

In order to improve the security by recognizing an extremely distant sign by improving the processing capability of hardware such as a processor that performs the sign recognition process 209a, the calculation omission area may be reduced. In this example, detection of a sign in the shape of a circle has been described as an example, but signs such as a rectangle, an octagon, and a triangle can also be detected in the graphic detection processing described later.

FIG. 5 shows a processing flow for detecting a regulation release sign in the embodiment of the present invention.

The search for the regulation release sign (regulation end sign) is performed at the timing of the sign recognition processing 209a in the various object recognition processing 209. In the sign recognition process 209a, a group of sign candidates called main signs such as speed signs and overtaking prohibitions is extracted by first performing graphic detection 502 from the image buffer 501 storing the image of the target frame.

(4) For the plurality of extracted marker candidates, the respective patterns are cut out from the image and identification 503 is performed. When performing identification, the recognition dictionary 210 is used, and the type of the sign is identified by combining a weak classifier such as a binary discriminant tree and a strong classifier such as a nearest neighbor method or a discriminant function.

Here, the arithmetic processing unit 105 (processor) functions as a main marker identifying unit that identifies the main marker.

Next, if a sign that is likely to appear in a co-occurrence relationship with the regulation release sign is detected from the identified signs, as described below, an independent process suitable for detecting and determining the regulation release sign is performed. . The unique processing for the restriction release sign includes a diagonal line search 504, a diagonal line determination 505, and an identification processing 506 for the restriction release sign.

The diagonal line search 504 is a process of detecting a diagonal line portion at the center of the restriction release marker based on an edge image obtained by extracting an edge component from a frame image to be recognized. Here, the arithmetic processing unit 105 (processor) functions as a search unit that searches for an oblique line candidate from an image. By searching for a hatched line which is a characteristic design of the regulation release sign, the detection accuracy of the regulation release sign can be improved. The arithmetic processing unit 105 (search unit) searches for a diagonal line candidate, for example, in response to the identification of the real sign. Thereby, the oblique line of the regulation release sign co-occurring with the present sign can be detected in a timely manner. The specific contents of the processing are described in FIG.

The oblique line determination 505 is a process for determining, from the detected oblique line candidates, oblique lines considered to be signs and oblique lines considered to be electric wires or pillars. Here, the arithmetic processing unit 105 (processor) functions as a selection unit that selects, from the detected diagonal line candidates, a diagonal line candidate of a restriction release marker indicating release of the restriction corresponding to the real marker. This will be described later with reference to FIG.

識別 Furthermore, the identification process 506 is a process of identifying the restriction release marker from the image of the selected hatched candidate of the restriction release marker. The arithmetic processing unit 105 (processor) functions as a regulation release marker identification unit that performs the identification process 506.

FIG. 6A is a diagram showing an outline of a process of the oblique line search 504 (oblique line detection) shown in FIG.

(4) A horizontal scanning line 602 is determined for the image 601 of the target regulation release marker, and image scanning is performed in a direction 603, such as left to right. The image to be scanned is an edge image. The deregulation sign has a distinctive design in Europe, Japan and elsewhere. By observing the direction and intensity of the edge, the left edge 604 and the right edge 605 of this oblique line are determined, and an edge pair is determined by utilizing the fact that the left edge and the right edge are within a certain distance.

Here, the arithmetic processing unit 105 (search unit) scans the image in the horizontal direction, searches for a pair of a left edge and a right edge, and detects two or more pairs arranged continuously as oblique line candidates. By searching for a pair of a left edge and a right edge of at least two pixels or more, it is possible to narrow down the oblique line candidates of the restriction release sign. Note that a pair of a left edge and a right edge as oblique line candidates are continuously arranged adjacently without any gap.

The distance (horizontal length) to be determined in the edge pair is determined by a certain range from the size of the main sign which triggered the search for the regulation release sign, or the distance to the target sign in the case of a stereo camera. It is also possible to convert using.

Specifically, the storage unit 106 stores the size of the main sign and the range of the length of the hatched horizontal direction of the restriction release sign in association with each other. The arithmetic processing unit 105 (search unit) determines that the distance between the left edge and the right edge is within the range of the horizontal length of the oblique line of the restriction release marker associated with the size of the main marker. Search for a pair. Thereby, the diagonal line candidates can be narrowed down by the horizontal length of the diagonal line of the restriction release marker according to the size of the real sign.

(6) With respect to the extracted edge pairs, as shown in FIG. 6B, diagonal candidates are narrowed down by the length of the continuous edge pairs. The narrowing down (selection) of diagonal line candidates will be described with reference to FIG. The output information of the diagonal line search (diagonal line detection) includes the position, length, intensity, contrast ratio with the peripheral portion, color component, and line thickness of the edge pair.

(7) The diagonal line determination (upper limit / lower limit determination) in FIG. 7 is a process of determining a diagonal line that seems to be a design in a sign and a diagonal line component such as an electric wire or a pillar that is not related to the sign. The basic idea is determined by using the length and thickness of the edge pair, the contrast ratio with the periphery, the color, and the like.

In Fig. 7, the concept based on length is described. The length has an upper limit and a lower limit. Edge pairs exceeding the upper limit length are regarded as long diagonal lines such as telephone poles and are excluded, and edge pairs below the lower limit length are regarded as short diagonal lines such as patterns and excluded.

Specifically, the storage unit 106 stores the size of the permanent sign and the range of the length of the hatched vertical direction of the restriction release sign in association with each other. The arithmetic processing unit 105 (selection unit) selects a diagonal line candidate whose vertical length of the detected diagonal line candidate is within the range of the vertical length of the diagonal line of the restriction release marker associated with the size of the main sign. I do. Thereby, it is possible to prevent erroneous detection of the hatching of the regulation release sign.

The question is how to set the upper and lower limits. This is set based on the size of the main sign which has triggered the search for the regulation release sign. In general, the size of the sign and the size range allowed for the deregulation sign are defined by the driving region and country. Using this information, the upper and lower limits of the size permitted as the deregulation sign are determined from the size of the sign. Can be estimated. As shown in FIG. 8, information on the traveling area can be obtained in cooperation with an external position information system.

The storage unit 106 may further store information indicating the relative position of the regulation release marker with respect to the position of the main marker. In this case, the arithmetic processing unit 105 (processor) searches for a diagonal line candidate from an image area corresponding to the relative position of the restriction release marker with respect to the identified position of the official marker (for example, above the official marker in Japan). As a result, the area for searching for the hatched candidate for the regulation release sign is limited. As a result, it is possible to efficiently search for a hatched candidate for a regulation release sign.

FIG. 8 is a diagram for explaining a mechanism (configuration) for obtaining an opportunity to search for a regulation release sign in cooperation with GPS and map information.

画像 Image processing 805 is performed with the left camera 801 and the right camera 802 as starting points, and a parallax processing 807 detects a three-dimensional object on a side road such as a forest or a forest by looking at an area around the traveling road. Furthermore, the latitude and longitude information is obtained from the GPS, and the data of the border is stored inside the stereo camera, so that the current traveling area is determined by the country determination process. Further, information such as which country the vehicle is running, the current direction to which the vehicle is heading, and the running time zone is obtained from the running log 809.

If the external position information sensor has further high-definition map information or section information on road speed regulation, the appearance position of the sign and the switching position of the speed regulation can be known. Search is possible.

Although system investment costs are incurred by linking with an external location information system, for example, search according to search parameters (thickness, inclination, and contrast ratio of edge pairs) of deregulation signs with slightly different designs in Northern Europe, Eastern Europe, and Southern Europe Alternatively, the exposure control 811 can be performed.

By coordinating with more detailed map information, the co-occurrence status of the regulation release sign was used by changing the ratio of the processing time devoted to searching for edge pairs at the switching position of the appearance position of the sign and the speed section of the road Searching can be performed.

Specifically, the storage unit 106 stores information indicating the relative position of the restriction release sign with respect to the position of the main sign for each travel location area. The arithmetic processing unit 105 (search unit) searches for a diagonal line candidate from the area of the image corresponding to the relative position of the regulation release sign for each travel location area with respect to the position of the identified real sign. This limits the area in which to search for a diagonal line candidate for a regulation release sign according to the traveling area. As a result, it is possible to efficiently search for a hatched candidate for a regulation release sign.

For example, in Europe, the sign and the deregulation sign do not co-occur, and the location of the sign is different from the place of the deregulation sign. When the length of the section between the main sign and the restriction release sign can be acquired, the arithmetic processing unit 105 (search unit) determines the distance from the area of the image corresponding to the position (forward) separated by the length of the section from the main sign. You may search for a diagonal line candidate.

As described above, according to the present embodiment, the detection accuracy of the regulation release sign can be improved.

The present invention is not limited to the above-described embodiment, but includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the described configurations.

では In the above embodiment, a stereo camera has been described, but the number of cameras is arbitrary.

The above-described configurations, functions, and the like may be partially or entirely realized by hardware, for example, by designing an integrated circuit. In addition, the above-described configurations, functions, and the like may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

Note that the embodiment of the present invention may be the following aspects.

(1). A circular object information acquisition unit that obtains information on a circular object from an image (circle detection unit), an identification unit that identifies a first sign for the circular object obtained by the circular object information acquisition unit, and the circular object from the image A search unit that searches for a tilted object around the object, information on the tilted object obtained by the search unit, and information on the first sign obtained by the identification unit, based on the second sign related to the tilted object A camera device comprising: a determination unit that performs a determination.

(2). In (1), the front search unit searches for a pair of a left edge and a right edge in a horizontal direction of the image, and the pair of the left edge and the right edge forms a slope that is continuous in a vertical direction. In this case, the determination unit determines the second marker with respect to the inclined object based on an upper limit and a lower limit corresponding to the first marker with respect to the circular object.

(3). The camera device according to (1) or (2), wherein the search for the restriction release sign is performed by performing the search processing for the restriction release sign in cooperation with not only the camera image but also external GPS information, map information, and the like. Is set.

100 vehicle

stereo camera devices

101, 102 camera 103 image input interface 104 image processing unit 105 arithmetic processing unit 106 storage unit 107 CAN interface 108 control processing unit 109

buses

203 and 204 image data 205 Image processing 206 Image buffer 207 Parallax processing 209 Various object recognition processing 209a Mark recognition processing 210 Recognition dictionary 211 Vehicle control processing 303 Right image input 304 Left image input 400 Center 404 Line segment 406 Circle Detection process 407 Identification process 408 Center estimation step 409 Radius estimation step 501 Image buffer 502 Graphic detection 503 Identification 504 Oblique line search 505 Oblique line determination 506 Identification process 601 Image example 602 of restriction release sign … Scan line 603… one direction 604… left edge 605 ... right edge 801 ... left camera 802 ... right camera 805 ... image processing 807 ... parallax processing 809 ... traveling log 811 ... exposure control

Claims

A search unit for searching for oblique line candidates from the image,
A selection unit that selects a diagonal line candidate of a restriction release sign indicating the release of restriction corresponding to the main sign from the detected diagonal line candidate,
A restriction release sign identifying unit that identifies the restriction release sign from the image of the hatched candidate of the selected restriction release sign,
A camera device comprising:
The camera device according to claim 1,
The search unit,
Scan the image horizontally to search for a pair of left and right edges,
2. A camera device, wherein two or more pairs arranged consecutively are detected as the oblique line candidates.
The camera device according to claim 2, wherein
Further comprising a storage unit that stores the size of the main sign and the range of the length of the oblique line of the restriction release sign in the vertical direction,
The selection unit includes:
Selecting the diagonal line candidate whose detected vertical length of the diagonal line candidate is within the range of the vertical length of the diagonal line of the restriction release marker associated with the size of the official sign. Camera device.
The camera device according to claim 3,
The storage unit,
The size of the main sign and the range of the length of the hatched horizontal direction of the regulation release sign are stored in association with each other,
The search unit,
The pair of the left edge and the right edge in which a distance between the left edge and the right edge is within a range of a horizontal length of a diagonal line of the restriction release sign associated with the size of the main sign. A camera device characterized by searching.
The camera device according to claim 3,
Further comprising a main sign identification unit for identifying the main sign,
The storage unit,
Further storing information indicating the relative position of the regulation release marker with respect to the position of the main sign,
The search unit,
A camera device, wherein the diagonal line candidate is searched for from a region of the image corresponding to a relative position of the restriction release marker with respect to a position of the identified real marker.
The camera device according to claim 5, wherein
The search unit,
A camera device that searches for the diagonal line candidate in response to the identification of the real sign.
The camera device according to claim 5, wherein
The storage unit,
Storing information indicating the relative position of the regulation release sign with respect to the position of the main sign for each traveling area,
The search unit,
A camera device, wherein the diagonal line candidate is searched from a region of the image corresponding to a relative position of the restriction release marker for each of the travel destination areas with respect to a position of the identified main sign.