JP6789151B2 - Camera devices, detectors, detection systems and mobiles - Google Patents

Description

The present invention relates to camera devices, detection devices, detection systems and mobile objects.

A detection device capable of detecting a recognition object such as a pedestrian or a vehicle from an image captured by a camera and issuing an alarm is known (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 11-215487

In a device that detects an object using a camera image outdoors, such as a detection device mounted on a moving body such as a vehicle, deposits such as mud or raindrops may adhere to the lens of the camera. Even when the camera is placed inside a moving object and the outside is imaged through the window, deposits such as mud or raindrops may adhere to the window in the field of view of the camera. In such a case, if the deposits are reflected in the captured image, a part of the deposits may be erroneously recognized as a recognition target such as a vehicle or a pedestrian.

If the deposit is mistakenly recognized as a recognition target, an unnecessary alarm is frequently generated for the user, which may cause annoyance. If the object to be recognized is not recognized and the necessary alarm is not provided to the user, the function as an alarm device may not be fully exerted.

Therefore, an object of the present invention made by paying attention to these points is to provide a camera device, a detection device, a detection system, and a moving body having improved recognition accuracy of a recognition target object in the presence of deposits.

The camera device according to the embodiment of the present disclosure includes a camera and a controller. The camera is attached to the moving body and images the periphery of the moving body. The controller recognizes the deposits attached to the translucent member exposed to the outside of the moving body in the captured image from the camera, and detects the first region on the captured image in which the deposits are located. A second process is performed to detect a portion of the captured image as a detection image based on the shape or feature of the recognition target. The controller generates a signal based on the result of the first process and the result of the second process. The controller has a plurality of algorithms for detecting the detected image from the captured image in the second process, and uses different algorithms in the first region and the second region, which is a region other than the first region of the captured image. Recognize the object to be recognized

The detection device according to the embodiment of the present disclosure includes an input unit and a controller. The input unit receives an image captured by a camera attached to the moving body and images the surroundings of the moving body. The controller recognizes the deposits attached to the translucent member exposed to the outside of the moving body in the captured image from the camera received by the input unit, and detects the first region on the captured image in which the deposits are located. The first process and the second process of detecting the detected image based on the shape or feature of the recognition object are performed. The controller generates a signal based on the result of the first process and the result of the second process. The controller has a plurality of algorithms for detecting the detected image from the captured image in the second process, and uses different algorithms in the first region and the second region, which is a region other than the first region of the captured image. To recognize the object to be recognized.

The detection system according to the embodiment of the present disclosure includes a camera and a detection device. The camera is attached to the moving body and images the periphery of the moving body. The detection device includes an input unit and a controller. The input unit receives the captured image from the camera. The controller recognizes the deposits attached to the translucent member exposed to the outside of the moving body in the captured image from the camera received by the input unit, and detects the first region on the captured image in which the deposits are located. The first process and the second process of detecting the detected image based on the shape or feature of the recognition object are performed. The controller generates a signal based on the result of the first process and the result of the second process. The controller has a plurality of algorithms for detecting the detected image from the captured image in the second process, and uses different algorithms in the first region and the second region, which is a region other than the first region of the captured image. To recognize the object to be recognized.

The moving body according to the embodiment of the present disclosure includes a camera device. The camera device includes a camera and a controller that image the surroundings. The controller recognizes the deposits attached to the translucent member exposed to the outside of the moving body in the captured image from the camera received by the input unit, and detects the first region on the captured image in which the deposits are located. The first process is performed, and the second process is performed to detect the detected image based on the shape or feature of the object to be recognized. The controller generates a signal based on the result of the first process and the result of the second process. The controller has a plurality of algorithms for detecting the detected image from the captured image in the second process, and uses different algorithms in the first region and the second region, which is a region other than the first region of the captured image. To recognize the object to be recognized.

According to the present invention, it is possible to improve the recognition accuracy of the recognition target object when there is an deposit.

It is a block diagram which shows the schematic structure of the camera apparatus which concerns on one of a plurality of embodiments. It is a figure which looked at the vehicle equipped with the detection system of FIG. 1 from the left side. It is a block diagram which shows the schematic structure of the camera apparatus which concerns on another one of a plurality of embodiments. This is the first example of the functional block diagram of the controller of FIG. It is a figure which shows the example of the 1st image imaged by a camera. It is a figure which shows the 1st example of the 2nd image obtained by converting the 1st image. This is the first example of a flowchart showing the processing executed by the controller. It is a figure explaining the detection of the detection image from the 2nd image of FIG. It is a figure which shows the 2nd example of the 2nd image obtained by converting the 1st image. This is the second example of the flowchart showing the process executed by the controller. This is a third example of a flowchart showing the processing executed by the controller. This is the second example of the functional block diagram of the controller of FIG. This is a fourth example of a flowchart showing the processing executed by the controller. It is a 3rd example of the functional block diagram of the controller of FIG. It is a figure explaining the process by the 3rd example of the functional block diagram of FIG. This is a fifth example of a flowchart showing the processing executed by the controller.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The detection system 10 according to one of the plurality of embodiments will be described with reference to FIG.

The detection system 10 includes a camera 20 and a detection device 30. The camera 20 and the detection device 30 can transmit and receive information via, for example, a network 15. The network 15 may include, for example, wireless, wired, CAN (Controller Area Network), or the like.

As shown in FIG. 2, the detection system 10 can be mounted in the moving body 50. The detection device 30 can transmit and receive information via the network 15 with the alarm device 40 and / or the ECU (Electronic Control Unit) 45 or the like in the mobile body 50.

The alarm device 40 is a device that issues an alarm to the driver 60. The alarm device 40 may include a speaker that emits a warning sound and a vibration generator that vibrates the seat of the driver 60. The alarm device 40 can also serve as a display device in the moving body 50. The ECU 45 is a computer for controlling the mobile body 50. When the moving body is a vehicle, the ECU may include an ECU for brake control of the moving body 50, an ECU for engine control, and an ECU for steering control.

The "moving body" in the present disclosure may include, for example, a vehicle, a ship, an aircraft, and the like. Vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, living vehicles, fixed-wing aircraft traveling on runways, and the like. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, trolley buses and the like. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, lawnmowers and the like. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, crane trucks, dump trucks, road rollers and the like. The vehicle may include a vehicle that runs manually. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can travel on the road. The same vehicle may be included in multiple categories. Vessels may include, for example, marine jets, boats, tankers and the like. Aircraft may include, for example, fixed-wing aircraft, rotorcraft, and the like.

The camera 20 is fixed to the moving body 50. The camera 20 can be detachably fixed to the moving body 50. The camera 20 can image an external region of the moving body 50. The position of the camera 20 is arbitrary inside and outside the moving body 50. For example, in one embodiment, the camera 20 is located behind the moving body 50 as shown in FIG. 2 and can image an external region behind the moving body 50. In another embodiment, the camera 20a may be provided inside the moving body 50 instead of the camera 20. For example, in one embodiment, the camera 20 faces the rear window of the moving body 50 and can image an external region behind the moving body 50 as shown in FIG. The position of the detection device 30 is arbitrary within the moving body 50. The alarm device 40 issues an alarm. The driver 60 can perceive the alarm issued by the alarm device 40. For example, when the alarm device 40 is a display device, the alarm device 40 is visibly placed by the driver 60. For example, as shown in FIG. 2, the alarm device 40 is located in the dashboard of the moving body 50.

The camera 20 will be described in detail. For example, as shown in FIG. 1, the camera 20 includes an image pickup optical system 21, an image pickup element 22, a control unit 23, and a communication unit 24.

The image pickup optical system 21 forms a subject image on the light receiving surface of the image pickup element 22. For example, the imaging optics 21 may include an aperture and one or more lenses. A wide-angle lens can be adopted as the lens. As the lens, a fisheye lens may be used. The fisheye lens may have a horizontal angle of view of 180 ° or more, for example. The fisheye lens can have, for example, a horizontal angle of view of less than 180 °.

The image sensor 22 has a plurality of pixels arranged in two dimensions. The image pickup device 22 may include, for example, a CCD (Charge Coupled Device) image pickup device or a CMOS (Complementary Metal Oxide Semiconductor) image pickup device. The image pickup device 22 can take an image of a subject imaged by the image pickup optical system 21 and generate an image pickup image. In the present application, the captured image may be a moving image. Video can be rephrased as video.

The control unit 23 includes one or more processors. The "processor" in the present disclosure may include a dedicated processor specialized for a specific process, and a general-purpose processor that executes a specific function by loading a specific program. Dedicated processors may include DSPs (Digital Signal Processors) and application specific integrated circuits (ASICs). The processor may include a programmable logic device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The control unit 23 may be either a SoC (System-on-a-Chip) in which one or a plurality of processors cooperate, or a SiP (System In a Package).

The control unit 23 controls the operation of the entire camera 20. The control unit 23 may cause the image pickup device 22 to generate an image pickup image at an arbitrary frame rate. The frame rate may substantially match the frame rate that can be displayed by the display device, which is an example of the alarm device 40, for example. The frame rate may be higher than the frame rate that can be displayed by the display device, which is an example of the alarm device 40, for example. When the frame rate is larger than the frame rate that can be displayed, the display device, which is an example of the alarm device 40, can display a part of the frames. The image processing may include, for example, exposure adjustment processing, white balance processing, distortion correction processing, and the like. The control unit 23 outputs the captured image to the detection device 30 via the communication unit 24. For example, the control unit 23 may sequentially output captured images at the above-mentioned frame rate. Hereinafter, each captured image output at the above-mentioned frame rate is also simply referred to as a frame. The plurality of captured images output from the camera 20 are also referred to as a first image.

The communication unit 24 includes a communication interface capable of communicating with the detection device 30. The communication unit 24 may include a communication interface capable of communicating with the alarm device 40, the ECU 45 and other external devices. The communication unit 24 may be able to send and receive information via the network 15. The "communication interface" in the present disclosure may include, for example, a physical connector and a wireless communication device. The physical connector may include an electric connector corresponding to transmission by an electric signal, an optical connector corresponding to transmission by an optical signal, and an electromagnetic connector corresponding to transmission by an electromagnetic wave. The electric connector includes a connector compliant with IEC60603, a connector compliant with the USB standard, a connector corresponding to the RCA terminal, a connector corresponding to the S terminal specified by EIAJ CP-1211A, and a D terminal specified by EIAJ RC-5237. It may include a corresponding connector, a connector conforming to the HDMI® standard, and a connector corresponding to a coaxial cable including a BNC (British Naval Connector or Baby-series N Connector, etc.). Optical connectors may include various connectors according to IEC 61754. The wireless communication device may include a wireless communication device conforming to each standard including Bluetooth (registered trademark) and 802.11. The wireless communication device includes at least one antenna.

The detection device 30 will be described in detail. The detection device 30 includes a controller 31, a memory 32, an input unit 33, and an output unit 34.

The input unit 33 is an input interface for receiving an image captured by the camera 20. The output unit 34 is an output interface that outputs a signal to the alarm device 40 and the ECU 45. The input unit 33 and the output unit 34 may be one communication unit having a communication interface similar to that of the communication unit 24 of the camera 20.

The controller 31 includes one or more processors. The controller 31 may be either an SoC or a SiP in which one or more processors cooperate. The controller 31 can execute a process including the first process and the second process. In the first process, the controller 31 recognizes the deposits attached to the translucent member exposed to the outside of the moving body 50 in the image captured by the camera 20. In the second process, the controller 31 detects a portion of the captured image as a detection image based on the shape or feature of the recognition object. The controller 31 generates a signal based on the result of the first process and the result of the second process. In the present application, the "recognition target object" is an object that the detection device 30, the detection system 10, or the camera device 11 described later recognizes from the captured image. In the drawings, the recognition object may be simply described as "object".

The translucent member can be the lens closest to the subject of the image pickup optical system 21 when the image pickup optical system 21 of the camera 20 is exposed to the outside of the moving body 50. The light projecting member can be a window when the camera 20 is provided inside the moving body 50, as shown by the camera 20a in FIG. Windows can include rear windows, front windows, and side windows. This window can be called a windshield.

The deposit is an object that adheres to the light projecting member and at least partially obstructs the imaging field of view of the camera 20. Deposits include solids, liquids, and mixtures thereof. The deposit may have light transmission and may not have light transmission. Deposits can include raindrops during rainfall, snow during snowfall, muddy water splashed from the ground, leaves, insects, debris, and the like.

The recognition target is a subject to be detected. The object to be recognized may include one or more of pedestrians, automobiles, bicycles, and utility poles. The detected image is an image of a candidate recognition object that can be determined to be a recognition object detected from the captured image. When the recognition target is in the traveling direction of the moving body 50 in the captured image, the controller 31 can recognize the recognition target as an obstacle.

The signal includes an alarm signal sent to the alarm device 40 via the output unit 34. The alarm device 40 can issue an alarm to the driver 60 in response to the alarm signal sent from the detection device 30. When the alarm device 40 is a display device, the signal may be an image signal obtained by processing an image captured by the camera 20. For example, when the controller 31 detects a recognition object that is a pedestrian in the captured image and the pedestrian is recognized as an obstacle, the driver 60 pays attention to the pedestrian image displayed on the alarm device 40. Can be processed to evoke. For example, the controller 31 can add a color such as red to the image of a pedestrian.

The alarm signal may be transmitted to the ECU 45. For example, when the ECU 45 is an ECU for brake control, when the ECU 45 receives a warning signal, the moving body 50 is stopped to avoid a collision with a pedestrian. For example, when the ECU 45 is an ECU for course control, when the ECU 45 receives a warning signal, the ECU 45 changes the course of the moving body 50 to avoid a collision with a pedestrian. For example, when the ECU 45 is an ECU for acoustic control, when the ECU 45 receives a warning signal, it emits a sound and issues a warning to at least one of the driver 60 and the pedestrian.

The memory 32 may include a primary memory and a secondary memory. The memory 32 may be configured by using, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk, magnetic tape, and the like. The memory 32 stores various information and programs necessary for the operation of the controller 31. For example, the memory 32 may temporarily store the captured image output from the camera 20 and the results of the first processing and the second processing. The memory 32 may store data used for the first process and the second process. For example, the memory 32 may store in advance templates for recognition objects such as pedestrians, automobiles, and bicycles, algorithms obtained by machine learning, and programs for detecting recognition objects, which are used for template matching described later.

As shown in FIG. 3, the camera 20 of the detection system 10 and the detection device 30 can be integrated as the camera device 11. The camera device 11 includes a camera 20a including an image pickup optical system 21a and an image pickup element 22a, a controller 31a, a memory 32a, and an output unit 34a. The image pickup optical system 21a, the image sensor 22a, the controller 31a, the memory 32a, and the output unit 34a have the same functions as the image pickup optical system 21, the image sensor 22, the controller 31, the memory 32, and the output unit 34, respectively. Have. The detection system 10 can be regarded as a camera device whose functions are distributed between the camera 20 and the detection device 30.

The operation of the controller 31 according to one of the plurality of embodiments will be described in more detail with reference to the functional block diagram of FIG.

In one of the plurality of embodiments, the controller 31 includes a coordinate conversion unit 35, a detection unit 36, a recognition unit 37, a determination unit 38, and a signal generation unit 39 (hereinafter, also referred to as each unit). Each of the parts may be a hardware module or a software module. The controller 31 can execute the operations that each part can perform. The controller 31 is not limited to the form including the coordinate conversion unit 35, the detection unit 36, the recognition unit 37, the determination unit 38, and the signal generation unit 39, and the coordinate conversion unit 35, the detection unit 36, the recognition unit 37, the determination unit 38, and the signal. One or more of the generation units 39 may be omitted. In one of the plurality of embodiments, the controller 31 may perform all operations of each part. The operation performed by each unit may be paraphrased as the operation performed by the controller 31. The controller 31 may execute the process performed by the controller 31 by using any of the parts. The operation of each part will be described below.

The coordinate conversion unit 35 performs coordinate conversion of the captured image input from the camera 20 and corrects the distortion. As shown in FIG. 5, the first image 70, which is an image captured by the image pickup optical system 21 of FIG. 1 using a fisheye lens, has a larger distortion toward the periphery, as shown in FIG. The effective region of the first image 70 is determined according to the angle of view of the lens and the pixel area of the image sensor. When the pixel area of the image pickup device is large with respect to the angle of view of the lens, the effective region of the first image 70 has an arcuate end. In FIG. 5, images 71a and 71b of the deposit and images 72 of the recognition target object which is a pedestrian are displayed. The coordinate conversion unit 35 converts the captured image into a second image 75 with distortion corrected as illustrated in FIG. In the second image 75, images 76a and 76b of deposits and images 77 of the recognition target object that is a pedestrian are displayed. It should be noted that FIGS. 5 and 6 show images before and after the coordinate conversion, and do not accurately perform the coordinate conversion. Further, FIGS. 5 and 6 show parking lots, and white lines indicating parking spaces are shown by solid lines.

By performing coordinate conversion by the coordinate conversion unit 35, the recognition unit 37 can perform matching using the same pattern on the entire second image 75. The pattern includes a regional pattern such as a template matching template and a feature pattern. The controller 31 does not have to have the coordinate conversion unit 35. For example, when the control unit 23 of the camera 20 corrects the distortion of the image, or when the imaging optical system 21 does not use a lens having a large distortion such as a fisheye lens, the coordinate conversion unit 35 of the controller 31 becomes unnecessary.

The detection unit 36 executes the first process on the second image 75 output from the coordinate conversion unit 35. In the first process, the detection unit 36 can detect the first region to which the deposits are attached from the captured image based on various methods. The first region is shown in FIG. 6 as the region where the images 76a and 76b of the deposits are displayed. For example, the first region may be set including the coordinates in the vicinity of the portion where the deposit is attached on the captured image. For example, the first region may include a predetermined number of pixels around the region to which the deposits have adhered. For example, the first region may be set so as to convexly enclose the deposits for each deposit.

As a first method of detecting the first region, the detection unit 36 can detect a blurred region of the captured image as the first region. This is because the camera 20 is largely out of focus on the deposits. More specifically, the detection unit 36 can divide the captured image into fine blocks, perform a Fourier transform, and detect the block having a low high frequency component as the first region. As a second method, if the moving body 50 is moving, the detection unit 36 can adopt a method based on motion detection. In this method, the detection unit 36 divides the captured image into fine blocks, integrates regions in which the directions of the motion vectors of the blocks are similar, and determines that the region with less motion with respect to the background image is the first region. Can be done. As a third method, the detection unit 36 performs detection based on the feature amount of the deposit. The detection unit 36 may use the result of learning the image of the deposit by machine learning in advance for the determination of the first region. As a fourth method, the detection unit 36 can determine a region having a low brightness value of the captured image as the first region. The detection unit 36 may use the first to fourth methods in combination for the detection of the first region. Further, the detection unit 36 can adopt various methods other than the first to fourth methods.

The recognition unit 37 executes the second process on the second image 75 output from the coordinate conversion unit 35. In the second process, the recognition unit 37 detects a detected image which is a candidate image of the recognition target with respect to the captured image. The recognition unit 37 can have an algorithm for determining the recognition target-likeness of the detected image with respect to a specific recognition target. The recognition target-likeness can be a probability indicating the certainty that the detected image is a specific recognition target such as a pedestrian, a car, or a bicycle. The detection of the recognition target candidate performed by the recognition unit 37 and the output of the recognition target-likeness of the recognition target candidate are referred to as recognition of the recognition target.

The method of recognizing the object to be recognized shall include various known methods. For example, the method of recognizing a recognition object includes a method of recognizing the shape of an object, a method of template matching, and a method of calculating a feature amount from an image and using it for matching.

In the shape recognition of an object, an object is extracted from the captured image by a method such as background subtraction or motion detection, and it is determined whether or not the shape of the object matches the shape of the target object.

In template matching, a template of a recognition object is prepared and slid on the captured image, and a partial image having a high degree of similarity to the template image is searched for in the entire captured image. For the calculation of similarity, an index such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference), NCC (Normalized Cross-Correlation), ZNCC (Zero-means Normalized Cross-Correlation) can be used.

In the method using the feature amount, the feature amount obtained by quantifying the features useful for recognizing the pattern is extracted from the image and used for detecting the candidate of the recognition target object. In this method, Har-like features, SIFT (Scale-Invariant Feature Transform) features, SURF (Speed Up Robust Features) features, HOG (Histograms of Oriented Gradients) features, JOINT Haar-like features, Joint HOG A feature selected from many types of features can be used, including features, Shapelet features, CoHOG (Co-occurrence Histograms of Oriented Gradients) features, and the like. In this method, the detected image that is a candidate for the recognition target in the captured image is detected by matching the feature amount pattern of the recognition target with the captured image. In this recognition method, a feature amount is calculated in advance from a large number of images for learning, and the result is learned by a computer. The classifier recognizes the recognition object based on the learned result. The classifier used includes SVM (Support Vector Machine), AdaBoost, Real AdaBoost and the like. The set value of the feature amount can be a value calculated by artificial intelligence by deep learning.

Both methods quantify the uniqueness of the object to be recognized. If the recognition object likeness is equal to or more than the threshold value, it can be determined that it is a recognition object, and if it is less than the threshold value, it can be determined that it is not a recognition object. The recognition object-likeness can express the possibility of being a recognition object with a probability.

The determination unit 38 determines whether the detected image is a recognition target based on the first region detected by the detection unit 36 in the first process and the recognition target likeness of the detected image detected by the recognition unit 37 in the second process. Judge whether or not. The determination unit 38 may execute different processes in order to determine that the first region and the second region, which is a region other than the first region, are recognition objects.

For example, the determination unit 38 has a plurality of determination criteria for the recognition object-likeness for determining whether or not the detected image is a recognition object. The determination unit 38 can make the determination criteria different from each other in the first region and the second region. The determination criterion can be a threshold value for the recognition object-likeness for the determination unit 38 to determine that the detected image is a specific recognition object such as a pedestrian, an automobile, or a bicycle. For example, the determination unit 38 can reduce the possibility of erroneously determining the deposit as a recognition target such as a pedestrian, an automobile, or a bicycle by raising the threshold value in the first region as compared with the second region. Further, for example, the determination unit 38 lowers the threshold value in the first region as compared with the second region, so that the recognition target such as a pedestrian, an automobile, or a bicycle is not determined to be the recognition target and is not detected. It is possible to reduce the possibility of becoming.

For example, the determination unit 38 recognizes based on the result of excluding a part or all of the detected image detected in the first process from the detected image detected in the second process at least partially overlapping the first region. The object may be determined. In this case, the first process is executed after the second process or in parallel with the second process. The controller 31 may determine the recognition target object and generate a signal based on the result of excluding the detection image that completely overlaps the first region from the detection image detected in the second process.

When the determination unit 38 determines that there is a recognition object, the signal generation unit 39 generates a signal including an alarm regarding the determined recognition object. The signal generation unit 39 may generate an alarm when it is determined that the recognition target is located in the traveling direction or the predicted course of the moving body 50. The signal including the alarm is transmitted to the alarm device 40 and / or the ECU 45 as an alarm signal via the output unit 34 and the network 15.

Next, a first example of the process executed by the controller 31 according to the functional block diagram of FIG. 4 will be described with reference to the flowchart of FIG.

When the coordinate conversion unit 35 receives the input of the first image 70 which is the captured image from the camera 20 (step S101), the coordinate conversion unit 35 coordinates the first image 70 into the second image 75 corrected for distortion (step S102).

In FIG. 7, the processing sandwiched between the upper and lower double lines means that parallel processing is performed. That is, in the first example, the first process and the second process are performed in parallel.

The detection unit 36 recognizes the deposit from the second image 75 (step S103). When the detection unit 36 recognizes the deposit (step S104), the detection unit 36 delivers the coordinates of the first region on the second image 75 to which the deposit is attached to the determination unit 38. The determination unit 38 changes the determination criteria for the first region (step S105). In the example of the flowchart of FIG. 7, it is assumed that the determination standard for the second region is given to the entire second image before step S103, and the determination criterion of the first region is changed in step S105. In the flowchart of FIG. 7, it is not assumed that the determination criteria for the second region is given to the entire second image before step S103, and after step S104, the first region and the second region are determined according to the presence or absence of deposits. A different criterion may be set for each of the regions.

As an example of changing the determination criteria, the threshold value for determining that the object is a recognition object can be changed based on the appearance of the recognition object. The threshold value can be variously changed depending on the object to be detected, the surrounding environment, the purpose of changing the threshold value, and the like. For example, the determination unit 38 can set the threshold value of the first region to 80% and the threshold value of the second region to 60%. By doing so, it is possible to make the determination of the recognition target object in the first region where the deposit is detected strict and reduce the possibility of erroneously determining the deposit as the recognition target object. When no deposit is detected in step S104, the determination unit 38 sets all the regions as the second region without deposits.

The recognition unit 37 recognizes the recognition target object in parallel with steps S103 to S105 (step S106). Recognition of the recognition target includes detecting a detection image which is a candidate image of the recognition target from the second image and calculating the uniqueness of the specific recognition target.

The determination unit 38 performs the determination process according to the determination criteria set in steps S104 and S105 (step S107).

When the determination unit 38 determines in step S107 that there is an object to be recognized (step S108), the signal generation unit 39 generates a signal to be transmitted to the alarm device 40 and / or the ECU 45 (step S109).

For example, in step S106, with respect to the captured image which is the second image 75 of FIG. 6, the recognition unit 37 can detect the detection images 78a and 78b of the candidates of the recognition target object which is a pedestrian as shown in FIG. In FIG. 8, the detected images 78a and 78b are shaded to emphasize them. The detected image 78b erroneously recognizes a part of the attached image 76a as a pedestrian. The recognition unit 37 outputs the recognition target-likeness of the detected image together with the detected images 78a and 78b. For the sake of explanation, the recognition target-likeness (that is, pedestrian-likeness) of the detected images 78a and 78b is set to 70% for both. When the determination unit 38 sets the threshold value of the recognition object-likeness of the first region to 80% and the threshold value of the recognition object-likeness of the second region to 60%, the detection image 78a belonging to the second region is considered to be a pedestrian. Although it is determined, the detected image 78b belonging to the first region is not determined to be a pedestrian. As a result, the possibility of erroneously determining a part of the image 76a of the deposit as a recognition target is reduced.

FIG. 9 shows another example of the second image 75. In FIG. 9, the first regions 79a to 79c may coincide with the region where the image of the deposit is displayed. The first regions 79a to 79c may include regions in which an image of the deposit is displayed. In FIG. 9, the image of the deposit is omitted. The regions other than the first regions 79a to 79c of the second image 75 are the second regions. In this example, the image 77a of the recognition target object, which is a pedestrian, overlaps the first region 79a in which the deposit is detected by the detection unit 36.

The recognition unit 37 can evaluate the shape and characteristics of the upper region that does not overlap with the first region 79a of the image 77a of the recognition target and detect it as the detection image 78c. The determination unit 38 sets the threshold value of the recognition target-likeness to 80% in the first regions 79a to 79c, and sets the threshold value of the recognition target-likeness to 60% in the second region excluding the first regions 79a to 79c from the second image 75. And. The determination unit 38 can determine that the person is a pedestrian using the threshold value 60% of the second region based on the detection image 78c from the recognition unit 37 and the appearance of the recognition object. On the other hand, the portion of the second image 75 that overlaps with the first region 79a of the image 77a of the recognition target is not detected by the recognition unit 37 because it is blocked by the deposits, or is recognized by the determination unit 38 because the threshold value is high. Not judged as a thing. However, in the second region, since it is determined that the detected image 78c of a part of the image 77a of the image to be recognized is the object to be recognized, the signal generation unit 39 sends the alarm device 40 and / or the ECU 45. It is possible to generate a signal to be transmitted.

The detection system 10 raises the threshold value for the recognition object-likeness of the second region higher than the threshold value for the recognition object-likeness of the first regions 79a to 79c, for example, for deposits when the deposits are rain or the like. As a result of the image being distorted, the possibility of misrecognition can be reduced. On the contrary, when the recognition omission that cannot recognize the existence of the recognition object becomes a problem, the detection system 10 may set the threshold value of the first region 79a to 79c lower than the threshold value of the second region. In FIG. 8, since the image 77b of the recognition target object, which is a car, does not overlap with the first regions 79a to 79c, it is determined whether or not the image 77b is a recognition target object using the threshold value of the second region. When the entire detected image is included in the first regions 79a to 79c, the threshold value set in the first region is applied to the entire detected image.

In this way, according to the detection system 10 according to one of the plurality of embodiments of the present disclosure, mud and / on the lens of the camera 20 or the window when the camera 20 is located inside the moving body 50. Alternatively, it is possible to improve the recognition accuracy of the recognition target object when deposits such as raindrops adhere to it. Here, the recognition accuracy of the recognition object means the accuracy of detecting the candidate of the recognition object from the captured image and determining the correct recognition object from the detected candidates of the recognition object.

Next, a second example of the process executed by the controller 31 according to the functional block diagram of FIG. 4 will be described with reference to FIG.

Since steps S201 to S204, S206, S208, and S209 of FIG. 10 are the same as steps S101 to S104, S106, S108, and S109 of the first example shown in FIG. 7, description thereof will be omitted. In step S205, when the determination unit 38 hands over the coordinates of the first region from the detection unit 36, the determination unit 38 sets the first region as the determination exclusion region. In step S207, the determination unit 38 excludes the detected image recognized in step S206 from the target of the determination process when all or at least a part of the detected image overlaps with the determination exclusion region.

For example, in FIG. 9, the image 77a of the recognition target has an overlap with the first region 79a, and is therefore excluded from the determination process. On the other hand, the image 77b of the recognition target object that does not overlap with the first regions 79a to 79c is detected as a candidate for the detection target object and undergoes determination processing. According to the second example of the process executed by the controller 31, the determination unit 38 recognizes the deposit as the recognition target because the contour of the deposit resembles the contour shape of the recognition target such as a pedestrian or an automobile. It is possible to reduce the possibility of erroneously recognizing the existence. Further, since the determination unit 38 excludes the image overlapping with the adhered object from the determination, the possibility of erroneously determining an image other than the recognition object as the recognition object is lower than in the first example, and the determination unit 38 The process performed by 38 is also simple.

Even if the detected image overlaps the determination exclusion region, the determination unit 38 may change whether or not to exclude the detected image from the target of the determination processing according to the size of the overlap and the overlapping method. For example, the determination unit 38 may exclude the entire detected image from the determination process only when it is included in the determination exclusion region. Further, for example, the determination unit 38 may exclude the detection image from the determination process when the area on the second image 75 of the detection image that does not overlap with the determination exclusion region is equal to or larger than a predetermined size.

Next, a third example of the process executed by the controller 31 according to the functional block diagram of FIG. 4 will be described with reference to FIG. In the second example, the first process of recognizing the deposit and the second process of recognizing the detected image were processed in parallel. In the third example, the first process is sequentially executed after the second process is performed. In FIG. 11, steps S301 and S302 are the same processes as steps S201 and S202 in FIG. 10, respectively. In FIG. 11, step S303 is the same process as step S206 in FIG. In FIG. 11, steps S304 to S306 are the same processes as steps S203 to S205 in FIG. 10, respectively. In FIG. 11, steps S307 to S309 are the same processes as steps S207 to S209 in FIG. 10, respectively.

The operation of the controller 31 according to the other one of the plurality of embodiments will be described with reference to the functional block diagram of FIG.

In FIG. 12, the detection unit 36 delivers the coordinates of the first region on the second image 75 to which the deposits are attached to the recognition unit 37. The recognition unit 37 has a plurality of algorithms for detecting the detected image of the recognition target object from the captured image in the second process, and detects the detected image by using algorithms different from each other in the first region and the second region. It can be performed.

Using different algorithms includes changing the basic algorithms of object recognition, such as object shape recognition, template matching, and recognition using features. For example, with respect to the second image 75 of FIG. 9, the recognition unit 37 performs pattern matching using the feature amount on the first regions 79a to 79c, and detects the detected image by template matching on the second region. You may go.

Using different algorithms includes making some elements used in the recognition algorithm different. For example, the recognition unit 37 may change the template used for template matching. Alternatively, the recognition unit 37 can change the learning result obtained by machine learning in the recognition using the feature amount. Further, the recognition unit 37 can change any parameter used for the image recognition process.

A fourth example of the process executed by the controller 31 according to the functional block diagram of FIG. 12 will be described with reference to FIG. In FIG. 13, steps S401 to S403 and S407 to 409 are the same processes as steps S101 to S103 and S107 to S109 in FIG. 7, and thus description thereof will be omitted. Unlike the flowchart of FIG. 7, each step S401 to S409 is sequentially processed without including parallel processing. In step S404, the detection unit 36 delivers the coordinates of the first region on the second image 75 to which the deposits are attached to the recognition unit 37. In step S405, the recognition unit 37 changes the algorithm for recognizing the recognition object for all or part of the captured image from the preset algorithm. In step S406, the recognition unit 37 detects the detection image of the candidate recognition target based on the modified algorithm.

According to the fourth example of the process executed by the controller 31, the recognition unit 37 has a plurality of algorithms for detecting candidates for recognition objects from the captured image in the second process performed by the recognition unit 37, and has a first region. And in the second region, algorithms different from each other can be used. As a result, the recognition unit 37 can select an algorithm with higher recognition accuracy depending on whether or not it overlaps with the deposit on the captured image.

The operation of the controller 31 according to still one of the plurality of embodiments will be described with reference to the functional block diagram of FIG.

As shown in FIG. 14, the controller 31 of the detection system 10 according to one of the plurality of embodiments of the present disclosure includes the coordinate conversion unit 35, the detection unit 36, the recognition unit 37, the determination unit 38, and the signal generation unit 39. , Including the pretreatment unit 80. In FIG. 14, the detection unit 36 delivers the coordinates of the first region on the second image 75 to which the deposits are attached to the preprocessing unit 80. The preprocessing unit 80 performs a process of changing the second image 75 of the captured image output from the coordinate conversion unit 35. The change of the captured image that the preprocessing unit 80 performs on the second image 75 before the second processing is also referred to as preprocessing below. The preprocessing includes deleting the first region from the second image 75 and processing it into a blank. Processing the first region blank from the captured image is also referred to as masking the first region.

The preprocessing performed by the preprocessing unit 80 is not limited to masking the first region. For example, the pretreatment unit 80 may blur the image of the deposits in the first region. By doing so, the recognition unit 37 makes it difficult for the recognition unit 37 to recognize the deposit as a recognition target.

FIG. 15 shows an example of the second image 75 in which the first regions 79a to 79c are masked. In FIG. 15, the masked area is shown surrounded by a broken line. The first regions 79a to 79c are blank. The recognition unit 37 can execute the recognition of the recognition target object for the second image in which the pixel information of the first regions 79a to 79c is deleted.

A fifth example of the process executed by the controller 31 according to the functional block of FIG. 14 will be described with reference to FIG. In FIG. 16, steps S501 to S503 and S507 to 509 are the same processes as steps S101 to S103 and S107 to S109 in the flowchart of FIG. 7, and thus description thereof will be omitted. Unlike the flowchart of FIG. 7, each step S401 to S409 is sequentially processed without including parallel processing. In step S504, the detection unit 36 delivers the coordinates of the first regions 79a to 79c on the second image 75 to which the deposits are attached to the preprocessing unit 80. In step S505, the preprocessing unit 80 processes the pixel information of the detection points of the first regions 79a to 79c of the second image 75. In step S506, the recognition unit 37 executes the recognition process based on the second image 75 in which the pixel information of the first regions 79a to 79c is processed.

According to the fifth example, the detection system 10 can exclude the information of the first region to which the deposits are attached from the detection and determination of the detected image without changing the processing of the recognition unit 37 and the determination unit 38. This reduces the possibility of erroneously recognizing the deposit as a recognition target. Further, the portion of the recognition object that does not overlap with the first regions 79a to 79c on the captured image can be recognized and determined as a part of the recognition object by the recognition unit 37 and the determination unit 38.

Although the present invention has been described based on the drawings and embodiments, it should be noted that those skilled in the art can easily make various modifications and modifications based on the present disclosure. Therefore, it should be noted that these modifications and modifications are within the scope of the present invention. For example, the functions included in each means, each step, etc. can be rearranged so as not to be logically inconsistent, and a plurality of means, steps, etc. can be combined or divided into one. ..

10 Detection system 11 Camera device 15 Network 20,20a, 20b Camera 21,21a Image sensor 22,22a Image sensor 23 Control unit 24 Communication unit 30 Detection device 31, 31a Controller 32, 32a Memory 33 Input unit 34, 34a Output unit 35 Coordinate conversion unit 36 First detection unit 37 Recognition unit 38 Judgment unit 39 Signal generation unit 40 Alarm device 45 ECU
50 Moving object 60 Driver 70 First image 71a, 71b Image of deposit 72 Image of recognition object 75 Second image 76a, 76b, 76c Image of deposit 77, 77a, 77b Image of recognition target 78a, 78b, 79 Detected image 79a, 79b, 79c 1st area 80 Preprocessing unit

Claims (8)

A camera attached to a moving body and taking an image of the surroundings of the moving body,
With respect to the image captured from the camera, the first process of recognizing the deposits attached to the translucent member exposed to the outside of the moving body and detecting the first region on the captured image where the deposits are located , and A controller that performs a second process of detecting a portion of the captured image as a detection image based on the shape or feature of the recognition target, and generates a signal based on the result of the first process and the result of the second process. equipped with a,
The controller has a plurality of algorithms for detecting the detected image from the captured image in the second process, and the first region and the second region, which is a region other than the first region of the captured image, , A camera device that recognizes the recognition object using algorithms different from each other . In the second process, the controller determines the recognition target-likeness of the detected image, and the controller determines whether or not the detected image is the recognition target. A claim in which the determination criteria are different between the detection image having a plurality of determination criteria for object-likeness and having all or part of the first region overlapping and the detection image belonging only to the second region. The camera device according to 1 . The camera device according to claim 1 or 2 , wherein the controller modifies the captured image based on the first region detected in the first process, and performs the second process on the changed captured image. The camera device according to claim 3 , wherein the controller deletes the image of the first region from the captured image and performs the second process. The camera device according to any one of claims 1 to 3 , wherein the controller generates the signal when the detected image is determined to be the recognition target. An input unit that is attached to a moving body and receives an image captured by a camera that images the surroundings of the moving body.
Regarding the captured image from the camera received by the input unit, the deposits adhering to the translucent member exposed to the outside of the moving body are recognized, and the first region on the captured image where the deposits are located is detected. With a controller that performs the first process of performing the first process and the second process of detecting the detected image based on the shape or feature of the recognition target, and generates a signal based on the result of the first process and the result of the second process.
The controller has a plurality of algorithms for detecting the detected image from the captured image in the second process, and is a second region other than the first region and the first region of the captured image. A region is a detection device that recognizes the recognition target using algorithms different from each other . A camera attached to a moving body and taking an image of the surroundings of the moving body,
With respect to the input unit that receives the image captured from the camera and the image captured from the camera received by the input unit, the deposits adhering to the translucent member exposed to the outside of the moving body are recognized and attached. The first process of detecting the first region on the captured image in which the kimono is located and the second process of detecting the detected image based on the shape or feature of the recognition target are performed, and the result of the first process and the said look including a controller for generating a signal based on the result of the second processing, the controller, in the second processing, a plurality of algorithms for detecting said detection image from the captured image, said first region, A detection system including a detection device that recognizes the recognition object by using algorithms different from those of the second region, which is a region other than the first region of the captured image . With respect to a camera that images the surroundings and an image captured by the camera, a first region on the captured image in which the deposits are located is recognized by recognizing deposits adhering to a translucent member exposed to the outside. 1 process, and performs a second process of detecting a detection image on the basis of the shape or characteristics according to the object to be recognized, unrealized a controller that generates a signal based on the result of the result and the second processing of the first processing The controller has a plurality of algorithms for detecting the detected image from the captured image in the second process, and includes the first region and a second region which is a region other than the first region of the captured image. Then, a moving body including a camera device that recognizes the recognition object by using different algorithms .

Images