JP2013041481A - Image processing device, image processing system, camera and camera system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of efficiently detecting an object, and further to provide an image processing system, a camera, a camera system or the like.SOLUTION: An image processing device 100 includes: an image acquiring section 110 for acquiring infrared image data; a region setting section 120 for setting boundary lines that divide an image region into a plurality of regions with respect to the image region on the basis of the infrared image data acquired by the image acquiring section 110 and for setting at least one region out of the plurality of regions divided by the boundary lines to a pixel density changing region ARMD; and a processing section that performs a process for reducing pixel density of the infrared image data of the pixel density changing region ARMD, performs a detection process of an object on the basis of the infrared image data of the image region including the pixel density changing region ARMD and generates image data for display on the basis of a result of the detection process.

Description

  The present invention relates to an image processing apparatus, an image processing system, a camera, a camera system, and the like.

  Visual information from outside the vehicle is extremely important for the driver for driving. However, it is often necessary to drive in an environment that is dazzled by a low-light environment at night or the presence of a powerful light source such as a light from an oncoming vehicle. In such an environment with poor visibility with the naked eye, even a careful driver tends to delay the discovery of objects such as pedestrians approaching the vehicle.

  Patent Document 1 discloses a technique for detecting pedestrians and the like from images obtained by an infrared camera and a visible camera, and processing and displaying the images as a technique for covering such physiological problems on the driver side. It is disclosed.

  However, this method has a problem that the calculation processing load of the detection process for detecting the target object increases, so that the detection time becomes longer and the detection device becomes expensive.

JP 2008-236349 A

  According to some aspects of the present invention, an image processing apparatus, an image processing system, a camera, a camera system, and the like that can efficiently detect an object can be provided.

  One aspect of the present invention is an image acquisition unit that acquires infrared image data, and a boundary line that divides the image region into a plurality of regions with respect to an image region based on the infrared image data acquired by the image acquisition unit. And setting an area setting unit that sets at least one of a plurality of areas divided by the boundary line as a pixel density change area, and reducing the pixel density of the infrared image data in the pixel density change area A processing unit that performs processing, performs detection processing of an object based on the infrared image data of the image region including the pixel density change region, and generates display image data based on a result of the detection processing. It relates to an image processing apparatus.

  According to one aspect of the present invention, for an image area based on infrared image data, an area where there is a low possibility that an object such as a pedestrian is present is set as a pixel density change area, and infrared image data of the area is Pixel density can be reduced. By doing so, it is possible to reduce the calculation processing burden for the detection processing without reducing the detection capability of the object. As a result, the detection time can be reduced, the configuration of the processing unit can be simplified, and an efficient image processing apparatus can be realized.

  In one aspect of the present invention, the region setting unit determines a setting pattern that divides the image region using the boundary line in which at least one of a position and a shape in the image region is variably set, In the determined setting pattern, at least one of the position and shape of the boundary line in the image area may be determined.

  In this way, the area setting unit can variably set at least one of the position and the shape of the boundary line in the image area in the determined setting pattern. Therefore, an appropriate pixel density change region can be set.

  In the aspect of the invention, the area setting unit may be configured to determine a first setting pattern that divides the image area using the boundary line that divides the image area, and the determined setting pattern includes the determined setting pattern. In the case of the first setting pattern, one of the image areas divided by the boundary line may be set as the pixel density change area.

  In this way, for example, in a situation where the vehicle is traveling relatively smoothly, such as a suburban road, there is a possibility that the object is present by dividing the image area by the first setting pattern. The image area below the low boundary line can be set as the pixel density change area. As a result, it is possible to reduce the calculation processing load for the detection process without reducing the detection capability of the object.

  Moreover, in one aspect of the present invention, the area setting unit includes a second setting pattern that divides the image area using two boundary lines that are divided into three and do not intersect with each other. When the determined setting pattern is the second setting pattern, an image area sandwiched between the two boundary lines among the image areas divided into three by the boundary line is set as the pixel density change area. It may be set.

  In this way, for example, in a situation where the vehicle is traveling on an urban road, two boundary lines that are less likely to be present by dividing the image area into three by the second setting pattern. The image area sandwiched between the pixels can be set as a pixel density change area. As a result, it is possible to reduce the calculation processing load for the detection process without reducing the detection capability of the object.

  In the aspect of the invention, the region setting unit may be configured to determine a third setting pattern that divides the image region using a closed curve surrounding a part of the image region as the boundary line. When the set pattern that has been set is the third set pattern, the image area outside the closed curve may be set as the pixel density change area.

  In this case, for example, in a situation where the vehicle is traveling on a road with a small amount of traffic such as a mountainous area, the image area may be partitioned with a closed curve by the third setting pattern, and the object may exist. The image area outside the closed curve with a low can be set as the pixel density change area. As a result, it is possible to reduce the calculation processing load for the detection process without reducing the detection capability of the object.

  In the aspect of the invention, the display image data may include an alert display based on the result of the detection process.

  In this way, it is possible to easily predict the danger of the driver by displaying a highly visible alert symbol.

  Moreover, in one aspect of the present invention, the infrared image data is image data of an infrared image of an external area of a vehicle, and the area setting unit determines the setting pattern based on traveling state information of the vehicle, In the determined setting pattern, at least one of the position and shape of the boundary line in the image area may be determined.

  In this way, since the area setting unit can set the pixel density change area based on the traveling state of the vehicle, there is a possibility that an object such as a pedestrian exists depending on the speed of the vehicle, for example. An image area excluding a high area can be set as a pixel density change area.

  In the aspect of the invention, the traveling state information of the vehicle may be speed information of the vehicle.

  In this way, for example, the pixel density changing region is set so that the farther the object can be detected efficiently as the vehicle speed is faster, and the closer object can be detected more efficiently as the vehicle speed is slower. Thus, it is possible to set a pixel density change region.

  In the aspect of the invention, the region setting unit may determine the setting pattern based on information input by a user, and may include a position and a shape of the boundary line in the image region in the determined setting pattern. At least one may be defined.

  In this way, the user can set an appropriate pixel density change region by determining the vehicle driving condition, road condition, and the like.

  In one aspect of the present invention, a mode setting unit that performs mode switching setting between a driving mode and a crime prevention mode is further included. When the driving mode is set, the region setting unit includes the pixel density changing region. The processing unit performs a process of reducing the pixel density of the infrared image data in the pixel density change area, and the detection process based on the infrared image data of the image area including the pixel density change area And performing processing for generating display image data including an alert display based on the result of the detection processing, and when the crime prevention mode is set, the region setting unit unsets the boundary line. The processing unit performs a process of reducing the pixel density of the infrared image data with respect to the entire area of the image area, and performs processing based on the infrared image data of the image area. Performs detection processing may be performed a process of generating display image data including the alert display based on a result of the detection process.

  In this way, when the vehicle is traveling, the mode setting unit sets the traveling mode so that, for example, a pedestrian crossing the road can be efficiently detected and notified to the driver. Further, when the vehicle is parked, the mode setting unit sets the crime prevention mode, so that, for example, it is possible to detect a suspicious person entering the house at night and to notify the user.

  In the aspect of the invention, the processing unit may make the contents of the detection process different between the pixel density change area and an area other than the pixel density change area.

  In this way, when performing, for example, a template matching process as a detection process for detecting an object, a low-accuracy template is selected for the pixel density change area, and an area other than the pixel density change area is selected. It is possible to select a highly accurate template. As a result, since the detection processing time can be shortened, it is possible to detect the object efficiently.

  In the aspect of the invention, the processing unit may perform the detection process in the pixel density change region based on a result of the detection process in a region other than the pixel density change region.

  In this way, for example, when an object is detected in an area other than the pixel density change area, the detection process in the pixel density change area can be performed based on the detection result. Can be performed.

  In the aspect of the invention, the detection processing result in an area other than the pixel density change area may include a motion vector of the object.

  In this way, for example, when an object is detected in an area other than the pixel density changing area, the movement destination of the object can be predicted based on the motion vector of the object. When the object is predicted to move to the pixel density change region, for example, a template corresponding to the object can be selected in advance. As a result, it is possible to efficiently detect an object in the pixel density change region.

  In one aspect of the present invention, the image acquisition unit acquires visible image data, and the processing unit performs the detection process and superimposes the alert symbol based on the result of the detection process and the visible image data. Then, processing for generating display image data may be performed.

  In this way, when an object is detected, an alert display corresponding to the object can be superimposed and displayed on the visible image, so the driver can detect danger by using an alert display with good visibility. It becomes easy. As a result, for example, the burden on the driver during night driving can be reduced.

  Another aspect of the present invention relates to a camera system including any one of the image processing apparatuses described above and an infrared image capturing unit.

  According to another aspect of the present invention, there is provided a camera including the image processing device according to any one of the above, an infrared image capturing unit, and a housing in which the image processing device and the infrared image capturing unit are disposed. Involved.

  Another aspect of the present invention relates to a camera system including any one of the image processing apparatuses described above, an infrared image capturing unit, and a display unit that displays an image based on the display image data.

  According to another aspect of the present invention, there is provided the image processing device according to any one of the above, an infrared image capturing unit, a display unit that displays an image based on the display image data, the image processing device, and the infrared image capturing unit. The display unit is related to a camera including a housing provided therein.

  Another aspect of the present invention is an image acquisition unit that acquires infrared image data, and a boundary that divides the image region into a plurality of regions with respect to an image region based on the infrared image data acquired by the image acquisition unit. A line setting unit configured to set at least one of a plurality of regions divided by the boundary line as a pixel density change region; and to reduce the pixel density of the infrared image data in the pixel density change region A processing unit that performs processing, performs detection processing of an object based on the infrared image data of the image region including the pixel density change region, and generates display image data based on a result of the detection processing. Related to image processing system.

  Another aspect of the present invention relates to an image processing system including any one of the image processing apparatuses described above and a display unit that displays an image based on the display image data.

  Another aspect of the present invention includes an imaging region in which infrared imaging elements are two-dimensionally arranged, and in a part of the imaging region, the interval between the infrared imaging elements is a first interval, In an area other than a part of the area, an infrared image capturing unit in which the interval between the infrared imaging elements is a second interval wider than the first interval, and an image for acquiring infrared image data captured by the infrared image capturing unit An object is detected based on the infrared image data for an image region corresponding to the imaging region in the infrared image data acquired by the acquisition unit and the image acquisition unit, and the result of the detection processing And a processing unit that generates image data for display based on the above.

  According to another aspect of the present invention, the image acquisition unit acquires infrared image data having a high pixel density in a part of the imaging region and a low pixel density in a region other than the partial region of the imaging region. Can do. By doing so, it is possible to reduce the number of pixels of the infrared image data, and thus it is possible to reduce the calculation processing burden in the detection process. As a result, the detection time can be shortened, the configuration of the processing unit can be simplified, and an efficient camera can be realized.

1 is a first configuration example of an image processing apparatus, an image processing system, a camera, and a camera system. The 2nd structural example of an image processing apparatus, an image processing system, a camera, and a camera system. The 3rd structural example of an image processing apparatus, an image processing system, a camera, and a camera system. The 4th structural example of a camera. FIG. 5A to FIG. 5C show setting examples of pixel density change areas in three driving situations. FIGS. 6A and 6B are diagrams illustrating pixel density reduction processing according to the first configuration example. The figure explaining the pixel density reduction process by the 1st structural example. 8A and 8B are diagrams for describing pixel density reduction processing according to the second configuration example. FIG. 9A and FIG. 9B are diagrams for describing pixel density reduction processing according to the second configuration example. The flowchart of the detection process in an image processing apparatus etc. FIGS. 11A to 11C show examples of alert display in three driving situations. The figure explaining the crime prevention mode in the 3rd example of composition. The flowchart of the process of the crime prevention mode by the 3rd structural example. 14A and 14B show examples of mounting the camera system on a vehicle.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Image Processing Device, Image Processing System, Camera, and Camera System FIG. 1 shows a first configuration example of an image processing device 100, an image processing system 200, a camera 300, and a camera system 310 of this embodiment. The image processing apparatus 100 of the first configuration example includes an image acquisition unit 110, an area setting unit 120, and a processing unit 130, and may further include a storage unit 150. The image processing system 200 includes the image processing apparatus 100 and a display unit 230. The camera 300 includes the image processing apparatus 100, an infrared image capturing unit 210, a housing (not shown), and may further include a display unit 230. The camera system 310 includes the image processing apparatus 100 and the infrared image capturing unit 210, and may further include a display unit 230. The image processing system 200 may include an image acquisition unit 110, an area setting unit 120, and a processing unit 130.

  Note that the image processing apparatus 100, the image processing system 200, the camera 300, and the camera system 310 of the present embodiment are not limited to the configuration in FIG. 1, and some of the components are omitted or replaced with other components. Various modifications such as adding other components are possible.

  The image acquisition unit 110, the region setting unit 120, and the processing unit 130 can be realized by a processor such as a CPU or an ASIC such as a gate array, for example. The storage unit 150 can be realized by, for example, a RAM, a flash memory, or a hard disk drive (HDD). Note that the storage unit 150 may be omitted.

  The image acquisition unit 110 acquires an infrared image from the infrared image capturing unit 210 and outputs infrared image data to the processing unit 130. The infrared image data is, for example, image data of an infrared image of an external area of the vehicle.

  In addition, the image acquisition unit 110 acquires visible image data from the visible image capturing unit 220 and outputs the visible image data to the processing unit 130.

  The region setting unit 120 sets a boundary line for dividing the image region into a plurality of regions for the image region based on the infrared image data acquired by the image acquisition unit 110, and sets the plurality of regions divided by the boundary line. At least one of these areas is set as a pixel density change area. Specifically, the region setting unit 120 determines a setting pattern that divides the image region using a boundary line in which at least one of the position and shape in the image region is variably set. Then, at least one of the position and shape of the boundary line in the image area is determined in the determined setting pattern.

  More specifically, the region setting unit 120 determines and determines the setting pattern based on the vehicle travel status information, vehicle position information acquired by the global positioning system (GPS), user input information, and the like. In the set pattern, at least one of the position and the shape of the boundary line in the image area is determined. For example, the region setting unit 120 can determine a setting pattern based on vehicle speed information, headlight control information, and the like, and determine at least one of the position and shape of the boundary line. In addition, the setting pattern according to a driving | running | working condition is demonstrated later.

  Here, the image region is, for example, image data for one screen (frame) captured by the infrared image capturing unit 210 in which an image sensor is two-dimensionally arranged and acquired by the image acquiring unit 110 at predetermined time intervals. is there. That is, one image area is infrared image data that constitutes one screen (frame) captured at a certain timing by the infrared image capturing unit 210.

  The processing unit 130 performs processing for reducing the pixel density of the infrared image data in the pixel density changing region (pixel density reduction processing). An object detection process (template matching process) is performed based on the infrared image data of the image area including the pixel density change area, and display image data is generated based on the result of the detection process. This display image data may include an alert display based on the result of the detection process. In addition, the processing unit 130 performs a detection process, and performs a process of generating display image data by superimposing an alert symbol and visible image data based on a result of the detection process.

  Specifically, the processing unit 130 performs processing for reducing the pixel density on the infrared image data in the pixel density changing region ARMD set by the region setting unit 120. This process of reducing the pixel density (pixel density reduction process) is a process of equivalently reducing the pixel density on the image data, that is, the number of pixels per unit area, for example, an image constituting one screen (frame). This is a process (thinning process) for thinning out pixels in a certain area (pixel density changing area) of data. On the other hand, the pixel density reduction process is not performed on the infrared image data in the image area excluding the pixel density change area ARMD. Hereinafter, the image area excluding the pixel density change area ARMD is also referred to as a pixel density non-change area ARNO. The pixel density reduction process will be described in detail later.

  Further, the processing unit 130 performs an object detection process (template matching process) based on the infrared image data of the image area including the pixel density changing area ARMD, that is, the infrared image data after performing the pixel density reduction process. That is, in the pixel density changing area ARMD, detection processing for detecting an object (for example, a pedestrian or a bicycle) is performed based on infrared image data having a lower pixel density than the pixel density non-changing area ARNO. Alternatively, based on the infrared image data in which the pixel density is lower than the first threshold value in the pixel density change area ARMD and the pixel density is higher than the second threshold value in the pixel density non-change area ARNO, A detection process for detection is performed. Here, the first threshold value may be a value equal to or smaller than the second threshold value, and the second threshold value may be a value in a pixel density range that can be realized in the infrared image capturing unit 210. . The first and second threshold values may be the same value.

  The processing unit 130 includes a template matching processing unit 131, an image composition processing unit 132, and a memory 133. The memory 133 stores infrared image data after the pixel density reduction process (thinning process) is performed on the pixel density change area ARMD. By doing so, the pixel density of the pixel density changing region ARMD of the infrared image data stored in the memory 133 is lower than the first threshold value, and the pixel density of the pixel density non-changing region ARNO is the second threshold value. It becomes higher than the threshold. The template matching processing unit 131 performs template matching processing for detecting an object on the infrared image data after thinning processing stored in the memory 133.

  The image composition processing unit 132 performs processing for superimposing an alert display (alert symbol) or the like on the visible image data based on the template matching processing result. The image composition processing unit 132 may perform a process of further compositing (superimposing) other information as necessary. The image composition processing unit 132 may perform processing for superimposing the alert display on the infrared image data, or may perform processing for combining (superimposing) the visible image data, the infrared image data, the alert display, and the like. .

  The alert display is a symbol (alert symbol) for notifying the driver of the presence of a pedestrian or bicycle. The visible image on which the alert display is superimposed is displayed on the display unit 230. By displaying an alert symbol with good visibility, the driver can easily predict the danger. Details of the template matching process and the alert display will be described later.

  The storage unit 150 stores a plurality of setting patterns for setting a boundary line between the pixel density changing area ARMD and the pixel density non-changing area ARNO. The region setting unit 120 can select and read any one of a plurality of setting patterns from the storage unit 150 and set a boundary line based on the read setting patterns. By doing so, for example, since a plurality of setting patterns corresponding to a plurality of driving situations can be stored in the storage unit 150 in advance, it becomes possible to select an optimal setting pattern according to the driving situations. .

  The infrared image capturing unit 210 is an infrared camera, and has sensitivity to far infrared rays having a wavelength of 8 to 12 μm, for example. Infrared sensor cameras using pyroelectric elements to detect distant objects (for example, 200 m away) are even better.

  The visible image capturing unit 220 is a camera having sensitivity in the visible light region, and uses, for example, a CCD or a CMOS as an image sensor. This camera is preferably compatible with image data in a high dynamic range (HDR) format. An HDR-compatible camera can capture an image of a subject with different shutter speeds, high and low, using an electronic shutter function, and can process the two types of image signals to obtain an image having a wide dynamic range. By doing so, for example, when traveling at night, it is possible to clearly show a pedestrian or an oncoming vehicle that is usually obstructed by the headlight of the oncoming vehicle and is difficult to see.

  The display unit 230 displays an image including an infrared image, a visible image, an alert symbol, and the like. The display unit 230 can be realized by, for example, a liquid crystal display, an organic EL display, or the like.

  Although the housing is not shown, the image processing apparatus 100 and the infrared image capturing unit 210 are disposed therein.

  According to the image processing apparatus 100 or the like of the first configuration example shown in FIG. 1, the number of pixels of infrared image data to be subjected to template matching processing can be reduced, so that the calculation processing burden in detection processing is reduced. Can do. In addition, when an object is detected, an alert display with good visibility can be superimposed and displayed at a corresponding position in the visible image, so that it is possible to reduce the burden on the driver during night driving. .

  Further, the area setting unit 120 determines a setting pattern based on the vehicle driving situation information, the vehicle position information, the user input information, and the like, and among the position and shape of the boundary line in the image area in the determined setting pattern At least one can be defined. In this way, the pixel density changing area ARMD and the pixel density non-changing area ARNO can be variably set according to the driving situation of the vehicle, so that the pedestrian can be adapted to various driving situations and traffic conditions. An area where there is a low possibility of the presence of an object such as a pedestrian can be set as the pixel density change area ARMD, and an area where there is a high possibility that an object such as a pedestrian exists can be set as the pixel density non-change area ARNO. As a result, it is possible to reduce the processing load without reducing the detection capability (detection accuracy) of the object. As a result, the processing time of the detection process can be shortened, the circuit configuration of the processing unit 130 can be simplified, and an efficient image processing apparatus or the like can be realized.

  FIG. 2 shows a second configuration example of the image processing apparatus 100, the image processing system 200, the camera 300, and the camera system 310 of the present embodiment. The image processing apparatus 100 of the second configuration example includes an image acquisition unit 110, an area setting unit 120, and a processing unit 130, and may further include a storage unit 150. The image processing system 200 includes the image processing apparatus 100 and a display unit 230. The camera 300 includes the image processing apparatus 100, an infrared image capturing unit 210, a housing (not shown), and may further include a display unit 230. The camera system 310 includes the image processing apparatus 100 and the infrared image capturing unit 210, and may further include a display unit 230. The image processing system 200 may include an image acquisition unit 110, an area setting unit 120, and a processing unit 130.

  Note that the image processing apparatus 100, the image processing system 200, the camera 300, and the camera system 310 of the present embodiment are not limited to the configuration in FIG. 2, and some of the components are omitted or replaced with other components. Various modifications such as adding other components are possible.

  Since the image acquisition unit 110, the region setting unit 120, and the storage unit 150 are the same as those described in the first configuration example (FIG. 1), detailed description thereof is omitted here.

  The processing unit 130 performs processing for reducing the pixel density of the infrared image data in the pixel density changing region (pixel density reduction processing). An object detection process (template matching process) is performed based on the infrared image data of the image area including the pixel density change area, and display image data is generated based on the result of the detection process. This display image data may include an alert display based on the result of the detection process. In addition, the processing unit 130 performs a detection process, and performs a process of generating display image data by superimposing an alert symbol and visible image data based on a result of the detection process.

  Specifically, the processing unit 130 performs processing for reducing the pixel density on the infrared image data in the pixel density changing region ARMD set by the region setting unit 120. This process of reducing the pixel density (pixel density reduction process) is a process of equivalently reducing the pixel density on the image data, that is, the number of pixels per unit area, for example, an image constituting one screen (frame). This is a process (pixel mixture calculation process) of mixing image data corresponding to a plurality of adjacent pixels into image data corresponding to one pixel in a certain area (pixel density changing area) of the data. . On the other hand, pixel density reduction processing is not performed on infrared image data in the image area (pixel density non-change area) ARNO excluding the pixel density change area ARMD. The pixel density reduction process will be described in detail later.

  Further, the processing unit 130 performs an object detection process (template matching process) based on the infrared image data of the image area including the pixel density changing area ARMD, that is, the infrared image data after performing the pixel density reduction process. That is, in the pixel density changing area ARMD, detection processing for detecting an object (for example, a pedestrian or a bicycle) is performed based on infrared image data having a lower pixel density than the pixel density non-changing area ARNO. Alternatively, based on the infrared image data in which the pixel density is lower than the first threshold value in the pixel density change area ARMD and the pixel density is higher than the second threshold value in the pixel density non-change area ARNO, A detection process for detection is performed. Here, the first threshold value may be a value equal to or smaller than the second threshold value, and the second threshold value may be a value in a pixel density range that can be realized in the infrared image capturing unit 210. . The first and second threshold values may be the same value.

  The processing unit 130 includes a template matching processing unit 131, an image composition processing unit 132, two memories 133 and 135, and a mixing calculation processing unit 134. The memory 135 stores infrared image data from the image acquisition unit 110. The mixing calculation processing unit 134 performs pixel mixing calculation processing on image data corresponding to the pixel density changing region ARMD in the infrared image data stored in the memory 135. The memory 133 stores infrared image data after the pixel mixture calculation process is performed. By doing so, the pixel density of the pixel density changing region ARMD of the infrared image data stored in the memory 133 is lower than the first threshold value, and the pixel density of the pixel density non-changing region ARNO is the second threshold value. It becomes higher than the threshold. The template matching processing unit 131 performs template matching processing for detecting an object on the infrared image data after the pixel mixture calculation processing stored in the memory 133.

  The image composition processing unit 132 performs processing for superimposing an alert display (alert symbol) or the like on the visible image data based on the template matching processing result. The image composition processing unit 132 may perform a process of further compositing (superimposing) other information as necessary. The image composition processing unit 132 may perform processing for superimposing the alert display on the infrared image data, or may perform processing for combining (superimposing) the visible image data, the infrared image data, the alert display, and the like. .

  Since the infrared image capturing unit 210, the visible image capturing unit 220, the display unit 230, and the housing (not shown) are the same as those described in the first configuration example (FIG. 1), detailed description will be given here. Omitted.

  According to the image processing apparatus 100 and the like of the second configuration example illustrated in FIG. 2, the number of pixels of the infrared image data to be subjected to template matching processing can be reduced as in the first configuration example. It is possible to reduce the calculation processing burden in the processing. In addition, when an object is detected, an alert display with good visibility can be superimposed and displayed at a corresponding position in the visible image, so that it is possible to reduce the burden on the driver during night driving. .

  Further, similarly to the first configuration example, the region setting unit 120 determines a setting pattern based on the vehicle traveling state information, the vehicle position information, the input information by the user, and the like, and the boundary line in the determined setting pattern is determined. At least one of position and shape in the image region can be defined. By doing so, the pixel density changing area ARMD and the pixel density non-changing area ARNO can be variably set according to the traveling state of the vehicle and the like, without reducing the detection capability (detection accuracy) of the target object. An efficient image processing apparatus or the like can be realized.

  Further, in the second configuration example, since the pixel mixture calculation process can be performed as the pixel density reduction process, it is possible to prevent a decrease in sensitivity (infrared detection sensitivity) in the pixel density change region ARMD as described later.

  FIG. 3 shows a third configuration example of the image processing apparatus 100, the image processing system 200, the camera 300, and the camera system 310 of the present embodiment. The image processing apparatus 100 of the third configuration example includes an image acquisition unit 110, an area setting unit 120, a processing unit 130, and a mode setting unit 140, and may further include a storage unit 150. The image processing system 200 includes the image processing apparatus 100 and a display unit 230. The camera 300 includes the image processing apparatus 100, an infrared image capturing unit 210, a housing (not shown), and may further include a display unit 230. The camera system 310 includes the image processing apparatus 100 and the infrared image capturing unit 210, and may further include a display unit 230. The image processing system 200 may include an image acquisition unit 110, an area setting unit 120, and a processing unit 130.

  Note that the image processing apparatus 100, the image processing system 200, the camera 300, and the camera system 310 of the present embodiment are not limited to the configuration in FIG. 3, and some of the components are omitted or replaced with other components. Various modifications such as adding other components are possible.

  Since the image acquisition unit 110, the region setting unit 120, and the storage unit 150 are the same as those described in the first configuration example (FIG. 1), detailed description thereof is omitted here.

  Since the processing unit 130 is the same as that described in the first configuration example (FIG. 1) or the second configuration example (FIG. 2), detailed description thereof is omitted here.

  The mode setting unit 140 performs mode switching setting between the running mode and the crime prevention mode. The travel mode is a mode that is set while the vehicle is traveling, and is a mode in which a pedestrian or the like in front of the vehicle is detected and a warning is issued to the driver when the vehicle is traveling at night, for example. The crime prevention mode is a mode that is set while the vehicle is parked. For example, when the vehicle is parked adjacent to the home at night, an intruder entering the home is detected and a warning is issued. is there.

  When the image processing apparatus 100 is set to the travel mode, the region setting unit 120 determines the setting pattern as described above, determines the boundary line in the determined setting pattern, and sets the pixel density change region ARMD. Set. Then, the processing unit 130 performs a process of reducing the pixel density of the infrared image data in the pixel density change area ARMD, and performs a detection process based on the infrared image data of the image area including the pixel density change area ARMD. And the process which produces | generates the image data for a display containing the alert display based on the result of a detection process is performed.

  On the other hand, when the image processing apparatus 100 is set to the crime prevention mode, the region setting unit 120 does not set the boundary line. Then, the processing unit 130 performs a process of reducing the pixel density of the infrared image data for the entire area of the image area, performs a detection process based on the infrared image data of the image area, and includes an alert display based on a result of the detection process. Processing to generate display image data is performed. When a suspicious person is detected by the detection process, the processing unit 130 can perform a process of notifying the user's mobile terminal or the like. Details of the crime prevention mode will be described later.

  According to the image processing apparatus 100 or the like of the third configuration example shown in FIG. 3, when set to the travel mode, the infrared image that is the target of the template matching process as in the first and second configuration examples. Since the number of data pixels can be reduced, the calculation processing burden in the detection process can be reduced. In addition, when an object is detected, an alert display with good visibility can be superimposed and displayed at a corresponding position in the visible image, so that it is possible to reduce the burden on the driver during night driving. .

  Furthermore, in the third configuration example, by setting the crime prevention mode, it is possible to detect suspicious people around the home when parking at night, so it is not necessary to install a dedicated security system for crime prevention, It is possible to reduce crime prevention costs.

  FIG. 4 shows a fourth configuration example of the camera 300 of the present embodiment. The camera 300 of the fourth configuration example includes an infrared image capturing unit 210, an image acquisition unit 110, and a processing unit 130, and may further include a display unit 230. Note that the camera 300 of the present embodiment is not limited to the configuration of FIG. 4, and various modifications such as omitting some of the components, replacing them with other components, and adding other components. Implementation is possible.

  The infrared image capturing unit 210 is an infrared camera, and has sensitivity to far infrared rays having a wavelength of 8 to 12 μm, for example. Infrared sensor cameras using pyroelectric elements to detect distant objects (for example, 200 m away) are even better.

  The infrared image capturing unit 210 includes an imaging region in which infrared imaging elements are two-dimensionally arranged. In the partial region IRHD of the imaging region, the interval between the infrared imaging devices is the first interval, and the partial region of the imaging region. In the imaging region IRLD other than IRHD, the interval between the infrared imaging elements is a second interval wider than the first interval. That is, the pixel density of the infrared image is high in the imaging region IRHD, and the pixel density of the infrared image is low in the imaging region IRLD as compared with IRHD. Alternatively, the pixel density of the imaging region IRHD is higher than the first threshold value, and the pixel density of the imaging region IRLD is lower than the second threshold value. Here, the first threshold value is greater than or equal to the second threshold value, and both the first and second threshold values are values within the range of pixel density that can be realized in the infrared image capturing unit 210. is there. The first and second threshold values may be the same value.

  The interval between the infrared imaging elements is an arrangement interval between one pixel (infrared sensor element) and a pixel adjacent thereto on the imaging surface of the infrared imaging unit 210, for example. The pixel density is, for example, the number of pixels per unit area on the imaging surface of the infrared image capturing unit 210.

  The image acquisition unit 110 acquires the infrared image data captured by the infrared image capturing unit 210 and outputs the infrared image data to the processing unit 130.

  The processing unit 130 performs object detection processing (template matching processing) on the image region corresponding to the imaging region in the infrared image data acquired by the image acquisition unit 110 based on the infrared image data, and performs detection. Display image data is generated based on the processing result.

  The processing unit 130 includes a template matching processing unit 131 and an image composition processing unit 132. The template matching processing unit 131 performs template matching processing for detecting an object based on the infrared image data from the image acquisition unit 110. The image composition processing unit 132 performs processing for superimposing an alert display (alert symbol) or the like on the infrared image based on the template matching processing result. The image composition processing unit 132 may perform a process of further compositing (superimposing) other information as necessary.

  The image acquisition unit 110 and the processing unit 130 can be realized by a processor such as a CPU or an ASIC such as a gate array, for example.

  According to the camera 300 of the fourth configuration example shown in FIG. 4, a detection process (template matching process) is performed based on the infrared image data captured by the infrared camera (infrared image capturing unit) 210, and a pedestrian or the like is detected. An object can be detected. The image area of the infrared image data includes an image area corresponding to the imaging area IRHD having a high pixel density and an image area corresponding to the imaging area IRLD having a low pixel density. By doing so, it is possible to reduce the number of pixels of the infrared image data, and thus it is possible to reduce the calculation processing burden in the detection process.

  For example, in the imaging area of the infrared image capturing unit 210, an area where an object such as a pedestrian is likely to be imaged is set as the imaging area IRHD, and an area where the object is unlikely to be imaged is defined as the imaging area IRLD. By setting, it is possible to reduce the processing load without reducing the detection capability (detection accuracy) of the object. As a result, the processing time of the detection process can be shortened, the circuit configuration of the processing unit 130 can be simplified, and an efficient control system can be realized.

2. Pixel Density Change Area Setting and Detection Processing FIGS. 5A to 5C show setting examples of the pixel density change area ARMD in three driving situations. FIG. 5A shows a setting example when the setting pattern is determined to be the first setting pattern in a situation where the vehicle is traveling relatively smoothly, such as a suburban road. FIG. 5B shows a setting example when the setting pattern is determined to be the second setting pattern in a situation where the vehicle is traveling on a city road, for example. FIG. 5C shows a setting example when the setting pattern is determined as the third setting pattern in a situation where the vehicle is traveling on a road with a small amount of traffic such as a mountainous area. FIG. 5A to FIG. 5C are one frame (screen) of an infrared image in each traveling situation, and show the left end LT of the road, the right end RT of the road, and the center line CL.

  In the following description, in the infrared image from the infrared image capturing unit 210 (infrared camera) attached to the vehicle, the vertical direction (the direction of gravity) is “vertical direction”, and the direction orthogonal to the vertical direction is “horizontal direction”. And Alternatively, the direction perpendicular to the road surface may be “vertical direction” and the direction parallel to the road surface may be “horizontal direction”. Further, the vertical upward direction is “up”, the vertical downward direction is “down”, the right-hand horizontal direction toward the image is “right”, and the left-hand horizontal direction is “left”.

  In the case of the first setting pattern shown in FIG. 5A, the area setting unit 120 divides the image area using a boundary line BL that divides the image area based on the infrared image data. For example, as shown in FIG. 5A, the boundary line BL is set in the horizontal direction. Of the image areas divided by the boundary line BL set in the horizontal direction, the upper image area (one image area in a broad sense) is set as the pixel density non-change area ARNO, and the lower image area (broad sense). Is set to the pixel density changing area ARMD.

  In the traveling state shown in FIG. 5A, it is important to detect distant pedestrians (for example, 40 m or more ahead) and bicycles that are difficult to visually recognize at night. Therefore, a boundary line BL is set in the horizontal direction, and an area above the boundary line BL, that is, an area where a distant object is projected is set as a pixel density non-change area ARNO. On the other hand, the area below the boundary line BL is set to the pixel density change area ARMD. In this way, for example, a pedestrian (A1 in FIG. 5A) or a person riding a bicycle (A2 in FIG. 5A) who wants to cross the road is detected using infrared image data with a high pixel density. can do.

  In the case of the second setting pattern shown in FIG. 5B, the area setting unit 120 divides the image area based on the infrared image data into three and divides the image area using two boundary lines that do not intersect each other. For example, as shown in FIG. 5B, two boundary lines BL1 and BL2 are set in the vertical direction. Then, among the image areas divided into three by the boundary lines BL1 and BL2 set in the vertical direction, the central image area (image area sandwiched between two boundary lines in a broad sense) is set as the pixel density change area ARMD. .

  In the driving situation shown in FIG. 5B, it is important to detect pedestrians, bicycles, and the like that jump out of the roadside at short distances that are easily obscured by the headlights of oncoming vehicles. Therefore, the first and second boundary lines BL1 and BL2 are set in the vertical direction, and the area on the left side of the first boundary line BL1 and the area on the right side of the second boundary line BL2 are both in the pixel density non-change area ARNO. Set to. On the other hand, a region sandwiched between two boundary lines BL1 and BL2, that is, a central region is set as a pixel density changing region ARMD. In this way, for example, a pedestrian (B1 in FIG. 5B) or a person riding a bicycle (B2 in FIG. 5B) who wants to cross the vehicle is used infrared image data with a high pixel density. Can be detected.

  In the case of the third setting pattern shown in FIG. 5C, the area setting unit 120 divides the image area using a closed curve surrounding a part of the image area based on the infrared image data as a boundary line. For example, as shown in FIG. 5B, a boundary line BL that is a closed curve is set at the center of the image area. Then, the image area outside the closed curve is set as the pixel density changing area ARMD. Here, the closed curve may be a closed figure, for example, a polygon.

  In the traveling situation shown in FIG. 5C, it is important to detect a farther object (for example, 100 m or more ahead) in consideration of the high vehicle speed. Therefore, a limited area such as the center of the screen is set as the pixel density non-change area ARNO, and the other area (peripheral area of the screen) is set as the pixel density change area ARMD. By doing so, it is possible to detect, for example, a pedestrian (C1 in FIG. 5C) or the like who wants to cross a distant place of the vehicle using infrared image data having a high pixel density.

  The position of the boundary line is not fixed, and can be variably set based on, for example, vehicle speed information or headlight control information. Specifically, for example, in FIG. 5A, the position of the boundary line may be shifted upward as the vehicle speed increases, or the position of the boundary line is changed when the headlight is switched to a high beam. When shifting to the upper part of the screen and switching to the low beam, the position of the boundary line may be shifted to the lower part of the screen.

  Further, the position, shape, and the like of the boundary line can be variably set based on information input by the user (driver). Specifically, when the driver inputs boundary line setting information according to traffic conditions, for example, in FIG. 5B, the positions of the two boundary lines BL1 and BL2 are shifted to the left or right, For example, in FIG. 5C, the position or size of the closed curve (boundary line) BL at the center of the screen can be changed.

  FIGS. 6A, 6 </ b> B, and 7 are diagrams illustrating pixel density reduction processing (pixel thinning-out processing) according to the first configuration example (FIG. 1) of the image processing apparatus 100. In these drawings, a case where infrared image data for one screen (frame) acquired from the infrared image capturing unit 210 corresponds to pixels of 12 rows × 14 columns will be described as an example. In the following description, for example, the pixel in the second row and the third column is expressed as P (2,3). Further, reading the image data of the pixel P (2,3) and storing the image data in the memory 133 is referred to as “reading P (2,3)” for convenience.

  In FIG. 6A, among the regions divided by the two boundary lines BL1 and BL2 set in the vertical direction, both the left region and the right region are set as the pixel density non-change region ARNO, This is a case where a region (a region sandwiched between two boundary lines BL1 and BL2) is set as the pixel density changing region ARMD. The processing unit 130 performs pixel thinning processing as follows and stores the image data in the memory 133. Note that the positions of the boundary lines BL1 and BL2 are not limited to those illustrated, and may be set to other positions.

  The processing unit 130 sequentially reads pixels P (1,1) to P (1,4) in the first row. Next, P (1,5) is skipped without reading, and P (1,6) is read. Next, skip P (1,7) without reading P (1,8). Next, P (1,9) is skipped without reading, and P (1,10) is read. Then, P (1,11) to P (1,14) are read in order. Next, for the pixels in the second row, P (2,1) to P (2,4) are read, P (2,5) to P (2,10) are skipped, and P (2,11) to Read P (2,14). For the pixels in the third row, as in the first row, P (3,1) to P (3,4), P (3,6), P (3,8), P (3,10) , P (3, 11) to P (3, 14) are read, and other pixels are skipped. For the fourth and subsequent rows, pixel thinning processing is performed in the same manner as described above.

  By doing so, as shown in FIG. 6A, the pixels are not thinned out for the pixel density non-change region ARNO, and the number of pixels is reduced to ¼ for the pixel density change region ARMD. Density is reduced to 1/4. Image data corresponding to the pixels shaded in FIG. 6A is stored in the memory 133.

  FIG. 6B shows a case where the area above the boundary line BL set in the horizontal direction is set as the pixel density non-change area ARNO, and the area below the boundary line BL is set as the pixel density change area ARMD. is there. The processing unit 130 performs pixel thinning processing as follows and stores the image data in the memory 133. Note that the position of the boundary line BL is not limited to the illustrated one, and may be set to another position.

  For the pixel density non-change area ARNO, the processing unit 130 sequentially reads out pixels from the first row to the sixth row without performing the thinning process. As for the pixel density changing region ARMD, every other pixel is read out in the odd-numbered rows, and all pixels are skipped in the even-numbered rows. For example, in the seventh row, P (7,2), P (7,4), P (7,6), P (7,8), P (7,10), P (7,12), P ( 7, 14) are read out and all pixels are skipped in the eighth row.

  By doing so, as shown in FIG. 6B, the pixels are not thinned out for the pixel density non-change area ARNO, and the number of pixels is reduced to ¼ for the pixel density change area ARMD. Density is reduced to 1/4. Image data corresponding to the shaded pixels in FIG. 6B is stored in the memory 133.

  In FIG. 7, the area surrounded by the boundary line BL set at the center is set as the pixel density non-change area ARNO, and the area outside the boundary line BL set at the center is set as the pixel density change area ARMD. This is the case. The processing unit 130 performs pixel thinning processing as follows and stores the image data in the memory 133. Note that the position of the boundary line BL is not limited to the illustrated one, and may be set to another position.

  The processing unit 130 reads all the pixels for the pixel density non-change area ARNO. As for the pixel density changing region ARMD, every other pixel is read out in the odd-numbered rows, and all pixels are skipped in the even-numbered rows. For example, in the first row, P (1,2), P (1,4), P (1,6), P (1,8), P (1,10), P (1,12), P ( 1,14) is read and all pixels are skipped in the second row. In the fifth row, P (5,2), P (5,4), P (5,5) to P (5,10), P (5,12), and P (5,14) are read.

  By doing so, as shown in FIG. 7, the pixels are not thinned out for the pixel density non-change area ARNO, and the number of pixels is reduced to ¼ for the pixel density change area ARMD, that is, the pixel density is 1 Reduced to / 4. Image data corresponding to the pixels shaded in FIG. 7 is stored in the memory 133.

  8A, FIG. 8B, FIG. 9A, and FIG. 9B are pixel density reduction processing (pixel mixture calculation processing) according to the second configuration example (FIG. 2) of the image processing apparatus 100. FIG. FIG. In these drawings, a case where infrared image data for one screen (frame) acquired from the infrared image capturing unit 210 corresponds to pixels of 12 rows × 14 columns will be described as an example. In the following description, for example, the pixel in the second row and the third column is expressed as P (2,3).

  As shown in FIG. 8A, first, the processing unit 130 reads out all the pixels and stores the image data in the memory 135. Then, as described below, the pixel mixture calculation process is performed, and the result is stored in the memory 133.

  In FIG. 8B, among the areas divided by the two boundary lines BL1 and BL2 set in the vertical direction, both the left area and the right area are set as the pixel density non-change area ARNO. This is a case where a region (a region sandwiched between two boundary lines BL1 and BL2) is set as the pixel density changing region ARMD. Note that the positions of the boundary lines BL1 and BL2 are not limited to those illustrated, and may be set to other positions.

  The processing unit 130 stores the image data as it is in the memory 133 without performing the pixel mixture calculation process for the pixel density non-change area ARNO. On the other hand, for the pixel density changing region ARMD, the image data of the four pixels is subjected to the mixing calculation process, and the result is stored in the memory 133 as the image data of one pixel. For example, image data is mixed and processed for four pixels P (1,7), P (2,7), P (1,8), and P (2,8), and the result is displayed as one pixel P (1 , 8) is stored in the memory 133 as image data. In FIG. 8B, four pixels as units of the mixing calculation process are indicated by A, B, C, and D, and the processing result is stored in the memory 133 as image data of the pixel D. By doing so, the pixel density does not change for the pixel density non-change area ARNO, and the number of pixels for the pixel density change area ARMD is reduced to ¼, that is, the pixel density is reduced to ¼. .

  FIG. 9A shows a case where the area above the boundary line BL set in the horizontal direction is set as the pixel density non-change area ARNO, and the area below the boundary line BL is set as the pixel density change area ARMD. is there. Note that the position of the boundary line BL is not limited to the illustrated one, and may be set to another position.

  Also in this case, as in FIG. 8B, the processing unit 130 stores the image data as it is in the memory 133 without performing the pixel mixture calculation process for the pixel density non-change region ARNO. On the other hand, for the pixel density changing region ARMD, the image data of the four pixels is subjected to the mixing calculation process, and the result is stored in the memory 133 as the image data of one pixel. For example, the image data is mixed and processed for four pixels P (9, 1), P (10, 1), P (9, 2), and P (10, 2), and the result is converted into one pixel P (9 , 2) is stored in the memory 133 as image data. By doing so, the pixel density does not change for the pixel density non-change area ARNO, and the number of pixels for the pixel density change area ARMD is reduced to ¼, that is, the pixel density is reduced to ¼. .

  In FIG. 9B, the area surrounded by the boundary line BL set at the center is set as the pixel density non-change area ARNO, and the area outside the boundary line BL set at the center is the pixel density change area. This is a case of setting to ARMD. Note that the position of the boundary line BL is not limited to the illustrated one, and may be set to another position.

  Also in this case, as in FIGS. 8B and 9A, the processing unit 130 stores the image data as it is in the memory 133 without performing the pixel mixture calculation process for the pixel density non-change area ARNO. On the other hand, for the pixel density changing region ARMD, the image data of the four pixels is subjected to the mixing calculation process, and the result is stored in the memory 133 as the image data of one pixel. For example, the image data is mixed and processed for four pixels P (1,1), P (2,1), P (1,2), and P (2,2), and the result is converted into one pixel P (1 , 2) is stored in the memory 133 as image data. By doing so, the pixel density does not change for the pixel density non-change area ARNO, and the number of pixels for the pixel density change area ARMD is reduced to ¼, that is, the pixel density is reduced to ¼. .

  According to the pixel mixture calculation processing described above, the image data of the four pixels A, B, C, and D are mixed to form the image data of one pixel D. Therefore, the sensitivity of one pixel D is equivalent to 4 Will be doubled. Therefore, in the pixel density changing region ARMD, even when the pixel density is ¼, the sensitivity per unit area (infrared detection sensitivity) does not change.

  FIG. 10 is a flowchart of detection processing (template matching processing) in the image processing apparatus 100 or the like according to this embodiment. This template matching process is executed by the template matching processing unit 131 included in the processing unit 130.

  In the first step S1, parameters necessary for processing are initialized. In the next step S <b> 2, the image acquisition unit 110 acquires an infrared image from the infrared camera (infrared image capturing unit) 210. In the next step 3, a pixel density change region ARMD is set for each frame (one screen) of infrared image (thermal image) data based on the boundary line setting information from the region setting unit 120, and the pixel A pixel density reduction process is performed on the image data in the density change area ARMD. Note that in the case of the fourth configuration example (FIG. 4) of the camera 300, the image data acquired from the infrared image capturing unit 210 has an image area corresponding to the image area IRHD with a high pixel density and an image with a low pixel density. This step 3 may be omitted because it includes the image area corresponding to the area IRLD.

  Next, in step S4, a detection candidate area is extracted from the infrared image data after the pixel density reduction process. Specifically, a temperature region representing the characteristics of the object is extracted as a lump. Here, as an example, a case where a preceding vehicle and a pedestrian (person) are set as detection targets will be described. When a target other than this is set as a detection target, the temperature information of the target may be added as a feature amount to the processing target in this step.

  In the case of a preceding vehicle, by using the fact that the muffler region becomes a high temperature of about 100 ° C, a region having temperature information of, for example, 80 ° C or more is searched from the infrared image, and the position information and temperature of the region are searched. Detect information.

  On the other hand, in the case of a pedestrian, using the fact that the surface temperature of the exposed face and neck becomes a medium temperature of about 30 ° C, a region having temperature information of about 30 ° C is searched from the thermal image. Then, position information and temperature information of the area are detected. Here, among the results detected in each region, the position information may be, for example, the barycentric coordinate position of the lump, and the temperature information may be, for example, the average temperature of the lump. In addition to the center-of-gravity coordinate position, the existence of a part satisfying conditions such as the aspect ratio, the sufficiency rate, and the actual area is examined. As a result, the position of the pedestrian is obtained as a ratio to the width and height of the entire screen on the two-dimensional coordinates on the screen.

  In step S5, a position and temperature list in which the position information and temperature information of each region obtained in step S4 are associated is created, and the process proceeds to the subsequent processes.

  In the next step S6, based on the list created in step S5, it is determined whether or not there is a detection candidate, that is, whether or not temperature information indicating the feature amount of the preceding vehicle or pedestrian that is the detection target is extracted. To do. If no candidate object is extracted from the list, it is determined that there is no preceding vehicle or pedestrian to be detected in the current infrared image, and the next frame (screen) The process returns to step S2 for the process.

  On the other hand, if one or more candidate objects are extracted from the list, a detection candidate area (existing candidate area) is set in the next step S7, and the list created in step S5 is set in the subsequent step S8. Based on the temperature information, a template candidate to be used in the subsequent processing is selected. Specifically, a template having a value within the range of the average temperature information ± ΔT of the detection candidate area is set as a template candidate used for object detection within one detection candidate area.

  For example, a template having temperature information of 100 ± 3 ° C. is selected as a template used for the preceding vehicle presence candidate region. However, since the high temperature portion is special in the normal driving environment, an area of 80 ° C. or more may be selected in advance as a template. On the other hand, a template having a temperature information of 28 ± 3 ° C. is selected as a template used for the pedestrian presence candidate region. Regarding the high temperature part, it is necessary to consider that the temperature changes within a predetermined range even for the same object in consideration of the influence of the outside air temperature. In the above example, the object having intermediate temperature information is a pedestrian, but in order to distinguish it from an animal (such as a dog), an animal template is also selected as a candidate in this step.

  In this way, an optimal template is determined for each detection candidate region, and in the next step S9, detection processing (template matching processing) is performed using the determined template. That is, template matching processing is performed in the infrared image to detect the final presence or absence of the target object and its position in each detection candidate region. In this step, when a preceding vehicle is detected from the preceding vehicle presence candidate area, the presence / absence of the preceding vehicle is determined by the correlation calculation of the density value (gray value) in the preceding vehicle existence candidate area and the body template using the body template. The detailed position is detected. Further, when detecting a pedestrian from the pedestrian presence candidate area, the presence / absence of the pedestrian and its detailed position are detected by correlation calculation between the pedestrian existence candidate area and the density value in the human template.

  In the above description, the template matching process is performed by the correlation calculation using the density value during the template matching process. However, the image processing apparatus 100 according to the present embodiment is not limited to this, for example, an edge image. As a template, and the presence / absence of the final object and its position may be detected by this correlation value.

  Next, in step S10, it is determined whether or not detection processing has been performed for all of the detection candidate regions in the list created in step S5. If the processing has not been completed for all detection candidate areas, the process returns to step S7 in order to switch the processing areas, and the next detection candidate area is set. If the processing has been completed for all the areas in the list, the process proceeds to the next step S11 in order to output the processing results up to that point.

  In step S <b> 11, the template matching processing unit 131 outputs the detection result described above to the image composition processing unit 132. When the target is detected, the image composition processing unit 132 performs processing for superimposing the alert display on the visible image, and outputs it to the display unit 230.

  According to the image processing apparatus 100 and the like of the present embodiment, the processing unit 130 detects an object in the pixel density change region ARMD and the region other than the pixel density change region (that is, the pixel density non-change region ARNO). The contents of the detection process can be made different. Specifically, for example, in the template selection in step S8, a template with high accuracy (resolution) is selected for the detection candidate area of the pixel density non-change area ARNO, and the detection candidate area of the pixel density change area ARMD is selected. Can select a template with low accuracy (resolution). By doing so, a highly accurate detection process can be performed for the pixel density non-change area ARNO, and for the pixel density change area ARMD, the processing time can be shortened although the accuracy is reduced. More efficient detection processing (template matching processing) can be performed.

  Further, according to the image processing apparatus 100 or the like of the present embodiment, the processing unit 130 uses the detection processing result in the area other than the pixel density change area (pixel density non-change area ARNO) in the pixel density change area ARMD. Can be detected. Specifically, the processing unit 130 can perform the detection process in the pixel density change area ARMD based on the motion vector of the object obtained by the detection process in the pixel density non-change area ARNO. This motion vector is a vector that represents the speed and direction in which the object moves when the object such as a pedestrian moves.

  For example, as illustrated in B1 of FIG. 5B, when a pedestrian is detected in the pixel density non-change area ARNO, when the pedestrian's motion vector is directed rightward, the processing unit 130 Based on this motion vector, it can be predicted that the pedestrian moves to the pixel density changing region ARMD. And a pedestrian presence candidate area | region can be preset and the person template can be selected in the movement destination of a pedestrian. By doing so, it is possible to shorten the pedestrian detection processing time in the pixel density changing region ARMD.

  Furthermore, according to the image processing apparatus 100 or the like of the present embodiment, the processing unit 130 detects an object in the pixel density non-change area ARNO and predicts that the object moves to the pixel density change area ARMD. In the pixel density change area ARMD, detection processing can be performed based on infrared image data having a higher pixel density than that in a case where the pixel density change area ARMD is not predicted.

  When an object is detected in the pixel density non-change area ARNO, whether or not the object moves to the pixel density change area ARMD is predicted based on, for example, the motion vector of the object. It is executed by the determination process.

  For example, as indicated by B1 in FIG. 5B, a pedestrian is detected in the pixel density non-change area ARNO, and the pedestrian moves to the pixel density change area ARMD based on a motion vector in the right direction. When the determination process for predicting is performed, the processing unit 130 may not perform (omit) the pixel density reduction process on the infrared image data in the pixel density change region ARMD. By doing so, the same pixel density as the pixel density non-change area ARNO (in a broad sense, the object moves to the pixel density change area ARMD) even for an object (such as a pedestrian) moved to the pixel density change area ARMD. Since the detection process can be performed based on infrared image data having a higher pixel density than in a case where it is not predicted, the position of the object can be detected more accurately.

  FIGS. 11A to 11C show examples of alert display in three driving situations. 11 (A) to 11 (C) show images displayed on the display unit 230, and the first traveling situation (FIG. 5 (A)) and the second traveling situation (FIG. 5 (B)) described above. ), Respectively corresponding to the third traveling situation (FIG. 5C).

  As shown in FIG. 11 (A), alert symbols AM1 and AM2 for informing the driver of the presence of a pedestrian and a bicycle are superimposed on the visible image at a position where a pedestrian and a bicycle rider are detected. Is displayed.

  In FIG. 11B, an alert symbol AM1 for notifying the presence of a pedestrian is displayed superimposed on the visible image at the position where the pedestrian is detected, and the presence of the bicycle is displayed at the position where the bicycle is detected. An alert symbol AM2 for notification is displayed superimposed on the visible image.

  In FIG. 11C, an alert symbol AM1 for informing the presence of a pedestrian is displayed superimposed on the visible image at a position where a pedestrian in the center is detected.

3. Security Mode FIG. 12 is a diagram for explaining a security mode in the third configuration example (FIG. 3) of the image processing apparatus 100 and the like of the present embodiment. As shown in FIG. 12, for example, when a private vehicle is parked near the home at night, the intrusion of a suspicious person into the home at night is detected by setting the image processing apparatus 100 or the like to the crime prevention mode. Can do. The mode setting unit 140 switches between the running mode and the crime prevention mode.

  Specifically, an infrared image capturing unit (infrared camera) 210 provided on the roof of the vehicle or the like captures infrared images around the home at predetermined time intervals. When the crime prevention mode is set, the area setting unit 120 does not set the boundary line, and the processing unit 130 performs a process of reducing the pixel density of the infrared image data for the entire area of the infrared image area. Then, the processing unit 130 performs a detection process based on the infrared image data of the image area, and performs a process of generating display image data including an alert display based on the result of the detection process. When the processing unit 130 detects a suspicious person by the detection process, the processing unit 130 can perform a process of reporting to a user's mobile terminal, for example.

  In the security mode, higher accuracy is not required for the position detection of the object than in the traveling mode. This is because it is only necessary to detect the presence of a suspicious person. Therefore, by performing the process of reducing the pixel density on the entire area of the image area, it is possible to reduce the calculation processing burden in the detection process. As a result, power consumption in the crime prevention mode can be reduced, so that battery consumption during parking can be suppressed.

  The optical axis direction of the infrared camera 210 can be variably set by providing the infrared camera 210 on a roof rail or the like and moving the position of the infrared camera 210 or by providing the infrared camera 210 on a pedestal capable of rotating. . By doing so, it is possible to set the optical axis direction of the infrared camera 210 in a direction in which a suspicious person may enter.

  FIG. 13 is a flowchart of the security mode process according to the third configuration example (FIG. 3). This process is executed by the image acquisition unit 110 and the processing unit 130.

  In the first step S21, parameters necessary for processing are initialized. In the next step S22, it is determined whether the set mode is the traveling mode or the crime prevention mode. If the crime prevention mode is set, the next step S23 is executed. On the other hand, when the travel mode is set, the processes after step S2 in the travel mode flowchart (FIG. 10) described above are executed.

  In step S <b> 23, the image acquisition unit 110 acquires infrared image data from the infrared camera (infrared image capturing unit) 210. In the next step S24, the processing unit 130 performs a process for reducing the pixel density on the entire image area of the infrared image data, and subsequently executes an object detection process (template matching process) in step S25. . Since this detection process is the same process as steps S4 to S10 of FIG. 10 already described, detailed description thereof is omitted here. In the crime prevention mode, a higher accuracy is not required for detecting the position of the object than in the traveling mode, so that a template with a lower accuracy (resolution) can be selected.

  When the detection process ends, the process proceeds to step S26 to determine whether or not an object has been detected. If an object is detected, in step S27, for example, a process of notifying the outside such as a user's portable terminal is performed.

  If no object is detected, the process returns to step S22. If the security mode is still set, the next infrared image data is acquired in step S23 after a predetermined time has elapsed since the previous acquisition of the infrared image data. By acquiring infrared image data at regular time intervals, power consumption can be reduced, so that battery consumption during parking can be suppressed.

  As described above, according to the third configuration example of the image processing apparatus 100 and the like of the present embodiment, by setting the crime prevention mode, it is possible to detect suspicious persons around the home when parking at night. Since it is no longer necessary to install a dedicated security system, it is possible to reduce crime prevention costs. Further, since power consumption can be reduced in the crime prevention mode, battery consumption during parking can be suppressed.

4). Camera System FIGS. 14A and 14B show an example of mounting the camera system 310 of this embodiment on a vehicle. A camera system 310 shown in FIGS. 14A and 14B includes an image processing apparatus 100, an infrared image capturing unit 210, a visible image capturing unit 220, and a display unit 230. Note that the camera system 310 of the present embodiment is not limited to the configuration shown in FIGS. 14A and 14B, and some of the components may be omitted or replaced with other components. Various modifications such as adding components are possible.

  FIG. 14A is a plan view of the vehicle as viewed from above, and FIG. 14B is a side view of the vehicle. The infrared image capturing unit (infrared camera) 210 is provided, for example, in a bumper portion of the vehicle, and the visible image capturing unit (visible camera) 220 is provided, for example, in a roof of the vehicle. The image processing apparatus 100 is provided at a position where a driver such as a dashboard can be easily operated, and the display unit 230 is provided at a position where the driver can easily see. The mounting positions of these components are not limited to the illustrated positions, and for example, an infrared image capturing unit (infrared camera) 210 may be provided on the roof. The display unit 230 may be a head-up display.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Therefore, all such modifications are included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configuration and operation of the image processing apparatus, the image processing system, the camera, and the camera system are not limited to those described in the present embodiment, and various modifications can be made.

100 image processing apparatus, 110 image acquisition unit, 120 area setting unit, 130 processing unit,
131 Template matching processing unit, 132 Image composition processing unit, 133 Memory,
134 mixing operation processing unit, 135 memory, 140 mode setting unit, 150 storage unit,
200 image processing system, 210 infrared image capturing unit, 220 visible image capturing unit,
230 display unit, 300 camera, 310 camera system,
ARMD pixel density change area, ARNO pixel density non-change area, BL boundary line

Claims (21)

An image acquisition unit for acquiring infrared image data;
For an image region based on the infrared image data acquired by the image acquisition unit, a boundary line for dividing the image region into a plurality of regions is set, and at least of the plurality of regions divided by the boundary line An area setting unit for setting one area as a pixel density change area;
A process of reducing the pixel density of the infrared image data in the pixel density change area, a detection process of an object is performed based on the infrared image data of the image area including the pixel density change area, and the detection process And a processing unit that generates display image data based on the result. In claim 1,
The region setting unit
Determining a setting pattern that divides the image area using the boundary line in which at least one of a position and a shape in the image area is variably set;
An image processing apparatus, wherein at least one of a position and a shape of the boundary line in the image area is determined in the determined setting pattern. In claim 2,
The region setting unit
A first setting pattern for partitioning the image area using the boundary line that divides the image area can be determined,
When the determined setting pattern is the first setting pattern, one of the image areas divided by the boundary line is set as the pixel density change area. Processing equipment. In claim 2 or 3,
The region setting unit
A second setting pattern that divides the image area using two boundary lines that divide the image area into three and do not intersect with each other can be determined,
When the determined setting pattern is the second setting pattern, an image area sandwiched between the two boundary lines among the image areas divided into three by the boundary line is set as the pixel density change area. An image processing apparatus characterized by setting. In any of claims 2 to 4,
The region setting unit
A third setting pattern that partitions the image area using a closed curve that surrounds a part of the image area as the boundary line can be determined,
When the determined setting pattern is the third setting pattern, the image area outside the closed curve is set as the pixel density change area. In any of claims 2 to 5,
The image processing apparatus, wherein the display image data includes an alert display based on a result of the detection process. In any one of Claims 2 thru | or 6.
The infrared image data is image data of an infrared image of an external area of the vehicle,
The region setting unit
The image processing characterized in that the setting pattern is determined based on the traveling state information of the vehicle, and at least one of a position and a shape of the boundary line in the image region is determined in the determined setting pattern. apparatus. In claim 7,
The vehicle running status information is:
An image processing apparatus characterized by the speed information of the vehicle. In any of claims 2 to 8,
The region setting unit
An image processing apparatus, wherein the setting pattern is determined based on input information by a user, and at least one of a position and a shape of the boundary line in the image region is determined in the determined setting pattern. In any one of Claims 1 thru | or 9,
It further includes a mode setting unit for setting mode switching between the driving mode and the crime prevention mode,
When the travel mode is set,
The region setting unit sets the pixel density change region,
The processing unit performs a process of reducing the pixel density of the infrared image data in the pixel density change area, performs the detection process based on the infrared image data of the image area including the pixel density change area, and Perform processing to generate image data for display including alert display based on the result of detection processing,
If the security mode is set,
The region setting unit unsets the boundary line,
The processing unit performs a process of reducing the pixel density of the infrared image data for the entire area of the image area, performs the detection process based on the infrared image data of the image area, and results of the detection process An image processing apparatus for performing display image data generation including alert display based on the above. In any one of Claims 1 thru | or 10.
The processor is
The image processing apparatus according to claim 1, wherein contents of the detection process are different between the pixel density change area and an area other than the pixel density change area. In claim 11,
The processor is
An image processing apparatus that performs the detection processing in the pixel density change region based on a result of the detection processing in a region other than the pixel density change region. In claim 12,
The result of the detection process in an area other than the pixel density change area is:
An image processing apparatus comprising a motion vector of the object. In any one of Claims 1 thru | or 13.
The image acquisition unit
Get visible image data,
The processor is
An image processing apparatus that performs the detection process, and performs a process of generating display image data by superimposing an alert symbol based on a result of the detection process and the visible image data. An image processing device according to any one of claims 1 to 14,
A camera system comprising an infrared imaging unit. An image processing device according to any one of claims 1 to 14,
An infrared imaging unit;
A camera comprising: a housing in which the image processing device and the infrared image capturing unit are disposed. An image processing device according to any one of claims 1 to 14,
An infrared imaging unit;
And a display unit that displays an image based on the display image data. An image processing device according to any one of claims 1 to 14,
An infrared imaging unit;
A camera comprising: a display unit that displays an image based on the display image data; and a housing in which the image processing device, the infrared imaging unit, and the display unit are disposed. An image acquisition unit for acquiring infrared image data;
A boundary line for dividing the image region into a plurality of regions is set for the image region based on the infrared image data acquired by the image acquisition unit, and at least one of the plurality of regions divided by the boundary line An area setting unit for setting one area as a pixel density change area;
A process of reducing the pixel density of the infrared image data in the pixel density change area, a detection process of an object is performed based on the infrared image data of the image area including the pixel density change area, and the detection process And a processing unit that generates display image data based on the result. An image processing apparatus according to any one of claims 1 to 13,
An image processing system comprising: a display unit configured to display an image based on the display image data. An infrared imaging device is provided with an imaging region in which two-dimensional infrared imaging devices are arranged. In a region of the imaging region, the interval of the infrared imaging device is a first interval, and in a region other than the partial region of the imaging region. An infrared image capturing unit in which the interval between the infrared imaging elements is a second interval wider than the first interval;
An image acquisition unit for acquiring infrared image data captured by the infrared image capturing unit;
An object detection process is performed on the image area corresponding to the imaging area of the infrared image data acquired by the image acquisition unit based on the infrared image data, and display is performed based on the result of the detection process. And a processing unit that generates image data.

Images