JP2006317193A - Image processing apparatus, imaging processing method and program for processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image processing method and a program for processing images capable of multilaterally and effectively processing various information contained in the images to be captured. <P>SOLUTION: The image processing apparatus comprises: an imaging means for photographing the predetermined visual field for forming images; a travel state detection means for detecting the travel state of the vehicle; a closed region setting means for setting a plurality of closed regions each of which makes a sum set of the whole visual fields to the images corresponding to the travel state of the vehicle detected by the travel state detection means; and a image processing means for performing the prescribed image processing for every closed region set by the closed region setting means. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for performing image processing on an image generated by imaging a predetermined visual field.

  2. Description of the Related Art Conventionally, there has been known a technique for capturing the traveling direction of a vehicle and recognizing a predetermined object from the captured image for the purpose of detecting the road surface condition in the traveling direction when the vehicle such as an automobile is traveling. (For example, see Patent Document 1). In this technique, a road segment line such as a white line and a road segment zone such as a median strip on the road on which the vehicle travels are recognized using the captured image.

Japanese Patent No. 3290318

  However, in the above-described prior art, since the processing content is specialized for recognition of white lines or the like, various information included in the image to be captured cannot be processed in a multifaceted manner.

  The present invention has been made in view of the above, and an object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of processing various information included in an image to be captured in a multifaceted manner. To do.

  In order to solve the above-described problems and achieve the object, an invention according to claim 1 is provided with an imaging unit mounted on a moving body, which captures a predetermined field of view and generates an image, and a moving state of the moving body. A moving state detecting unit for detecting, and a closed region setting unit for setting, for the image, a plurality of closed regions having the entire field of view as a union according to the moving state of the moving body detected by the moving state detecting unit; And image processing means for performing predetermined image processing for each closed area set by the closed area setting means.

  According to a second aspect of the present invention, in the first aspect of the present invention, the moving state of the moving body is at least one of a current state relating to the movement of the moving body and a surrounding current state in which the moving body moves. It is characterized by including.

  The invention according to claim 3 is the invention according to claim 1 or 2, further comprising object detection means for detecting a predetermined object for each closed area set by the closed area setting means. To do.

  According to a fourth aspect of the present invention, in the invention according to any one of the first to third aspects, an image processing technique applied to the image processing performed by the image processing unit is set for each closed region set by the closed region setting unit. It further comprises a process changing means for changing.

  According to a fifth aspect of the present invention, in the invention according to any one of the first to fourth aspects, the imaging unit is configured to electrically output a pair of imaging optical systems and an optical signal output from each of the pair of imaging optical systems. And a pair of imaging elements that convert signals.

  According to a sixth aspect of the present invention, in the invention according to any one of the first to fourth aspects, the imaging unit includes an imaging optical system and an imaging device that converts an optical signal output from the imaging optical system into an electrical signal. An imaging apparatus comprising: a stereo adapter that is provided in front of the imaging optical system, receives light at a plurality of parts, and guides the light received by the plurality of parts to the imaging optical system, respectively. To do.

  The invention according to claim 7 is the invention according to any one of claims 1 to 6, wherein the moving body is a vehicle.

  The invention according to claim 8 is an image processing method for performing image processing on an image mounted on a moving body and generated by an image pickup unit that picks up a predetermined field of view, and detects a moving state of the moving body. A moving state detecting step, a closed region setting step for setting a plurality of closed regions with the entire field of view as a union according to the moving state of the moving body detected in the moving state detecting step, and the closed region setting step And an image processing step for performing predetermined image processing for each closed region set in (5).

  According to the ninth aspect of the present invention, in order to perform image processing on an image that is mounted on a moving body and that is generated by an imaging unit that captures a predetermined field of view, a moving state for detecting a moving state of the moving body on a computer Set in the detection step, a closed region setting step for setting a plurality of closed regions with the entire field of view as a union according to the moving state of the moving body detected in the moving state detection step, and the closed region setting step And an image processing step for performing predetermined image processing for each closed region.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an image processing program that can process various information included in an image to be captured in a multifaceted manner.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention. An image processing apparatus 1 shown in FIG. 1 is an electronic device (computer), and is mounted on a moving body, particularly a vehicle such as a four-wheeled vehicle. The imaging unit 10 captures a predetermined field of view, and is generated by the imaging unit 10. An image analysis unit 20 that analyzes an image, a control unit 30 that controls the operation of the image processing apparatus 1, an output unit 40 that displays and outputs information such as images and characters, a storage unit 50 that stores various data, and a vehicle movement state It has the movement condition detection part 60 which detects.

  The imaging unit 10 is realized by a compound-eye stereo camera, and includes a right camera 11a and a left camera 11b. Among these, the right camera 11a is a lens 12a that collects light incident from a predetermined viewing angle, a CCD (Charge Coupled Device) that detects light transmitted through the lens 12a and converts it into an electrical signal (analog signal) or the like. An image sensor 13a such as a CMOS (Complementary Metal Oxide Semiconductor), an A / D converter 14a that converts an analog signal output from the image sensor 13a into a digital signal, and a frame memory 15a that temporarily stores image data for several memories. Have On the other hand, the left camera 11b has the same configuration as the right camera 11a. That is, the left camera 11b includes a lens 12b, an image sensor 13b, an A / D converter 14b, and a frame memory 15b.

  The lenses 12a and 12b as a pair of imaging optical systems included in the imaging unit 10 are arranged at positions where the focal lengths are equal (this value is assumed to be f) and the distance between the optical axes, that is, the base line length is separated by L. The image sensors 13a and 13b are arranged at positions away from the lenses 12a and 12b by a focal length f. The lenses 12a and 12b are usually configured by combining a plurality of lenses, but FIG. 1 shows a case where a single lens is used in order to simplify the description.

  The image analysis unit 20 includes a three-dimensional reconstruction unit 21 that generates three-dimensional information by reconstructing three dimensions based on an image captured by the imaging unit 10, and the three-dimensional information generated by the three-dimensional reconstruction unit 21. A closed region setting unit 22 that sets a plurality of closed regions whose union is the entire region of the image, and an image processing unit 23 that performs predetermined image processing for each closed region set by the closed region setting unit 22. In addition, the image analysis unit 20 calculates various parameters necessary for performing various processes to be described later (calibration function) and, when generating three-dimensional information, correction processing (rectification) as necessary. A function to perform processing.

  The control unit 30 is realized by a CPU (Central Processing Unit) or the like having a calculation and control function, and reads out an image processing program stored and stored in a storage unit 50 to be described later, whereby the imaging unit 10 and the image analysis unit 20. Control various operation processes.

  The output unit 40 displays and outputs images, character information, and the like, and is realized by a liquid crystal display, a plasma display, an organic EL (Electroluminescence) display, or the like. Further, as the output unit 40, a speaker that outputs audio information to the outside can be further provided.

  The storage unit 50 includes a ROM (Read Only Memory) in which a program for starting a predetermined OS (Operation System), an image processing program, and the like are stored in advance, and a RAM (Random Access) in which calculation parameters and data for each process are stored. Memory). The storage unit 50 includes image data 51 captured by the imaging unit 10, closed region setting information 52 necessary for setting a closed region in an image by the closed region setting unit 22, and processing contents 53 of image processing performed for each closed region. , And templates 54 representing patterns of various objects (vehicles, persons, road surfaces, white lines, signs, etc.) used for object recognition in the image in units of pixel points.

  Note that the above-described image processing program is a hard disk, flexible disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD ± R, DVD ± RW, DVD-RAM, MO disk, PC card, xD. It is also possible to record on a computer-readable recording medium such as a picture card or smart media and distribute it widely.

  The movement status detection unit 60 is realized by various types of sensors, and detects the current status regarding the movement of the vehicle itself and the current status of the surroundings where the vehicle is moving. Among these, the current status related to the movement of the vehicle itself includes the vehicle position / speed / acceleration, the vibration status of the vehicle body, the steering angle of the steering shaft of the vehicle, the pitch and roll of the suspension provided in the vehicle, etc. It is. Also, what is to be detected as the current situation around the vehicle is the road surface condition, the size of various objects in the vicinity, the distance to the object, the presence / absence of raindrops, brightness, and the like. The movement state detection unit 60 detects the movement state of the vehicle by using at least one of these pieces of information.

  The image processing method executed by the image processing apparatus 1 having the above configuration will be described in detail with reference to the flowchart shown in FIG. First, the imaging unit 10 performs an imaging process for generating an image by imaging a predetermined field of view (step S1). In the right camera 11 a and the left camera 11 b of the imaging unit 10, light is collected from a region included in a predetermined viewing angle using the lenses 12 a and 12 b under the control of the control unit 30.

FIG. 3 is an explanatory diagram conceptually showing an imaging process by a compound-eye stereo camera. In the figure, the optical axis z _b of the optical axis z _a and the left camera 11b of the right camera 11a indicates a parallel case. In this case, the point corresponding to the point A _b in the left image area I _b corresponding to the left camera specific coordinate system (left camera coordinate system) is the right image corresponding to the right camera specific coordinate system (right camera coordinate system). It exists on the straight line α _E (epipolar line) in the region I _a . Note that FIG. 3 illustrates the case where the corresponding point in the right camera 11a is searched with the left camera 11b as a reference, but the right camera 11a can be used as a reference in contrast to this.

  The lights collected by the lenses 12a and 12b are imaged on the surfaces of the image sensors 13a and 13b, respectively, and converted into electrical signals (analog signals). Analog signals output from the image sensors 13a and 13b are converted into digital signals by the A / D converters 14a and 14b, respectively, and the converted digital signals are temporarily stored in the frame memories 15a and 15b. The digital signals temporarily stored in the frame memories 15a and 15b are sent to the image analysis unit 20 after a predetermined time has elapsed.

In the image analysis unit 20 that has received the digital signal, the three-dimensional reconstruction unit 21 generates three-dimensional information (step S3). In the three-dimensional information generation processing here, first, calculation of coordinate values in the left and right camera coordinate systems and association between coordinate values (corresponding point search) are performed. When three-dimensional reconstruction is performed by this corresponding point search, it is desirable that pixel points located on an arbitrary straight line that passes through the reference image be located on the same straight line in the other image ( Epipolar restraint conditions). However, this epipolar constraint condition is not always satisfied. For example, in the case shown in FIG. 4, the point in the right image region I _a corresponding to the point A _b in the left image region I _b as a reference is a straight line α _A. while present in the above, the point in the right image area I _a corresponding to a point B _b in the left image area I _b existing on the straight line alpha _B.

Thus, when the epipolar constraint condition is not satisfied, the search range cannot be narrowed down, and the amount of calculation when searching for corresponding points becomes enormous. In such a case, the image analysis unit 20 performs processing (rectification processing) that normalizes the left and right camera coordinate systems in advance and converts them into a state that satisfies the epipolar constraint condition. FIG. 5 shows the correspondence between the left and right image areas after the rectification process. As shown in FIG. 5, if the epipolar constraint condition is satisfied, the search range can be narrowed down to the epipolar line α _E , so that the calculation amount required for the corresponding point search can be reduced.

Here, an example of the corresponding point search method will be described. First, in the reference left image region I _b , a local region is provided in the vicinity of the pixel point of interest, and a region similar to this local region is provided on the corresponding epipolar line α _E in the right image region I _a . Then, while scanning the local region of the right image region I _{a on} the epipolar line α _E , the local region having the highest similarity with the local region of the left image region I _b is searched. As a result of this search, the center point of the local area having the highest similarity is set as the corresponding point of the pixel point of the left image area I _b .

  As similarities used in the search for corresponding points, the sum of absolute values of differences between pixel points in the local region (SAD: Sum of Absolute Difference) and the square sum of differences between pixel points in the local region (SSD). : Sum of Squared Difference) or Normalized Cross Correlation (NCC) between pixel points in the local region can be applied. Among these, when SAD or SSD is applied, the point having the minimum value is the point with the highest similarity, while when NCC is applied, the point with the maximum value is the point with the highest similarity. And

  After step S3, the three-dimensional reconstruction unit 21 calculates the distance R from the forefront of the moving object to the imaged point using the coordinate values (x, y, z) calculated in step S3. May be. Here, the position of the forefront of the vehicle in the camera coordinate system needs to be measured in advance. When the distance R is calculated, a distance image in which the distance R is superimposed on the image obtained by the imaging unit 10 can be generated. FIG. 6 is a diagram illustrating a display output example of the distance image in the output unit 40. The distance image 301 shown in the figure represents the distance from the imaging unit 10 depending on the degree of shading, and is displayed darker as the distance increases. In addition, all the coordinate values (x, y, z) and the distance R of the calculated visual field configuration point are stored in the storage unit 50 in association with the corresponding image.

  In parallel with the processing (steps S1 and S3) in the imaging unit 10 and the three-dimensional reconstruction unit 21 described above, the movement state detection unit 60 detects the movement state of the vehicle (step S5). The detection result of the movement state is reflected in the closed region setting process in the closed region setting unit 22. More specifically, the control unit 30 that has received the detection result of the movement status refers to the closed region setting information 52 stored in the storage unit 50, and sends a control signal regarding the setting of the closed region according to the detection result. The image is sent to the closed region setting unit 22 in the image analysis unit 20. The closed area setting information 52 includes information for associating the detection result of the movement state with the closed area setting mode. Here, the closed region setting mode is set in advance in accordance with an image pattern that is expected to be photographed in accordance with the moving state of the vehicle and the processing content to be performed on the image pattern.

  The closed region setting unit 22 that has received the control signal from the control unit 30 sets the closed region for the image captured by the imaging unit 10 (step S7). FIG. 7 is a diagram showing an example of a closed region setting mode in this step. A setting mode 401 shown in the figure is an example of setting a closed region when the moving state detection unit 60 detects that the host vehicle travels straight at a certain speed and the road surface continues to travel straight.

  The specific configuration of the setting mode 401 is that a rectangular closed region d1 is provided at the center, each side forming the boundary of the closed region d1 is provided on one bottom side, and the boundary line (periphery) of the screen is set on the other side. An isosceles trapezoidal closed region d2, d3, d4, and d5, which is the base, is provided around the closed region d1. When the vehicle goes straight, the preceding vehicle is often captured and displayed in the closed area d1. In addition to the signal, the sky, the sun, and the clouds are often projected in the closed region d2 positioned above the closed region d1. In the closed areas d3 and d5, which are located on the side of the closed area d1, road signs installed on the shoulders, side walls, vehicles running on the adjacent lanes in parallel, oncoming vehicles, and the like are often displayed. Furthermore, in the closed region d4 located below the closed region d1, there is a high possibility that the road surface on which the vehicle travels is projected. In this way, there is a high possibility that characteristic objects that reflect the moving situation in which the vehicle goes straight are projected in the five closed areas. Therefore, multifaceted image processing can be realized by detecting these characteristic objects for each closed region.

  An example of setting other closed regions will be described. FIG. 8 is a diagram illustrating an example of setting a closed region in a state where the straight traveling state continues for a while as in FIG. 7 but the vehicle speed is higher than the setting mode 401 illustrated in FIG. It is. In the situation where the setting mode 402 shown in FIG. 8 is selected, there is a high possibility that the inter-vehicle distance between the host vehicle and the preceding vehicle is larger than in the situation where the setting mode 401 shown in FIG. 7 is applied. For this reason, the closed region d <b> 1 located in the center is provided smaller than the setting mode 402.

  FIG. 9 is a diagram illustrating an example of setting the closed region when the movement state detection unit 60 detects a state in which the vehicle is turning to the left by measuring the rotation of the steering angle of the steering shaft, for example. In the setting mode 403 shown in the figure, the vehicle detection closed region d1 is also moved to the left in the horizontal direction compared to the setting mode 401. This is because when traveling on a curve that turns to the left, there is a high possibility that a preceding vehicle traveling in the same traveling lane is displayed on the left side of the screen. The reason why the area of the closed region d1 is substantially the same as the area of the closed region d1 in the setting mode 401 is that the vehicle traveling speed in the setting mode 403 is assumed to be substantially the same as the vehicle traveling speed in the setting mode 401. It is. Further, when the vehicle travels on a curve that turns to the right, the setting mode 401 is reversed to the left and right, and the closed region d1 moves to the right in the horizontal direction as compared with the setting mode 401.

  FIG. 10 is a diagram illustrating an example of setting a closed region when the vehicle travels straight on a slope that protrudes downward in the vertical direction. In the setting mode 404 shown in the figure, since there is a high possibility that the preceding vehicle is detected at the top of the image, the vehicle detection closed region d1 is arranged at the top of the screen compared to the setting mode 401. When the vehicle travels straight on a slope that is convex upward in the vertical direction, the setting mode 404 is turned upside down, that is, the closed region d1 is arranged below the setting mode 401.

  Incidentally, in addition to the closed region setting mode described above, various setting modes can be applied depending on the movement mode of the vehicle on which the image processing apparatus 1 is mounted. For example, the number of closed regions to be divided is not limited to five. In addition, if the entire field of view can be covered by taking the union of all closed regions, adjacent closed regions may have intersections.

  After the closed region is set as described above, the image processing unit 23 reads the processing content 53 from the storage unit 50 under the control of the control unit 30, and sets the processing content 53 in step S7 according to the read processing content 53. Predetermined image processing is performed for each closed region (step S9). FIG. 11 is an explanatory diagram showing the processing contents of the image processing to be performed for each closed region set in step S7. The correspondence table 71 shown in the figure shows the correspondence between the closed area and the processing contents, and shows the specific object detection and the processing to be performed when the object is detected as specific processing contents. ing. Here, when each object is detected, the object pattern stored in advance in the template 54 of the storage unit 50 and the pattern in the image captured by the imaging unit 10 are compared to examine the correlation between them. (Template matching). As a result of the template matching, when a corresponding pattern is detected, processing corresponding to the detected object is performed. Therefore, the image processing unit 23 includes a function of an object detection unit that detects an object for each closed region.

  The processing contents described in the correspondence table 71 in FIG. 11 will be specifically described. First, a preceding vehicle or a person is detected for the closed region d1. Since the pattern here includes the size of the vehicle (or person), even if a small image of a vehicle located far away enters the closed region d1, it is not recognized as an object at that stage. A vehicle is detected only when it matches a template corresponding to the size of the vehicle when approaching a predetermined distance.

  FIG. 12 is a diagram illustrating a display example of the output unit 40 when a preceding vehicle is detected in the closed region d1. In the display image 501 shown in the figure, the closed region set on the image captured by the imaging unit 10 is displayed in an overlapping manner, and a vehicle of a size that can be regarded as dangerous is detected in the closed region d1, so that a lower portion of the image is displayed. The warning message “Please apply the brake” is output. In accordance with the output of the warning message, for example, a warning sound can be generated, or a message similar to the display can be output by voice. When a person (or an obstacle assumed to be present on the road) is detected in the closed region d1, a warning that prompts the user to actuate the brake or to avoid it may be output.

  In the closed region d2, signal detection is performed. A display image 502 shown in FIG. 13 shows a case where a signal is detected in a predetermined closed region d2, a warning window is provided at a position surrounding the signal on the screen, and a warning message “There is a signal” is displayed. Show. It is also possible to detect the signal and the color of the signal. For example, when the signal is red, it is possible to output a message for prompting the brake preparation. By the way, in the case shown in FIG. 11, in the closed region d2, detection of sky color, luminance, cloud amount, and the like is simultaneously performed. As a result of this detection, for example, when a large number of clouds are detected in the closed region d2, it is determined that the traveling direction gradually becomes dark, and a message for prompting lighting of the vehicle's light is displayed. As the determination of the situation of the sky, a message prompting the operation of the wiper by detecting raindrops may be displayed and output. Note that the movement status detection unit 60 may detect the color of the sky, the brightness, the amount of clouds, and the like. In this case, the movement status detection unit 60 plays a part of the function of the object detection means.

  In the closed regions d3 and d5, recognition of the type of road sign provided on the road shoulder and the detection of the side wall are performed. In the display image 503 shown in FIG. 14, the sign detected in the closed region d3 is indicated by an arrow, and the contents of the sign (such as “it is an overtaking prohibited section”) are displayed at the bottom of the image. In addition, when it is difficult to identify the contents of the sign, for example, only a display indicating that there is a sign may be performed. When a side wall is detected, a warning message or a warning sound is emitted when the distance from the side wall is closer than a certain value. By the way, since there is a high possibility that the vehicles traveling in the adjacent traveling lanes are displayed in the closed areas d3 and d5, these vehicles may be detected.

  In the closed region d4, by detecting the road surface and the white line on the road surface, the vehicle's course condition (for example, whether the road surface is bent or whether the road surface is frozen) and the traveling lane are known. In addition, it may be possible to recognize and grasp a road segment zone other than the white line (yellow line or the like). FIG. 15 is a diagram illustrating an image processing result when it is recognized that the road surface detected in the closed region d4 draws a curve turning to the left. In the display image 504 shown in the figure, an arrow that turns to the left is displayed in the closed region d4, and a message “It is a left curve” is displayed and output at the bottom of the image. In this case, when recognizing that the host vehicle is deviating from the traveling lane by detecting a white line or the like, a warning to that effect may be output or a warning sound may be generated.

  Up to this point, the case where one image processing is performed in an image has been mainly described, but it is of course possible to perform image processing for each closed region in parallel. In this case, information for each closed area is switched and displayed for a predetermined time, or a closed area that requires urgent display can be displayed with the highest priority. Further, in FIGS. 12 to 15, the boundary line of the closed region is clearly shown in the display image, but the boundary line may not be explicitly displayed.

  According to the embodiment of the present invention described above, the captured image is divided into closed regions based on the movement status of the host vehicle, and predetermined image processing is performed for each of the divided closed regions, thereby capturing images. Various information included in the image to be processed can be processed in a multifaceted and efficient manner.

  In particular, according to this embodiment, the configuration of the closed region can be optimized according to the traveling state of the vehicle, and it is possible to perform optimal image processing in accordance with the current state.

  FIG. 16 is a diagram illustrating a configuration of an imaging unit of an image processing apparatus according to a modification of this embodiment. The imaging unit 110 shown in the figure is obtained by mounting a stereo adapter 116 to a camera 111 that is a monocular imaging device. The configuration of the camera 111 is the same as that of the right camera 11a and the left camera 11b, and includes a lens 112, an image sensor 113, an A / D converter 114, and a frame memory 115. The stereo adapter 116 includes two light receiving mirrors 117 a and 117 b that receive light incident from a subject, and the light 112 reflected by the light receiving mirrors 117 a and 117 b as a lens 112 that is an imaging optical system of the camera 111. And light guide mirrors 118a and 118b.

  Since the imaging unit 110 having such a configuration forms and outputs images from two different viewpoints, the imaging unit 110 has a function equivalent to that of the imaging unit 10 that is a stereo camera. Therefore, according to this modification, various information included in the image to be captured can be processed in a multifaceted manner, as in the above-described one embodiment. Further, according to this modification, it is possible to easily capture a stereo image using a general camera that is not for stereo image capturing.

  The preferred embodiment of the present invention has been described in detail above, but the present invention is not limited only to this embodiment. For example, the image processing apparatus according to the present invention may further include processing change means for changing the processing content for each closed region. By using this processing change means, for example, it is possible to change the processing content of each closed region according to the weather or day / night distinction that can be grasped from the sky detection result.

  Further, by providing an input unit in the image processing apparatus according to the present invention, a configuration in which the user can freely set the processing content and the closed region setting method may be adopted.

  In addition, using a monocular camera as the imaging unit, tertiary such as shape from focus, shape from defocus, shape from motion, shape from shading, etc. The distance to each constituent point of the image can also be calculated by applying the original reconstruction technique.

  As this imaging unit, a multi-lens stereo camera, for example, a three-eye stereo camera or a four-eye stereo camera can be used. Using these three-lens or four-eye stereo cameras, more reliable and stable processing results can be obtained in 3D reconstruction processing etc. (for example, Fumiaki Tomita, “High-function 3D visual system VVV”, (See Journal of Information Processing, “Information Processing”, Vol. 42, No. 4, pp. 370-375 (2001).) In particular, it is known that if a plurality of cameras have a base line length in two directions, three-dimensional reconstruction in a more complicated scene becomes possible. In addition, a multi-baseline stereo camera in which a plurality of cameras are arranged in one baseline length direction can be realized, and more accurate stereo measurement can be performed.

  By the way, it is also possible to use a distance measuring device such as a radar in addition to various cameras as the imaging means. This makes it possible to use radar measurement values that are more accurate than ordinary cameras, and to further improve the resolution of the ranging points.

  In addition to the template matching described above, a commonly used object recognition method such as a region segmentation method based on edge extraction or a statistical pattern recognition method based on cluster analysis can be applied as an image recognition method.

  The image processing apparatus according to the present invention can be mounted on a vehicle other than a four-wheeled vehicle, such as an electric wheelchair. In addition to the vehicle, it can also be mounted on a moving body such as a person or a robot.

  Thus, the present invention can include various embodiments and the like not described herein, and various design changes and the like can be made without departing from the technical matters specified by the claims. It is possible to apply.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the outline | summary of the image processing method which concerns on one embodiment of this invention. It is explanatory drawing which shows notionally the imaging process using a stereo camera. It is explanatory drawing which shows a response | compatibility of the left-right image area | region before a rectification process. It is explanatory drawing which shows a response | compatibility of the left-right image area | region after a rectification process. It is a figure which shows the example of an output of a distance image. It is a figure which shows the example of a setting of a closed area | region when a vehicle drive | works straight ahead. It is a figure which shows the example of a setting of the closed area | region when a vehicle goes straight at high speed. It is a figure which shows the example of a setting of a closed area | region when a vehicle drive | works while curving to the left. It is a figure which shows the example of a setting of a closed area | region when a vehicle travels straightly on a slope. It is explanatory drawing which shows the processing content of the image process performed for every closed area | region. It is a figure which shows the example of a display at the time of performing the image process regarding vehicle detection. It is a figure which shows the example of a display at the time of performing the image process regarding signal detection. It is a figure which shows the example of a display at the time of performing the image process regarding label | marker detection. It is a figure which shows the example of a display at the time of performing the image process regarding road surface detection. It is a figure which shows the structure of the imaging part of the image processing apparatus which concerns on the modification of one embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10,110 Image pick-up part 11a Right camera 11b Left camera 12a, 12b, 112 Lens 13a, 13b, 113 Image pick-up element 14a, 14b, 114 A / D conversion part 15a, 15b, 115 Frame memory 20 Image analysis part 21 3D reconstruction unit 22 Closed region setting unit 23 Image processing unit 30 Control unit 40 Output unit 50 Storage unit 51 Image data 52 Closed region setting information 53 Processing content 54 Template 60 Movement status detection unit 111 Camera 116 Stereo adapter 117a, 117b Light reception Mirror 118a, 118b Light guide mirror 301 Distance image 401, 402, 403, 404 Setting mode 501, 502, 503, 504 Display image d1, d2, d3, d4 Closed region

Claims (9)

An imaging unit mounted on a moving body and imaging a predetermined field of view to generate an image;
Movement status detection means for detecting the movement status of the mobile body;
Closed region setting means for setting a plurality of closed regions with the entire field of view as a union according to the movement status of the moving body detected by the movement status detection unit;
Image processing means for performing predetermined image processing for each closed area set by the closed area setting means;
An image processing apparatus comprising: The movement status of the mobile body is
The image processing apparatus according to claim 1, wherein the image processing apparatus includes at least one of a current situation related to the movement of the moving body and a surrounding current situation where the moving body moves.   The image processing apparatus according to claim 1, further comprising an object detection unit configured to detect a predetermined object for each closed region set by the closed region setting unit.   4. The apparatus according to claim 1, further comprising a process changing unit that changes an image processing method applied to the image processing performed by the image processing unit for each closed region set by the closed region setting unit. The image processing apparatus according to item. The imaging means includes
A pair of imaging optical systems;
A pair of imaging elements for converting an optical signal output from each of the pair of imaging optical systems into an electrical signal;
The image processing apparatus according to claim 1, further comprising: The imaging means includes
An imaging apparatus including an imaging optical system and an imaging element that converts an optical signal output from the imaging optical system into an electrical signal;
A stereo adapter that is provided in front of the imaging optical system, receives light at a plurality of sites, and guides the light received by the plurality of sites to the imaging optical system;
The image processing apparatus according to claim 1, further comprising:   The image processing apparatus according to claim 1, wherein the moving body is a vehicle. An image processing method for performing image processing on an image that is mounted on a moving body and is generated by an imaging unit that images a predetermined field of view,
A movement status detection step for detecting the movement status of the mobile body;
A closed region setting step for setting a plurality of closed regions with the entire field of view as a union according to the moving state of the moving body detected in the moving state detecting step;
An image processing step for performing predetermined image processing for each closed region set in the closed region setting step;
An image processing method comprising: In order to perform image processing on an image that is mounted on a moving body and generated by an imaging means that images a predetermined field of view,
A movement status detection step for detecting the movement status of the mobile body;
A closed region setting step for setting a plurality of closed regions with the entire field of view as a union according to the moving state of the moving body detected in the moving state detecting step;
An image processing step for performing predetermined image processing for each closed region set in the closed region setting step;
The program for image processing characterized by performing these.

Images