JP2006322797A - Apparatus, method, and program for processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To output distance information at high speed by reducing the load required to process distance computations. <P>SOLUTION: This image processing apparatus 1 is provided with: an imaging part 10 mounted to a mobile body for creating a group of image signals corresponding to a prescribed imaging field of view; a detection part 60 for acquiring movement information of the mobile body; a window changeover part 31 for performing the processing of both the selection of a window matched with the movement information acquired by the detection part 60 and changeover to the selected window; and an operation part 21 for processing a group of image signals corresponding to an image processing range specified by the window changed by the window changeover part 31 and computing the distance to an object located within the imaging field of view. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing device, an image processing method, and an image processing program for performing image processing on an image generated by imaging a predetermined field of view, and more particularly to an image processing device, an image processing method, and an image processing suitable for use in a vehicle. It is about the program.

  2. Description of the Related Art Conventionally, an inter-vehicle distance detection device is known that is mounted on a vehicle such as an automobile and processes an image obtained by imaging a preceding vehicle that travels in front of the host vehicle, and detects a distance from the host vehicle to the preceding vehicle (for example, Patent Document 1). This inter-vehicle distance detection device sets a plurality of distance measurement windows at predetermined positions in the image in order to capture the preceding vehicle on the image, and processes the image within each of the set distance measurement windows to arbitrarily set the distance measurement window. The distance to the target object is calculated, and the imaging position of the preceding vehicle is recognized based on the calculated result and the position information of the distance measurement window.

Japanese Patent No. 2635246 JP-A-8-171151

  By the way, in the conventional inter-vehicle distance detection device described above, in order to capture a preceding vehicle whose imaging position is unknown, the distance measurement window needs to be set so as to cover almost the entire area of the image. As a result, since the distance calculation is performed by processing almost the entire area of the image, there is a problem that the load of the calculation process is large and the processing time is long.

  The present invention has been made in view of the above, and when processing a captured image and calculating a distance to a predetermined object such as a preceding vehicle, the load of calculation processing is reduced, and distance information can be obtained at high speed. Is to provide an image processing apparatus, an image processing method, and an image processing program.

  In order to achieve the above object, an image processing apparatus according to claim 1 of the present invention is mounted on a moving body, has an imaging field of view, and generates an image signal group corresponding to the imaging field of view. A calculation means for processing the image signal group to calculate a distance to an object located in the imaging field of view, and selecting an image processing range corresponding to movement information of the moving body from a plurality of image processing ranges Switching processing means for switching the image processing range in the computing means to the selected image processing range, and the computing means is based on an image signal group corresponding to the image processing range switched by the switching processing means. And calculating the distance to the object.

  An image processing apparatus according to claim 2 of the present invention is the image processing apparatus according to the above invention, further comprising movement information acquisition means for acquiring the movement information and outputting the acquired movement information to the switching processing means. The means selects the image processing range corresponding to the movement information from the plurality of image processing ranges based on the movement information output from the movement information acquisition means.

  In the image processing apparatus according to claim 3 of the present invention, in the above invention, the movement information includes a moving speed of the moving body, a moving acceleration of the moving body, a moving direction of the moving body, It is a moving direction and / or position information of the moving body.

  The image processing apparatus according to claim 4 of the present invention is the image processing apparatus according to the above invention, wherein the imaging means is configured to transmit the first image signal group captured through the first optical path and the second optical path. Generating the imaged second image signal group, and the computing means detects an image signal that matches an arbitrary image signal of the first image signal group from the second image signal group, The distance to the object is calculated based on the amount of movement from the arbitrary image signal in the detected image signal. Note that the movement amount described here indicates a parallax amount generally referred to.

  In the image processing apparatus according to claim 5 of the present invention, in the above invention, the imaging unit converts a pair of imaging optical systems and an optical signal output from each of the pair of imaging optical systems into an electrical signal. And a pair of imaging elements.

  In the image processing apparatus according to claim 6 of the present invention, in the above invention, the imaging unit includes a pair of light guiding optical systems and imaging regions for the light guiding optical systems. And an imaging device that converts the guided optical signal into an electrical signal in each imaging region.

  According to a seventh aspect of the present invention, in the above invention, the moving body is a vehicle.

  An image processing method according to an eighth aspect of the present invention is an image processing method for processing an image signal group corresponding to an imaging field generated by an imaging unit mounted on a moving body and having a predetermined imaging field. A switching process step of selecting an image processing range corresponding to movement information of the moving body from a plurality of image processing ranges, and switching the image processing range for the image signal group to the selected image processing range; A calculation step of calculating a distance to an object located in the imaging field based on the image signal group corresponding to the image processing range switched in the switching processing step.

  An image processing program according to claim 9 of the present invention is an image processing program for processing an image signal group corresponding to an imaging field generated by an imaging unit mounted on a moving body and having a predetermined imaging field. A switching processing procedure for selecting an image processing range corresponding to movement information of the moving body from a plurality of image processing ranges, and switching the image processing range for the image signal group to the selected image processing range; Calculating means for calculating a distance to an object located in the imaging field of view based on an image signal group corresponding to the image processing range switched in the switching processing procedure.

  According to the image processing device, the image processing method, and the image processing program of the present invention, when processing a captured image and calculating a distance to a predetermined object such as a preceding vehicle, the load of the calculation process is reduced, and high speed Can output distance information.

  Exemplary embodiments of an image processing apparatus, an image processing method, and an image processing program according to the present invention will be described below in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention. An image processing apparatus 1 shown in FIG. 1 has a predetermined imaging field of view, images an image corresponding to the imaging field of view and generates an image signal group, and analyzes the image signal group generated by the imaging unit 10. An image processing unit 20 to be processed, a control unit 30 that controls the overall processing and operation of the image processing apparatus 1, an output unit 40 that outputs various types of information including distance information, and a memory that stores various types of information including distance information. Unit 50 and a detection unit 60 that detects movement information of the host vehicle that is a moving body such as a four-wheeled vehicle on which the image processing apparatus 1 is mounted. The imaging unit 10, the image processing unit 20, the output unit 40, the storage unit 50, and the detection unit 60 are electrically connected to the control unit 30. Note that this form of connection includes not only wired but also wireless.

  The imaging unit 10 is a stereo camera and includes a right camera 11a and a left camera 11b arranged side by side. The right camera 11a includes a lens 12a, an image sensor 13a, an analog / digital (A / D) converter 14a, and a frame memory 15a. The lens 12a collects light from an arbitrary object located within a predetermined imaging field on the imaging device 13a. The image pickup device 13a is an image pickup device such as a CCD or a CMOS, detects light from an object condensed by the lens 12a as an optical signal, converts it into an electrical signal that is an analog signal, and outputs it. The A / D converter 14a converts the analog signal output from the image sensor 13a into a digital signal and outputs the digital signal. The frame memory 15a stores the digital signal output from the A / D converter 14a, and outputs a digital signal group corresponding to one captured image as image information that is an image signal group corresponding to the imaging field. On the other hand, the left camera 11b has the same configuration as the right camera 11a, and includes a lens 12b, an image sensor 13b, an A / D conversion unit 14b, and a frame memory 15b. Each component of the left camera 11b has the same function as each corresponding component of the right camera 11a.

  The lenses 12a and 12b as a pair of imaging optical systems provided in the imaging unit 10 are positioned with a parallel optical axis and separated by a distance L. The image sensors 13a and 13b are located at a distance f on the optical axis from the lenses 12a and 12b, respectively. The right camera 11a and the left camera 11b capture the same object from different positions through different optical paths. The lenses 12a and 12b are usually configured by combining a plurality of lenses, and are in a state where lens aberrations such as distortion are corrected satisfactorily.

The image processing unit 20 includes a calculation unit 21 that processes the image signal group acquired from the imaging unit 10 and calculates the distance to the captured object. The computing unit 21 detects a right image signal that matches an arbitrary left image signal in the left image signal group output from the left camera 11b from the right image signal group output from the right camera 11a, and detects the detected right image signal. Based on the amount of movement of the right image signal, which is the distance from the corresponding left image signal, the distance to the object located in the imaging field of view is calculated. In other words, the calculation unit 21 superimposes the right image signal group generated by the right camera 11a and the left image signal group generated by the left camera 11b on the basis of the position of the optical axis of each imaging optical system. Detects any left image signal in the image signal group and the right image signal in the right image signal group that most closely matches this, and the distance on the image sensor from the corresponding left image signal to the right image signal For example, the distance R from the imaging unit 10 to the vehicle C in FIG. 1 is calculated using the following equation (1) based on the principle of triangulation. The movement amount I may be obtained based on the number of pixels of the image sensor and the pixel pitch. For simplicity, the parallel stereo is explained here, but the optical axes intersect with an angle, the focal lengths are different, the positional relationship between the image sensor and the lens is calibrated, and so on. It is also possible to realize parallel stereo by arithmetic processing.
R = f · L / I (1)

  The calculation unit 21 calculates a distance to an object corresponding to an arbitrary image signal within the image processing range. The image processing unit 20 creates distance information by associating the distance to the object calculated by the calculation unit 21 with the position of the object in the image, and outputs the distance information to the control unit 30. Note that the object that is the target of the distance calculation here is not limited to a tangible object, but is an arbitrary subject that has been imaged, and includes backgrounds such as road surfaces and the sky.

  The control unit 30 includes a CPU that executes a processing program stored in the storage unit 50, and controls various processes and operations performed by the imaging unit 10, the image processing unit 20, the output unit 40, the storage unit 50, and the detection unit 60. In particular, the control unit 30 according to the first embodiment includes a window switching unit 31. The window switching unit 31 acquires movement information from the detection unit 60, selects a window that matches the movement information from among a plurality of window information stored in the window information 51, and selects the window set by the calculation unit 21 Instruction information for switching to the window is output to the image processing unit 20 together with the window information. The window information is information regarding the size and shape of the window.

  The output unit 40 outputs various types of information including distance information. For example, the output unit 40 includes a display device such as a liquid crystal display or an organic EL (Electroluminescence) display, and displays various displayable information such as an image captured by the imaging unit 10 together with distance information. Furthermore, a voice output device such as a speaker may be provided, and various voice information such as distance information and a warning based on the distance information may be output.

  The storage unit 50 includes a ROM in which various types of information such as a program for starting a predetermined OS and an image processing program are stored in advance, and a RAM in which various processing parameters for each process and various types of information input to and output from each component are stored. Is provided. In addition, the storage unit 50 stores window information 51 storing a plurality of window information selected by the window switching unit 31, detection information 52 detected by the detection unit 60, image information 53 captured by the imaging unit 10, and calculation by the calculation unit 21. Then, the created distance information 54 is stored.

  The detection unit 60 detects movement information of the host vehicle using various sensors. In particular, the detection unit 60 according to the first embodiment includes a speed sensor 61 that detects the moving speed of the host vehicle. The speed sensor 61 is a speedometer normally provided in an automobile or the like. The detection unit 60 outputs speed information detected by the speed sensor 61 to the control unit 30 as movement information of the host vehicle. The speed sensor 61 may be a sensor that observes a road surface traveling from the host vehicle and calculates a speed, a sensor that detects acceleration and calculates a speed, and the like.

  Next, processing executed by the image processing apparatus 1 will be described with reference to the flowchart of FIG. FIG. 2 is a flowchart illustrating a processing procedure until the image processing apparatus 1 outputs distance information corresponding to a captured image.

  As shown in FIG. 2, the speed sensor 61 detects the moving speed of the host vehicle, performs speed detection processing that outputs the detected speed to the control unit 30 as detection information (step S102), and the window switching unit 31 A window that matches the detection information acquired from the detection unit 60 is selected from the window information 51, and a window switching process for instructing the image processing unit 20 to switch to the selected window is performed (step S104). In step S104, the window switching unit 31 outputs instruction information for switching to the selected window to the image processing unit 20 together with the window information. Next, the imaging unit 10 captures a predetermined field of view, performs an imaging process of outputting the generated image signal group to the image processing unit 20 as image information (step S106), and the calculation unit 21 starts from the window switching unit 31. The distance to the object is calculated based on the image signal group corresponding to the instructed window, distance information is created by associating the calculated distance with the position of the object on the image, and the distance is output to the control unit 30 Arithmetic processing is performed (step S108). The control unit 30 outputs the distance information and predetermined processing information based on the distance information to the output unit 40 (step S110), and ends a series of processing. The control unit 30 stores detection information 52, image information 53, and distance information 54, which are information generated in each processing step, in the storage unit 50 as needed.

  The above-described series of processing is continuously repeated unless, for example, a predetermined processing end or interruption instruction is given from the passenger of the host vehicle. In addition, although the series of processes described above has been described as being executed sequentially, it is preferable to speed up the processing cycle by executing processes independent of the processing mechanism in parallel. Further, the subsequent speed state may be predicted based on the time-series speed information stored in the detection information 52, and the speed detection process may be skipped as appropriate to speed up the processing cycle. Note that the imaging process in step S106 may be performed immediately before step S102 or step S104.

  Next, the window switching process in step S104 shown in FIG. 2 will be described. FIG. 3 is a flowchart showing the processing procedure of the window switching process. As shown in FIG. 3, the window switching unit 31 acquires detection information from the detection unit 60 (step S122), determines whether the speed of the host vehicle is low or high (step S124), and is low. If it is determined (step S124: low speed), a low speed window is selected from the window information 51 (step S126), and instruction information for switching to the selected window is output to the image processing unit 20 together with the window information (step S128). The process returns to S104. On the other hand, when it is determined that the speed is high (step S124: high speed), the window switching unit 31 selects a high speed window from the window information 51 (step S130), and the instruction information for switching to the selected window is displayed together with the window information. The data is output to the processing unit 20 (step S132), and the process returns to step S104.

  In the flowchart shown in FIG. 3, two types of windows, the low speed window and the high speed window, are selected based on the speed of the host vehicle, but more speed states are determined and various sizes are selected. May be selectable, or windows of different sizes may be selected for virtually any speed.

  Next, an example of a window selected by the window switching unit 31 will be described. FIG. 4 is a diagram illustrating an example of a low speed window. The low-speed window 51a illustrated in FIG. 4 corresponds to substantially the entire imaging range 17 corresponding to the imaging field of the imaging unit 10. The low-speed window 15a captures an object located within a range of a short distance or a long distance from the own vehicle within the imaging field of view on the image. On the other hand, FIG. 5 is a diagram illustrating an example of a high-speed window. The high-speed window 51b shown in FIG. 5 is smaller than the low-speed window 51a, and corresponds to a partial area in the center of the imaging range 17. This high-speed window 51b can selectively capture on the image an object located at a long distance from the host vehicle within the imaging field of view, and as a result, it is possible to extract distant image information that is important during high-speed traveling. it can.

  FIG. 6 is a diagram in which the low-speed window 51a is superimposed on the image 18a. As shown in FIG. 6, the low speed window 51 a captures the vehicle C <b> 1 traveling near the host vehicle together with the background, and sets it as a target for distance calculation. On the other hand, FIG. 7 is a diagram in which the high-speed window 51b is superimposed on the image 18b. As shown in FIG. 7, the high-speed window 51 b captures the vehicle C <b> 2 traveling at a position far from the own vehicle together with the background, and sets it as a distance calculation target. The images 18a and 18b are images captured by either the right camera 11a or the left camera 11b of the imaging unit 10.

  Next, an example of distance information calculated and created by the calculation unit 21 will be described. FIG. 8 is a diagram showing distance information 54a created by the calculation unit 21 based on the image 18a and the low speed window 51a shown in FIG. In the distance information 54a, the calculation result 21a represents the result of the distance calculation in the low speed window 51a. As shown in FIG. 8, the calculation unit 21 divides the inside of the low-speed window 51a into predetermined matrix-like small areas, and calculates the average distance to the corresponding object for each of the divided small areas, as shown in FIG. The result is expressed numerically. In the calculation result 21a, the result included in the area Esa represents the distance corresponding to the sky that is the background of the image 18a, the result included in the area Ec1 represents the distance corresponding to the vehicle C1, and the other results are on the road surface. It represents the corresponding distance. On the other hand, FIG. 9 is a diagram showing distance information 54b created by the calculation unit 21 based on the image 18b and the high-speed window 51b shown in FIG. In the distance information 54b, the calculation result 21b represents the result of the distance calculation in the high speed window 51b. Similar to the calculation result 21a, the calculation result 21b represents the average distance to the object calculated for each small area by a numerical value. In the calculation result 21b, the result included in the area Esb corresponds to the sky, the result included in the area Ec2 corresponds to the vehicle C2, and the other results correspond to the road surface. The numerical values indicated by the calculation results 21a and 21b are numerical values representing distances in a predetermined unit system, for example, numerical values expressed in meters.

  The image processing apparatus 1 according to the first embodiment described above selects an image processing range based on the moving speed of the host vehicle, and performs a distance calculation based on an image signal group corresponding to the selected image processing range. Therefore, compared to a conventional image processing apparatus that performs distance calculation on all image signals in the image signal group, it is possible to reduce the load of distance calculation processing and to shorten the time required for distance calculation. As a result, the image processing apparatus 1 can shorten the processing time required to output the distance information after acquiring the image, and can output the distance information at high speed.

(Embodiment 2)
Next, a second embodiment of the present invention will be described. In the first embodiment described above, the window switching unit 31 selects a window based on the speed detected by the speed sensor 61. However, in the second embodiment, the moving direction of the host vehicle is detected. , I am trying to select a window.

  FIG. 10 is a block diagram showing the configuration of the image processing apparatus according to the second embodiment of the present invention. An image processing apparatus 201 illustrated in FIG. 10 includes a detection unit 260 instead of the detection unit 60 of the image processing apparatus 1 according to the first embodiment described above, and the detection unit 260 includes a steering angle sensor 261. The image processing apparatus 201 includes a control unit 230 instead of the control unit 30, and the control unit 230 includes a window switching unit 231. Further, the image processing apparatus 201 includes a storage unit 250 instead of the storage unit 50. The storage unit 250 detects window information 251 storing a plurality of window information selected by the window switching unit 231, and the detection unit 260 detects the window information. The detection information 252 is stored, and the image information 53 and the distance information 54 are stored in the same manner as the storage unit 50. Other configurations are the same as those of the first embodiment, and the same reference numerals are given to the same components.

  The steering angle sensor 261 is a sensor that detects a steering angle that is a left and right rotation angle of the front wheel as a movement direction. For example, the steering angle sensor 261 is a sensor that detects a steering angle based on a rotation angle and a rotation direction of a handle. The detecting unit 260 outputs the steering angle detected by the steering angle sensor 261 to the control unit 230 as movement information of the host vehicle. The steering angle sensor 261 may be a sensor that detects a moving direction using a gyroscope, a sensor that detects a moving direction based on a blinking state of a direction indicator, or the like.

  Next, processing executed by the image processing apparatus 201 will be described. FIG. 11 is a flowchart illustrating a processing procedure until the image processing apparatus 201 outputs distance information corresponding to a captured image.

  As shown in FIG. 11, the steering angle sensor 261 detects the steering angle of the host vehicle, performs a steering angle detection process that outputs the detected steering angle as detection information to the control unit 230 (step S202), and a window switching unit. 231 selects a window suitable for the detection information acquired from the detection unit 260 from the window information 251, and performs window switching processing for instructing the image processing unit 20 to switch to the selected window (step S204). Here, the window switching unit 231 outputs instruction information for switching to the selected window to the image processing unit 20 together with the window information. Thereafter, as shown in FIG. 11, the image processing apparatus 201 performs the imaging process (step S206), the distance calculation process (step S208), and the distance information output (step S210), respectively, as shown in step S106 and step S108 shown in FIG. And it performs similarly to step S110. However, the image processing apparatus 201 is different in that the window used in the distance calculation process in step S208 is a window corresponding to the steering angle of the host vehicle and the steering angle detected by the steering angle sensor 261 is stored as the detection information 252. This process is different from the process of the image processing apparatus 1.

  Note that, similarly to the image processing apparatus 1, the image processing apparatus 201 preferably executes processes with independent processing mechanisms in parallel to increase the processing cycle speed. Further, the state of the subsequent steering angle may be predicted based on the time-series steering angle information stored in the detection information 252 and the steering angle detection process may be appropriately skipped to speed up the processing cycle. Note that the imaging process in step S206 may be performed immediately before step S202 or step S204.

  Next, the window switching process in step S204 shown in FIG. 11 will be described. FIG. 12 is a flowchart showing the processing procedure of the window switching process. As shown in FIG. 12, the window switching unit 231 acquires detection information from the detection unit 260 (step S222), and determines whether the traveling direction of the host vehicle is the right direction, the left direction, or the straight direction (step). If it is determined that the direction is right (step S224: right), the right window is selected from the window information 251 (step S226), and the instruction information for switching to the selected window is output to the image processing unit 20 together with the window information. (Step S228), the process returns to Step S204. On the other hand, when the window switching unit 231 determines that the direction is the left direction (step S224: left), the window switching unit 231 selects the left window (step S230), and outputs instruction information for switching to the selected window to the image processing unit 20 together with the window information. (Step S232), the process returns to Step S204. On the other hand, when the window switching unit 231 determines that the vehicle is traveling straight (step S224: straight traveling), the window switching unit 231 selects a straight traveling window (step S234), and outputs instruction information for switching to the selected window to the image processing unit 20 together with the window information. (Step S236), the process returns to step S204.

  In the flowchart shown in FIG. 12, three types of windows, the right window, the left window, and the straight traveling window, are selected based on the steering angle of the host vehicle, but more steering angle states are determined. Various windows with different positions and sizes may be selected, and different windows may be selected for virtually any steering angle.

Next, an example of a window selected by the window switching unit 231 will be described. FIG. 13 is a diagram illustrating an example of the right window. The right window 251a shown in FIG.
This corresponds to substantially the entire right half of the imaging range 17. The right window 251a can selectively capture an object located on the right side in the imaging field of view on the image, and as a result, extracts image information on the right front that is important when changing the course in the right direction. be able to. On the other hand, FIG. 14 is a diagram illustrating an example of the left window. A left window 251 b illustrated in FIG. 14 corresponds to substantially the entire left half of the imaging range 17. The left window 251b can selectively capture an object located on the left side in the imaging field of view on the image and extract image information on the left front. On the other hand, FIG. 15 is a diagram illustrating an example of a straight traveling window. A straight traveling window 251 c illustrated in FIG. 15 corresponds to substantially the entire imaging range 17. This straight-advancing window 251c captures on the image objects located in substantially the entire area within the imaging field of view.

  The image processing apparatus 201 according to the second embodiment described above selects an image processing range based on the steering angle of the host vehicle, and performs distance calculation based on the image signal group corresponding to the selected image processing range. Therefore, compared to a conventional image processing apparatus that performs distance calculation on all image signals in the image signal group, it is possible to reduce the load of distance calculation processing and to shorten the time required for distance calculation. As a result, the image processing apparatus 201 can shorten the processing time required to output the distance information after acquiring the image, and can output the distance information at high speed.

(Embodiment 3)
Next, a third embodiment of the present invention will be described. In Embodiments 1 and 2 described above, the window switching unit selects a window based on the speed or steering angle detected by the detection unit. However, in Embodiment 3, the position of the host vehicle is determined. Detection is performed, map information is referred to based on the detected position, and a window is selected based on the referenced map information.

  FIG. 16 is a block diagram showing the configuration of the image processing apparatus according to the third embodiment of the present invention. An image processing apparatus 301 illustrated in FIG. 16 includes an external communication unit 370 instead of the detection unit 60 of the image processing apparatus 1 according to the first embodiment described above. The image processing apparatus 301 includes a control unit 330 instead of the control unit 30, and the control unit 330 includes a window switching unit 331. Furthermore, the image processing apparatus 301 includes a storage unit 350 instead of the storage unit 50. The storage unit 350 receives window information 351 storing a plurality of window information selected by the window switching unit 331, and the external communication unit 370 receives. The position information 352 and the map information 353 referred to by the window switching unit 331 based on the position information 352 are stored, and the image information 53 and the distance information 54 are stored in the same manner as the storage unit 50. Other configurations are the same as those of the first embodiment, and the same reference numerals are given to the same components.

  The external communication unit 370 is a means for detecting the position of the host vehicle, for example, a communication means using GPS (Global Positioning System). The external communication unit 370 outputs the detected position to the control unit 330 as movement information of the host vehicle. The external communication unit 370 may detect the position of the host vehicle and acquire map information around the detected position. Further, the external communication unit 370 may be a sensor that detects a position to which a gyroscope or a speedometer is applied instead of the communication unit.

  Next, processing executed by the image processing apparatus 301 will be described. FIG. 17 is a flowchart illustrating a processing procedure until the image processing apparatus 301 outputs distance information corresponding to a captured image.

  As illustrated in FIG. 17, the external communication unit 370 receives position information of the host vehicle, receives position information that is output to the control unit 330 as the received position information as movement information (step S <b> 302), and the window switching unit 331. Refers to the map information 353 based on the position information acquired from the external communication unit 370, selects a window that matches the referenced map information from the window information 351, and switches the selected window to the image processing unit. The window switching process instructed to 20 is performed (step S304). Here, the window switching unit 331 outputs instruction information for switching to the selected window to the image processing unit 20 together with the window information. Thereafter, as shown in FIG. 17, the image processing apparatus 301 performs the imaging process (step S306), the distance calculation process (step S308), and the distance information output (step S310), respectively, in steps S106 and S108 shown in FIG. And it performs similarly to step S110. However, in that the window used in the distance calculation process in step S308 is a window corresponding to the position information and the map information, and the position detected by the external communication unit 370 is stored as the position information 352. The processing is different from the processing of the image processing apparatus 1.

  Note that, similarly to the image processing apparatus 1, the image processing apparatus 301 preferably executes processes with independent processing mechanisms in parallel to increase the processing cycle. Further, the subsequent position may be predicted based on time-series position information stored in the position information 352, and the position information reception in step S302 may be appropriately skipped to speed up the processing cycle. Note that the imaging process in step S306 may be performed immediately before step S302 or step S304.

  Next, an example of the window switching process in step S304 shown in FIG. 17 will be described. FIG. 18 is a flowchart illustrating an example of the processing procedure of the window switching process. As shown in FIG. 18, the window switching unit 331 acquires position information from the external communication unit 370 (step S322), refers to the map information 353 based on the acquired position information (step S324), and the host vehicle It is determined whether or not the vehicle is traveling on an automobile road (step S326). If the window switching unit 331 determines that the host vehicle is traveling on an automobile-only road (step S326: Yes), the window switching unit 331 selects a dedicated road window from the window information 351 (step S328), and switches to the selected window. The instruction information is output to the image processing unit 20 together with the window information (step S330), and the process returns to step S304. On the other hand, when the window switching unit 331 determines that the host vehicle is not traveling on the automobile road (step S326: No), the window switching unit 331 selects a standard window (step S332), and displays instruction information for switching to the selected window. The information is output to the image processing unit 20 together with the information (step S334), and the process returns to step S304.

  Here, an example of a window selected by the window switching unit 331 will be described. FIG. 19 is a diagram illustrating an example of a dedicated road window. A dedicated road window 351 a shown in FIG. 19 is a window composed of a plurality of small windows provided at the center of the imaging range 17. The dedicated road window 351a can selectively capture on the image an object located in the center of the imaging field of view, and further extracts image information at a predetermined ratio from the captured image information. be able to. As a result, the dedicated road window 351a can extract image information on the road that is important when traveling on an automobile-only road with an appropriate image resolution. On the other hand, FIG. 20 is a diagram illustrating an example of a standard window. A standard window 351 b shown in FIG. 20 corresponds to substantially the entire imaging range 17. The standard window 351b captures on the image an object that is located not only on the road but in a substantially entire area within the imaging field of view.

  Various patterns can be set for the dedicated road window. For example, the slit pattern of the dedicated road window 351a shown in FIG. 19 may be a narrower slit pattern including a large number of slits, or may be a dot pattern in which a plurality of rectangular small windows are randomly arranged. Various patterns are applicable. The dot shape of the dot pattern is not limited to a quadrangle, and may be an arbitrary polygon, a circle, or an ellipse.

  Next, another example of the window switching process in step S304 shown in FIG. 17 will be described. FIG. 21 is a flowchart showing a processing procedure of window switching processing including processing for selecting a window based on the tendency of the road surface gradient of the traveling road. As shown in FIG. 21, the window switching unit 331 acquires position information from the external communication unit 370 (step S342), refers to the map information 353 based on the acquired position information (step S344), and the host vehicle It is determined whether the road surface on which the vehicle travels is a concave curved surface, a convex curved surface, or a flat surface (step S346). When the window switching unit 331 determines that the road surface on which the host vehicle is traveling is a concave curved surface (step S346: concave curved surface), the window switching unit 331 selects a concave curved surface window from the window information 351 (step S348), and instructs to switch to the selected window. The information is output to the image processing unit 20 together with the window information (step S350), and the process returns to step S304. On the other hand, when the window switching unit 331 determines that the road surface is a convex curved surface (step S346: convex curved surface), the window switching unit 331 selects a convex curved surface window (step S352), and performs image processing with the window information and instruction information for switching to the selected window. The data is output to the unit 20 (step S354), and the process returns to step S304. On the other hand, if the window switching unit 331 determines that the road surface is a flat surface (step S346: flat surface), the window switching unit 331 selects a standard window (step S356), and the image processing unit 20 includes instruction information for switching to the selected window together with the window information. (Step S358), and the process returns to step S304.

  Note that the road surface being a concave curved surface means that the road surface gradient tends to increase in the elevation angle direction. For example, a road surface that changes from a flat road surface to an uphill road and a road surface that changes from a downhill road to a flat road surface are suitable. On the other hand, a road surface having a convex curved surface means that the road surface gradient tends to increase in the depression direction. For example, a road surface that changes from a flat road surface to a downward slope and a road surface that changes from an uphill road to a flat road surface are suitable.

  Here, an example of the window selected by the window switching unit 331 based on the tendency of the road surface gradient will be described. FIG. 22 is a diagram illustrating an example of a concave curved window. A concave curved surface window 351 c shown in FIG. 22 corresponds to substantially the entire upper half of the imaging range 17. The concave curved surface window 351c can selectively capture an object located on the upper side in the imaging field of view on the image, and as a result, image information on the upper front side that is important when traveling on a road surface that is a concave curved surface. Can be extracted. On the other hand, FIG. 22 is a diagram illustrating an example of a convex curved window. A convex curved surface window 351 d shown in FIG. 23 corresponds to substantially the entire lower half of the imaging range 17. The convex curved surface window 351d can selectively capture an object located on the lower side in the imaging field of view on the image and extract image information on the lower front side. Note that the standard window selected in the case of a flat surface is the same as the standard window 351b shown in FIG.

  By the way, in the flowcharts shown in FIGS. 18 and 21, the window is selected based on the type of road on which the host vehicle is traveling or the tendency of the road surface gradient as the map information. Various windows may be selected based on various map information, such as whether the current position is in a tunnel, whether the current position is in an urban area, or the like.

  The image processing apparatus 301 according to the third embodiment described above selects an image processing range based on the position information of the own vehicle and the map information referred to based on the position information, and selects the selected image. Since the distance calculation is performed based on the image signal group corresponding to the processing range, the load of the distance calculation process is reduced compared to the conventional image processing apparatus that performs the distance calculation on all the image signals of the image signal group, The time required for the distance calculation can be shortened. As a result, the image processing apparatus 301 can shorten the processing time required to output the distance information after acquiring the image, and can output the distance information at high speed.

(Embodiment 4)
Next, a fourth embodiment of the present invention will be described. The image processing apparatus according to the fourth embodiment includes all of the speed sensor 61, the steering angle sensor 261, and the external communication unit 370 included in the image processing apparatuses 1, 201, and 301 according to the first to third embodiments. The vehicle movement information is detected from various angles, and a window that matches the detected movement information is selected.

  FIG. 24 is a block diagram showing the configuration of the image processing apparatus according to the fourth embodiment of the present invention. An image processing device 401 illustrated in FIG. 24 further includes a detection unit 460 in the image processing device 301 according to the third embodiment described above, and the detection unit 460 includes a speed sensor 61 and a steering angle sensor 261. The image processing apparatus 401 includes a control unit 430 instead of the control unit 330, and the control unit 430 includes a window switching unit 431. Furthermore, the image processing apparatus 401 includes a storage unit 450 instead of the storage unit 350. The storage unit 450 detects window information 451 that stores a plurality of window information selected by the window switching unit 431, and the detection unit 460 detects. The detection information 452 is stored, and the image information 53, the distance information 54, the position information 352, and the map information 353 are stored in the same manner as the storage unit 350. Other configurations are the same as those of the third embodiment, and the same components are denoted by the same reference numerals.

  In the image processing apparatus 401, the window switching unit 431 individually or separately displays the speed detected by the speed sensor 61, the steering angle detected by the steering angle sensor 261, and the map information based on the position information received by the external communication unit 370. A window that matches the determination result is selected from the window information 451, and instruction information for switching to the selected window is output to the image processing unit 20 together with the window information. Thereafter, as in the first to third embodiments, the calculation unit 21 calculates the distance to the object based on the image signal group corresponding to the selected window, and generates and outputs distance information. The window switching unit 431 recognizes, as a mode, one or more possible combinations of various types of movement information acquired from the detection unit 460 or the external communication unit 370, and the state of the host vehicle for the mode specified by the passenger of the host vehicle or the like. May be determined. Further, the window switching unit 431 may predict the subsequent state of the host vehicle based on time-series movement information stored in the detection information 452 and the position information 352.

  The image processing apparatus 401 according to the fourth embodiment described above selects an image processing range based on the result of individually or comprehensively determining various pieces of movement information of the host vehicle, and sets the selected image processing range. Since distance calculation is performed based on the corresponding image signal group, the load of distance calculation processing is reduced compared to the conventional image processing apparatus that performs distance calculation on all image signals of the image signal group. The time required can be shortened. As a result, the image processing apparatus 401 can reduce the processing time required to output the distance information after acquiring the image, and can output the distance information at high speed.

(Embodiment 5)
Next, a fifth embodiment of the present invention will be described. In the first to fourth embodiments described above, the image signal group from the imaging unit 10 is processed to detect the distance to the imaged object, but in the fifth embodiment, the distance is within the imaging field of view. The distance to the object is detected by the radar.

  FIG. 25 is a block diagram showing the configuration of the image processing apparatus according to the fifth embodiment of the present invention. An image processing apparatus 501 shown in FIG. 25 further includes a radar 580 in addition to the image processing apparatus 401 according to the above-described fourth embodiment, and further has a function of controlling the radar 580 in place of the control unit 430. 530. Other configurations are the same as those of the fourth embodiment, and the same components are denoted by the same reference numerals.

  The radar 580 transmits a predetermined transmitted wave, receives a reflected wave reflected by the object surface, and based on the transmission state and the reception state, the distance from the radar 580 to the object that reflects the transmitted wave The direction in which the object is located is detected. The radar 580 is based on the transmission angle of the transmitted wave, the incident angle of the reflected wave, the reception intensity of the reflected wave, the time from when the transmitted wave is transmitted until it is received, and the frequency change in the received wave and the reflected wave. Detect the distance and direction to the object that reflected the transmitted wave. The radar 580 outputs the distance to the object located within the imaging field of the imaging unit 10 to the control unit 530 together with the direction in which the object is located. The radar 580 transmits, for example, laser light, infrared light, millimeter waves, microwaves, and ultrasonic waves as transmission waves.

  In the image processing apparatus 501 according to the fifth embodiment, since the distance is detected by the radar 580 instead of calculating the distance by processing the image information from the imaging unit 10, the distance information can be acquired with higher speed and higher accuracy. .

  In the image processing apparatus 501, the matching between the positional relationship in the image signal group captured by the imaging unit 10 and the positional relationship in the detection range of the radar 580 is obtained in advance as follows, and each process is performed. For example, the image processing apparatus 501 performs an imaging process by the imaging unit 10 and a detection process by the radar 580 on an object having a known shape, and obtains the positions of the known objects processed by the imaging unit 10 and the radar 580, respectively. . Thereafter, the image processing apparatus 501 obtains the positional relationship of the known objects processed by the imaging unit 10 and the radar 580 using the least square method, and the positional relationship in the image signal group captured by the imaging unit 10. And the positional relationship in the detection range of the radar 580 are matched.

  Further, in the image processing apparatus 501, even when the imaging origin of the imaging unit 10 and the detection origin of the radar 580 are misaligned, the imaging point and the distance from the detection point to the image processing apparatus 501 are sufficiently separated. If so, it can be considered that the imaging origin and the detection origin almost overlap. Furthermore, when the positional relationship in the image signal group captured by the imaging unit 10 and the positional relationship in the detection range of the radar 580 are accurately matched, a deviation between the imaging origin and the detection origin is performed by geometric transformation. It is also possible to correct.

  Note that the image processing apparatus 501 may be set so that the radar detection points of the radar 580 are positioned at predetermined intervals in the pixel row where the image signals of the image signal group captured by the imaging unit 10 are positioned. In addition, when each radar detection point is not set in this way, based on a plurality of radar detection points located in the vicinity of each image signal, the radar detection point is placed in the same pixel row as each image signal using a primary interpolation method or the like. The interpolation point may be obtained and the detection process may be performed using this interpolation point.

(Embodiment 6)
Next, a sixth embodiment of the present invention will be described. In Embodiments 1 to 5 described above, a stereo image is picked up by two cameras, the right camera 11a and the left camera 11b. However, in Embodiment 6, a pair of light guide optical systems and light guides are used. A stereo image is picked up by an image pickup device that has an image pickup region corresponding to the optical system and converts an optical signal guided by each light guide optical system into an electric signal in each image pickup region.

  FIG. 26 is a block diagram showing a partial configuration of the image processing apparatus according to the sixth embodiment of the present invention. An imaging unit 110 illustrated in FIG. 26 is an imaging unit included in the image processing device according to the sixth embodiment instead of the imaging unit 10 in the image processing device according to the first to fifth embodiments. The configuration of the image processing apparatus other than that shown in FIG. 26 is the same as that of any of the first to fifth embodiments described above.

  The imaging unit 110 includes a camera 111 as an imaging device having the same configuration and function as the right camera 11a or the left camera 11b of the imaging unit 10. The camera 111 includes a lens 112, an image sensor 113, an A / D conversion unit 114, and a frame memory 115. Furthermore, the imaging unit 110 includes a stereo adapter 119 as a pair of light guide optical systems configured by mirrors 119a to 119d in front of the camera 111. As shown in FIG. 26, the stereo adapter 119 includes mirrors 119a and 119b as a set and faces the reflecting surfaces substantially in parallel, and includes the mirrors 119c and 119d as another set and a reflecting surface that faces substantially parallel. The stereo adapter 119 includes the two pairs of mirror systems adjacently symmetrically with respect to the optical axis of the lens 112.

  The imaging unit 110 receives light from an object located in the imaging field of view with the two left and right mirror systems of the stereo adapter 119, collects it with a lens 112 as an imaging optical system, and captures an image with the imaging element 113. At this time, as shown in FIG. 27, the image sensor 113 has the right image 116a through the right mirror system of the mirrors 119a and 119b and the left image through the left mirror system of the mirrors 119c and 119d. 116b is imaged in an imaging area that is shifted left and right and does not overlap at all. In addition, the technique using such a stereo adapter is disclosed by patent document 2, for example.

  Since the imaging unit 110 according to the sixth embodiment captures a stereo image with one camera provided with a stereo adapter, the imaging unit is simplified and compact compared to the case of capturing a stereo image with two cameras. In addition, the mechanical strength can be increased and the left and right images can be captured in a relatively stable state. Furthermore, since the left and right images are picked up using a common lens and an image pickup device, variations caused by individual differences can be suppressed, and the labor of calibration and the complexity of assembly such as alignment can be reduced.

  As an example of the configuration of the stereo adapter, FIG. 26 shows an example in which plane mirrors are combined so as to face each other in parallel. However, a lens group may be combined, and a convex mirror, concave mirror, or the like has a curvature. The reflecting mirrors may be combined, and the reflecting surface may be configured by a prism instead of the reflecting mirror.

  As shown in FIG. 27, in the sixth embodiment, the left and right images are picked up so as not to overlap at all, but part or all of the left and right images are overlapped, for example, a shutter provided in the light receiving unit. For example, the left and right images may be sequentially switched and imaged, and the left and right images captured with a slight time difference may be subjected to image processing as a stereo image.

  Further, in the sixth embodiment, the left and right images are picked up by shifting the position to the left and right. For example, the stereo adapter is configured by combining the plane mirrors in a substantially orthogonal manner, and the left and right images are shifted in the vertical direction. You may make it image.

  By the way, in the above first to sixth embodiments, the moving speed, moving direction or position information is detected as the moving information of the host vehicle and the window is selected. However, the moving acceleration, moving direction and the like are detected, and the window May be selected. Further, the window may be selected by detecting the face direction and line of sight of the driver of the host vehicle.

  Furthermore, although the configuration of the imaging unit 10 in the first to fifth embodiments or the imaging unit 110 in the sixth embodiment has been described on the assumption that the light receiving units of a pair of cameras or stereo adapters are arranged side by side, It is good also as a structure arrange | positioned side by side, and good also as a structure arrange | positioned along with the diagonal direction.

  Further, a so-called three-eye stereo camera or a four-eye stereo camera may be configured as the stereo camera of the imaging unit. It is known that using a three-lens or four-lens stereo camera provides more reliable and stable processing results in three-dimensional reconstruction processing and the like (Fumiaki Tomita: “Information Processing” published by Information Processing Society of Japan) (Refer to Volume 42, No. 4, “Highly Functional 3D Vision System”). In particular, it is known that if a plurality of cameras are arranged so as to have a baseline length in two directions, three-dimensional reconstruction can be performed in a more complicated scene. If a plurality of cameras are arranged in one base line length direction, a so-called multi-baseline stereo camera can be realized, and more accurate stereo measurement can be performed.

  On the other hand, a monocular camera may be used instead of a stereo camera that is a compound eye as a camera of the imaging unit. In this case, in the arithmetic unit 21, for example, a shape from focus method, a shape from defocus method, a shape from motion method, a shape from shading (shape from shading) By applying a three-dimensional reconstruction technique such as a method, the distance to an object in the imaging field of view can be calculated. Here, the shape from focus method is a method for obtaining a distance from a focus position when the focus is best achieved. The shape from defocus method is a method for obtaining a relative blur amount from a plurality of images having different in-focus distances and obtaining a distance based on a correlation between the blur amount and the distance. The shape from motion method is a method for obtaining a distance to an object on the basis of the movement trajectory of a predetermined feature point in a plurality of temporally continuous images. The shape-from-shading method is a method for obtaining a distance to an object based on shading in an image, reflection characteristics of a target object, and light source information.

  In the first to sixth embodiments described above, the image processing apparatus mounted on a vehicle such as an automobile has been described. However, the image processing apparatus may be mounted and applied to another moving body. Further, the entire image processing apparatus may not be mounted on the moving body. For example, the imaging unit and the output unit are mounted on the moving body, and other components are configured outside the moving body, and wireless communication is performed between them. You may connect with.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. 2 is a flowchart showing a processing procedure until distance information is output in the image processing apparatus shown in FIG. 1. It is a flowchart which shows the process sequence of the window switching process shown in FIG. It is a figure which shows an example of the window for low speed selected by the window switching process of FIG. It is a figure which shows an example of the window for high speed selected by the window switching process of FIG. It is a figure which shows the state which overlap | superposed the window for low speed on the imaged image. It is a figure which shows the state which overlapped the window for high speed on the imaged image. It is a figure which shows an example of the distance information which a calculating part produces. It is a figure which shows an example of the distance information which a calculating part produces. It is a block diagram which shows the structure of the image processing apparatus concerning Embodiment 2 of this invention. It is a flowchart which shows the process sequence until it outputs distance information in the image processing apparatus shown in FIG. It is a flowchart which shows the process sequence of the window switching process shown in FIG. It is a figure which shows an example of the window for rights selected by the window switching process of FIG. It is a figure which shows an example of the window for left selected by the window switching process of FIG. It is a figure which shows an example of the window for straight advance selected by the window switching process of FIG. It is a block diagram which shows the structure of the image processing apparatus concerning Embodiment 3 of this invention. It is a flowchart which shows the process sequence until it outputs distance information in the image processing apparatus shown in FIG. It is a flowchart which shows the process sequence including the process which selects a window based on the road classification which drive | works in the window switching process shown in FIG. It is a figure which shows an example of the window for exclusive roads selected by the window switching process of FIG. It is a figure which shows an example of the standard window selected by the window switching process of FIG. It is a flowchart which shows the process sequence including the process which selects a window based on the tendency of the road gradient which drive | works in the window switching process shown in FIG. It is a figure which shows an example of the window for concave surfaces selected by the window switching process of FIG. It is a figure which shows an example of the window for convex curved surfaces selected by the window switching process of FIG. It is a block diagram which shows the structure of the image processing apparatus concerning Embodiment 4 of this invention. It is a block diagram which shows the structure of the image processing apparatus concerning Embodiment 5 of this invention. It is a block diagram which shows the partial structure of the image processing apparatus concerning Embodiment 6 of this invention. It is a figure which shows an example of the image which the imaging part shown in FIG. 26 images.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Imaging part 11a Right camera 11b Left camera 12a, 12b Lens 13a, 13b Image sensor 14a, 14b A / D conversion part 15a, 15b Frame memory 17 Imaging range 18a, 18b Image 20 Image processing part 21 Calculation part 30 Control unit 31 Window switching unit 40 Output unit 50 Storage unit 51a Low speed window 51b High speed window 54a, 54b Distance information 60 Detection unit 61 Speed sensor 110 Imaging unit 111 Camera 112 Lens 113 Imaging element 114 A / D conversion unit 115 Frame memory 116 Image 119 Stereo adapter 119a, 119b, 119c, 119d Mirror 251a Window for right 251b Window for left 251c Window for going straight 260 Detection unit 261 Steering angle sensor 351a Dedicated road Use windows 351b standard window 351c concave surface for window 351d convex curved for window 370 external communication unit 580 radar C, C1, C2 vehicle

Claims (9)

An imaging means mounted on a moving body, having a predetermined imaging field of view, and generating an image signal group corresponding to the imaging field;
A computing means for processing the image signal group and computing a distance to an object located in the imaging field;
Switching processing means for selecting an image processing range corresponding to movement information of the moving body from among a plurality of image processing ranges in the computing means, and switching the image processing range in the computing means to the selected image processing range;
With
The image processing apparatus, wherein the calculating means calculates a distance to the object based on the image signal group corresponding to the image processing range switched by the switching processing means. The movement information acquisition means for acquiring the movement information and outputting the acquired movement information to the switching processing means,
The switching processing unit selects the image processing range corresponding to the movement information from the plurality of image processing ranges based on the movement information output from the movement information acquisition unit. The image processing apparatus according to claim 1.   The movement information is a moving speed of the moving body, a moving acceleration of the moving body, a moving direction of the moving body, a moving direction of the moving body, and / or position information of the moving body. The image processing apparatus according to 1 or 2. The imaging means generates a first group of image signals captured through a first optical path and a second group of image signals captured through a second optical path;
The computing means detects an image signal that matches an arbitrary image signal of the first image signal group from the second image signal group, and moves the detected image signal from the arbitrary image signal. The image processing apparatus according to claim 1, wherein a distance to the object is calculated based on a quantity. The imaging means includes
A pair of imaging optical systems;
A pair of imaging elements for converting an optical signal output from each of the pair of imaging optical systems into an electrical signal;
The image processing apparatus according to claim 1, further comprising: The imaging means includes
A pair of light guiding optical systems;
An imaging device that has an imaging region for each light guide optical system and converts an optical signal guided by each light guide optical system into an electrical signal in each imaging region;
The image processing apparatus according to claim 1, further comprising:   The image processing apparatus according to claim 1, wherein the moving body is a vehicle. In an image processing method for processing an image captured by an imaging unit mounted on a moving body,
A switching process step of selecting an image processing range corresponding to movement information of the moving body from a plurality of image processing ranges, and switching the image processing range for the processed image to the selected image processing range;
An imaging step for generating an image signal group corresponding to a predetermined imaging field;
A calculation step of calculating a distance to an object located in the imaging field based on the image signal group corresponding to the image processing range switched in the switching processing step;
An image processing method comprising: In an image processing program for processing an image captured by an imaging unit mounted on a moving body,
A switching processing procedure for selecting an image processing range corresponding to movement information of the moving object from a plurality of image processing ranges and switching the image processing range for a processed image to the selected image processing range;
An imaging procedure for generating an image signal group corresponding to a predetermined imaging visual field;
A calculation procedure for calculating a distance to an object located in the imaging field based on the image signal group corresponding to the image processing range switched in the switching processing procedure;
An image processing program comprising:

Images