JP2009141456A - Image processing apparatus, drive support system, and vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce a delay until input images from a plurality of asynchronous cameras are compounded and displayed. <P>SOLUTION: An operation support system includes first through fourth cameras asynchronous to each other, an image processing circuit generating an output image by compounding the input images of the cameras, and a display for displaying an output image. The operation support system further includes a plurality of frame memories for respective cameras. A frame memory storing a newest input image of each camera is specified for each camera, and an output image is generated using an input image in each specified frame memory and displayed. Based on the arrangement position on an output image of an input image from an i-th camera, an output timing (t<SB>A</SB>) of an input image from an i-th camera is set as a select timing (i is 1, 2, 3 or 4). A newest frame memory available at the select timing is specified. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that generates an output image from a plurality of input images. The present invention also relates to a driving support system and a vehicle using the image processing apparatus.

  2. Description of the Related Art In recent years, due to the increasing safety orientation, rear view camera systems that display the rear of a vehicle as an image when driving backward and side camera systems that display the side of a vehicle as an image have been put into practical use and sold as driving assistance systems. Among these types of systems, a system has also been proposed in which images acquired by a plurality of cameras are combined and displayed simultaneously. For example, development of an all-around camera system that synthesizes and displays input images from a plurality of cameras attached to the vehicle body in all directions has become active. As a kind of all-around camera system, an all-around bird's-eye view system that generates and displays an all-around bird's-eye view image by projecting and synthesizing input images from a plurality of cameras on the ground has also been proposed (for example, See Patent Document 2 below).

  FIG. 18 shows an example of the all-around bird's-eye view image. The all-around bird's-eye view image of FIG. 18 is an image of the vehicle viewed from above. An all-around bird's-eye view image is formed by drawing an illustration of the vehicle body at the center of the image and arranging images obtained by projecting input images from the cameras on the ground on the front, rear, left and right sides of the vehicle body. In FIG. 18, ch1, ch2, ch3, and ch4 correspond to images from cameras installed on the front side, right side, rear side, and left side of the vehicle, respectively. In FIG. 18, the arrow shown in the illustration of the vehicle body indicates the traveling direction of the vehicle.

  When combining input images from a plurality of cameras and displaying them simultaneously in one screen, it is necessary to synchronize the plurality of input images. One of the synchronization methods is an external synchronization method. In the external synchronization method, an external synchronization signal common to each camera is generated, and a synchronized video signal is output from a plurality of cameras by giving this external synchronization signal to each camera.

  However, when the external synchronization method is adopted, a synchronization signal generating unit for generating an external synchronization signal is required and wiring for supplying the external synchronization signal to each camera is required, which complicates the system configuration. In addition, the camera module itself is adversely affected. For this reason, it is common to construct a system using an internal synchronization camera that does not require an external synchronization signal.

  However, when a system is constructed using a plurality of internal synchronous cameras, the delay of the display video and the acquisition time shift between the plurality of input images as pointed out in Patent Document 1 become problems.

  Since the driver recognizes the danger and then performs an avoidance action, the video display that is the basis for the danger recognition is required to have real-time characteristics. Therefore, as suggested in Patent Document 1, it is obvious that the shorter the delay until the input image from the camera is displayed in the vehicle system, the better. Considering this, in the method of the cited document 1, input images from a plurality of asynchronous cameras are stored in the input buffer, and the latest stored in the input buffer at the time of generation of the vertical synchronization signal of the output image. An input image is selected, and an output image is generated from the selected input image.

  Consider a case where an all-around bird's-eye view image as shown in FIG. 18 is generated and displayed as an output image using an image processing circuit (not shown) employing the technique of cited document 1. For convenience of explanation, it is assumed that the video signal is a non-interlace video signal, and the video is considered in units of frames. When the interlace method is adopted, the frame may be replaced with a field.

  Reference is made to FIG. 19 focusing on only the image corresponding to ch4. FIG. 19 shows the relationship between the input timing of the ch4 input image and the output timing of the output image. The input images of ch4 in the first, second, third, fourth and fifth frames are sequentially given to an image processing circuit (not shown), and are stored in the input buffer (input of the first frame). (Image not shown). The image processing circuit generates vertical synchronization signals for the output images 201, 202, 203, and 204 at timings t1, t2, t3, and t4.

  In the example shown in FIG. 19, at the timing t1, the entire input image of ch4 of the first frame is held in the input buffer, but the input image of ch4 of the second frame is being written to the input buffer. is there. Therefore, the image processing circuit generates an output image 201 using the ch4 input image of the first frame. Similarly, output images 202, 203, and 204 are generated using the second, third, and fourth frame ch4 input images, respectively. Each generated output image is sequentially displayed. In FIG. 19, the numerical values described in the portion corresponding to ch4 in the output images 201 to 204 represent the frame number of ch4.

  Further, consider a case where a display image as shown in FIG. 20 is displayed using an image processing circuit (not shown) employing the technique of the cited document 1. In this case, the entire display area of the display screen is divided into a left area, an upper right area, and a lower right area. Then, the all-around bird's-eye view image is displayed in the left area, the ch3 input image is enlarged and displayed in the upper right area, and additional information such as the vehicle speed is displayed in the lower right area. In FIG. 20, the arrows shown in the illustration of the vehicle body indicate the traveling direction of the vehicle.

  Reference is made to FIG. 21 focusing on only the image corresponding to ch3. FIG. 21 shows the relationship between the input timing of the ch3 input image and the output timing of the output image. The input images of ch3 in the first, second, third, fourth and fifth frames are sequentially given to an image processing circuit (not shown), and are stored in the input buffer (input of the first frame). (Image not shown). The image processing circuit generates vertical synchronization signals for the output images 211, 212, 213, and 214 at timings t11, t12, t13, and t14.

  In the example shown in FIG. 21, at the timing t11, the entire ch3 input image of the first frame is held in the input buffer, but the ch3 input image of the second frame is being written to the input buffer. is there. Therefore, the image processing circuit generates the output image 211 using the ch3 input image of the first frame. The output image 211 has two areas for displaying an image based on the input image of ch3. The input image of the first frame is applied to both of the two areas. Similarly, output images 212, 213, and 214 are generated using the second, third, and fourth frame ch3 input images, respectively. Each generated output image is sequentially displayed. In FIG. 21, the numerical value described in the portion corresponding to ch3 in the output images 211 to 214 represents the frame number of ch3.

JP 2006-180340 A JP 2006-287992 A

  As described above, the shorter the delay until the input image from the camera is displayed in an application where real-time performance is required, the better. However, the technique of Patent Document 1 provides the optimum delay reduction action for all situations. (It will be clear from the following explanation). A technique with a higher delay reduction effect is desired.

  Therefore, an object of the present invention is to provide an image processing apparatus that contributes to a reduction in delay when generating an output image from a plurality of input images. Another object of the present invention is to provide a driving support system and a vehicle using the same.

  In order to achieve the above object, an image processing apparatus according to the present invention receives input of image data of input images sequentially picked up by each of the first to n-th internal synchronous cameras, and each inputs one image. An input memory unit having a plurality of frame memories for storing image data of each image for each internal synchronous camera (n is an integer of 2 or more), and input of image data from each of the first to nth internal synchronous cameras Selection means for selecting a frame memory used for forming the output image as a designated frame memory for each of the internal synchronous cameras based on a timing relationship between timing and output timing of image data of the output image; and each designated frame memory Output means for generating and outputting image data of the output image by synthesizing the input images stored in the image processing apparatus, The selection means sets the selection timing individually for each internal synchronization camera based on the position on the output image that reflects each input image on the output image, and the designated frame memory for each internal synchronization camera. Is selected at the selection timing.

  By setting the selection timing of the designated frame memory for each internal synchronous camera, it is possible to optimize the delay reduction for each internal synchronous camera.

  Specifically, for example, when i is an integer greater than or equal to 1 and less than or equal to n, the selection unit pays attention to a frame memory in which image data of the latest input image from the i-th internal synchronous camera is to be stored. It is determined whether the image data of the frame memory focused on can be used at the timing when the image data of the input image from the internal synchronous camera should be output as the image data of the output image, and based on the determination result The designated frame memory for the i-th internal synchronous camera is selected.

  For example, the output means stores in each designated frame memory based on predetermined correspondence information that defines a correspondence relationship between each pixel on the input image and each pixel on the output image from each internal synchronization camera. The image data of the output image is generated from the image data of the input image, and the selection unit performs the determination based on information determined from the correspondence information.

  More specifically, for example, the difference in line position between the horizontal line on the input image from the i-th internal synchronization camera and the horizontal line on the output image corresponding to the horizontal line and the maximum amount of the difference are The selection means sets the selection timing in units of horizontal lines and performs the determination based on the maximum amount of difference.

  Alternatively, for example, the difference in pixel position between the pixel on the input image from the i-th internal synchronization camera and the pixel on the output image corresponding to the pixel and the maximum amount of the difference are determined by the correspondence information, and the selection The means sets the selection timing in pixel units and performs the determination based on the maximum amount of the difference.

  Also, for example, the selection means sets a plurality of selection timings for at least one of the first to n-th internal synchronization cameras, and the internal synchronization camera has a plurality of selection timings set. The command frame memory is selected for each of the plurality of selection timings.

  As a result, even when an image based on an input image from one internal synchronous camera is drawn in a plurality of regions in one output image, the delay reduction is optimized for each region.

  In addition, for example, the image processing apparatus further includes an arithmetic processing unit that includes the selection unit and includes a timer that generates an interrupt request, and the arithmetic processing unit is configured to generate the interrupt request at the selection timing. A timer is activated to select the designated frame memory when the interrupt request is generated.

  A driving support system according to the present invention includes first to n-th internal synchronous cameras (n is an integer equal to or greater than 2) attached to a vehicle, and the above-described image processing device. The output image is displayed on the display means by supplying image data of the generated output image to the display means.

  The vehicle according to the present invention is characterized in that the first to nth internal synchronous cameras are attached (n is an integer of 2 or more), and the image processing apparatus is installed.

  The present invention contributes to a reduction in delay when generating an output image from a plurality of input images.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to third embodiments will be described later. First, matters common to each embodiment or items referred to in each embodiment will be described.

  FIG. 1 is a plan view of a vehicle 100 to which the driving support system according to the present embodiment is applied as viewed from above, and shows a camera installation state on the vehicle 100. FIG. 2 is a view of the vehicle 100 as viewed from the left front side. 1 and 2 show a truck as the vehicle 100, the vehicle 100 may be a vehicle other than a truck (such as a normal passenger car). Moreover, the vehicle 100 shall be arrange | positioned on the ground (for example, road surface).

  As shown in FIG. 1, cameras CA1, CA2, CA3, and CA4 are attached to the front, right, rear, and left sides of the vehicle 100, respectively. In the present embodiment, the cameras CA1, CA2, CA3, and CA4 may be simply referred to as cameras or cameras without being distinguished from each other.

  As shown in FIG. 2, the camera CA <b> 1 is installed, for example, on the upper part of the front mirror of the vehicle 100, and the camera CA <b> 4 is installed, for example, on the uppermost part of the left side surface of the vehicle 100. Although not shown in FIG. 2, the camera CA <b> 3 is installed, for example, at the uppermost part of the rear part of the vehicle 100, and the camera CA <b> 2 is installed, for example, at the uppermost part of the right side surface of the vehicle 100.

  The direction of the optical axis of the camera CA1 is obliquely downward to the front of the vehicle 100, the direction of the optical axis of the camera CA2 is obliquely downward to the right of the vehicle 100, and the direction of the optical axis of the camera CA3 is Cameras CA <b> 1 to CA <b> 4 are installed in the vehicle 100 so that the camera 100 is obliquely downward and the optical axis of the camera CA <b> 4 is obliquely downward to the left of the vehicle 100.

  In FIG. 2, the fields of view of the cameras CA1 and CA4 (that is, the imaging area) are indicated by hatched areas. Camera CA1 images a subject (including a road surface) located within a predetermined area in front of vehicle 100. Camera CA2 images a subject located in a predetermined area on the right side of vehicle 100. Camera CA3 images a subject located in a predetermined area behind vehicle 100. Camera CA4 images a subject located within a predetermined area on the left side of vehicle 100.

  FIG. 3 is a block diagram showing the overall configuration of the driving support system according to the embodiment of the present invention. The driving support system includes each part referred to by the decoders DEC1 to DEC4 and reference numerals 11 to 17 in addition to the cameras CA1 to CA4.

  As the cameras CA1 to CA4, a camera using a CCD (Charge Coupled Devices) or a camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used. Each of the cameras CA1 to CA4 is an internal synchronization camera that does not require an external synchronization signal. That is, each camera includes an internal clock generation circuit that generates an internal clock, and the cameras CA1 to CA4 output video signals at independent timings according to the internal clocks. For this reason, the video signals from the cameras CA1 to CA4 are asynchronous with each other, and each camera can also be called an asynchronous camera.

  The video signal output from each camera is represented by Vin. Each camera sequentially captures images at a predetermined frame period, and outputs a video signal Vin including image data representing an image obtained by the imaging. Since an image obtained by this imaging is input to the image processing circuit 11, the image will be referred to as an “input image” hereinafter. The video signal Vin includes image data Din of an input image, a vertical synchronization signal Vsync, and a horizontal synchronization signal Hsync.

  Assume that an image, a signal, data, and the like related to the camera CA1 correspond to the channel ch1. Similarly, it is assumed that images, signals, data, and the like regarding the cameras CA2, CA3, and CA4 correspond to the channels ch2, ch3, and ch4, respectively.

  Video signals Vin from the cameras CA1, CA2, CA3, and CA4 are synchronized and separated into image data, a vertical synchronization signal Vsync, and a horizontal synchronization signal Hsync by the decoders DEC1, DEC2, DEC3, and DEC4, respectively, and sent to the image processing circuit 11. . The image data sent to the image processing circuit 11 is written and stored in the frame memory area 12a in the RAM unit 12. The RAM unit 12 is formed of SRAM (Static Random Access Memory), SDARM (Synchronous Dynamic Random Access Memory), or the like.

  The LUT area 12b in the RAM unit 12 stores table data representing the correspondence between each pixel of the input image from each camera and each pixel of the output image of the image processing circuit 11. By reading the image data of the input image written in the frame memory area 12a according to the table data, the image data of the output image is obtained, and the output image is displayed as the display image by sending the image data of the output image to the encoder 13. It is displayed on the display screen of the device 14. In the following description, when the display screen is simply used, it means the display screen of the display device 14.

  The table data is sent from the FROM (Flash Read Only Memory) 15 which is a nonvolatile memory to the LUT area 12b and stored in the LUR area 12b at an appropriate timing such as when the driving support system is turned on. There are a plurality of table data, and the plurality of table data are individually stored in the LUT area 12b. Each of the plurality of table data stored in the LUT area 12b forms an LUT (lookup table). The correspondence between each pixel of the input image and each pixel of the output image defined by the LUT is different among the plurality of LUTs.

  The plurality of LUTs formed includes the first and second LUTs. The first LUT is an LUT for generating an output image as shown in FIG. 18 from the input image from each camera and displaying it on the display device 14, and the second LUT is shown in FIG. 20 from the input image from each camera. This is an LUT for generating such an output image and displaying it on the display device 14.

  When the first LUT is used, the all-around bird's-eye view image is displayed using the entire display area of the display screen. An all-around bird's-eye view image is formed by projecting and synthesizing input images from the cameras on the ground. On the display screen of FIG. 18, an illustration of the vehicle body of the vehicle 100 is superimposed and displayed at the center position of the all-around bird's-eye view image. In FIG. 18, an arrow shown in the illustration of the vehicle body indicates the traveling direction of the vehicle 100.

  When the second LUT is used, the entire display area of the display screen is divided into a left area, an upper right area, and a lower right area. Then, the all-around bird's-eye view image is displayed in the left area, the input image of the camera CA3 is enlarged and displayed in the upper right area, and the additional information is displayed in the lower right area. The additional information is the traveling speed of the vehicle 100 measured by a sensor or the like provided in the vehicle 100, the number of revolutions of the engine of the vehicle 100, and the information representing them is a CAN (Control Area Network) provided in the vehicle 100. To the driving support system.

  It is possible to add additional information at the time of generating the output image to obtain a display image as shown in FIG. 20, but it is not shown after the output image including no additional information is generated by the image processing circuit 11. It is also possible to obtain a display image as shown in FIG. 20 by superimposing an image representing additional information on the output image in a separate circuit. The same applies to the illustration of the vehicle body in FIG. Drawing the additional information and the illustration of the vehicle body on the display screen itself is not related to the characteristic technique of the present embodiment, and therefore processing for drawing them is ignored in the following description.

  Any one of the plurality of LUTs formed in the LUT area 12b is selected by an MPU (Micro Processing Unit) 16, and an output image (and a display image) is generated using the selected LUT. The MPU 16 determines which LUT to select according to the operation content to the operation unit 17 or according to the LUT selection signal given to the MPU 16 via the CAN, and displays the LUT selection information according to the determined content. The data is sent to the processing circuit 11 (see FIG. 12). The operation unit 17 receives an operation by the user, and the operation content is transmitted to the MPU 16.

  The video signal of the output image is represented by Vout, and the image data, the vertical synchronization signal, and the horizontal synchronization signal included in the video signal Vout are represented by Dout, Vsync2, and Hsync2, respectively. The video signal Vin of the input image and the video signal Vout of the output image are asynchronous with each other. That is, the image processing circuit 11 generates and outputs the vertical synchronization signal Vsync2 and the horizontal synchronization signal Hsync2 of the video signal Vout of the output image independently of the vertical synchronization signal Vsync and the horizontal synchronization signal Hsync of the video signal Vin of the input image. The image data Dout of the output image is output in synchronization with the generation and output timings of the vertical synchronization signal Vsync2 and the horizontal synchronization signal Hsync2. The encoder 13 combines the vertical synchronization signal Vsync2 and horizontal synchronization signal Hsync2 from the image processing circuit 11 and the image data Dout to generate a video signal Vout, and supplies this to the display device 14 to supply an output image to the display screen. Display above.

  The display device 14 is formed from a liquid crystal display panel or the like. A display device included in a car navigation system or the like may be used as the display device 14 in the driving support system. Each part forming the driving support system is installed in an appropriate part of the vehicle 100, but the display device 14 is installed in the vicinity of the driver's seat of the vehicle 100.

  As shown in FIGS. 4A and 4B, x and y are introduced as symbols representing the horizontal position and vertical position of pixels on the input image or output image, and the position of each pixel on the input image or output image is determined. It is represented by (x, y). X and y for a pixel on the input image represent the positions of the vertical line and horizontal line of the input image to which the pixel belongs, respectively. X and y for a pixel on the output image represent the positions of the vertical line and horizontal line of the output image to which the pixel belongs, respectively. With respect to the input image, when x increases, the position on the input image moves to the right, and when y increases, the position on the input image moves downward. Similarly, with respect to the output image, when x increases, the position on the output image moves to the right, and when y increases, the position on the output image moves downward.

  The number of pixels in the horizontal direction and the vertical direction of the input image is assumed to be Xsize and Ysize, respectively. In the following description, it is assumed that Xsize = 640 and Ysize = 480. Assume that the number of pixels in the horizontal and vertical directions of the output image is the same as that of the input image. Therefore, x takes each integer value that satisfies “0 ≦ x <Xsize”, and y takes each integer value that satisfies “0 ≦ y <Ysize”.

  The video signals Vin and Vout can be converted to interlaced video signals. However, it is assumed that the video signals Vin and Vout are video signals based on the non-interlaced method. Are input and output in the order of raster scanning. That is, for one input image, the positions (0, 0), (1, 0), (2, 0),..., (Xsize-1, 0), (0, 1), (1, 1 ), (2, 1), ..., (Xsize-1, 1), ..., (0, Ysize-2), ..., (Xsize-1, Ysize-2), (0, Ysize) -1),..., (Xsize-1, Ysize-1) are input to the image processing circuit 11 in the order of the pixel signals. The same applies to the image data of the output image. A pixel signal is a value representing the luminance or luminance and color of a pixel forming an image. Image data of the input image is formed by the pixel signal of each pixel forming the input image, and image data of the output image is formed by the pixel signal of each pixel forming the output image.

  In this example, to simplify the description, it is assumed that the frame period of the output image is the same as that of the input image. Therefore, the generation periods of the vertical synchronization signals Vsync and Vsync2 are the same. In order to further simplify the description, in this example, the input cycle of the pixel signal of an input image (input cycle to the image processing circuit 11) and the output cycle of the pixel signal of the output image (output cycle from the image processing circuit 11). ) Are the same (pixel clocks are the same).

As shown in FIG. 5, four frame memories are provided for each camera in the frame memory area 12a of FIG. Four frame memories M ₁ [1] to M ₁ [4] are provided for the camera CA1, and four frame memories M ₂ [1] to M ₂ [4] are provided for the camera CA2. Are provided with four frame memories M ₃ [1] to M ₃ [4], and four frame memories M ₄ [1] to M ₄ [4] are provided for the camera CA4. Each frame memory can store one piece of image data of an input image. The input images periodically acquired by the camera CA1 are 4 in the order of M ₁ [1], M ₁ [2], M ₁ [3], M ₁ [4], M ₁ [1]. The input image before the frame is stored cyclically (cyclically) so as to be overwritten with the input image of the latest frame. The same applies to the cameras CA2 to CA4.

  The driving support system according to the present embodiment has a function of reducing a delay from when an input image is acquired until an output image based on the input image is displayed. The basic principle of this function will be described.

Consider a case where an output image is generated from each input image using the first LUT so that the all-around bird's-eye view image is displayed using the entire display area of the display screen as shown in FIG. Reference is made to FIG. 6 to be compared with FIG. 19, focusing on only the image corresponding to ch4. FIG. 6 shows the relationship between the input timing of the ch4 input image and the output timing of the output image. The input images of ch4 in the first, second, third, fourth, and fifth frames are sequentially supplied to the image processing circuit 11, and they are supplied to the ch4 frame memory (that is, the frame memory M ₄ [1] ˜). M ₄ [4]) is cyclically stored (the input image of the first frame is not shown). The image processing circuit 11 generates the vertical synchronization signal Vsync2 for the output images 301, 302, 303, and 304 at timings t1, t2, t3, and t4, respectively.

In the example shown in FIG. 6, at the timing t1, all the image data of the input image of ch4 of the first frame is held in the frame memory area 12a, while the image data of the input image of ch4 of the second frame is the frame The data is being written to the memory area 12a. However, the area where the input image of ch4 is reflected is located below the output image 301, and at the start timing t _A when the image data of the input image of ch4 is actually read from the frame memory area 12a, the second frame All the image data of the input image of ch4 is written in the frame memory area 12a (note that timing t _A is a timing between timings t1 and t2).

  Accordingly, in this case, the image processing apparatus 11 generates the output image 301 using the ch4 input image of the second frame. Similarly, output images 302, 303, and 304 are generated and displayed using the third, fourth, and fifth frame ch4 input images, respectively. In FIG. 6, the numerical values described in the portion corresponding to ch4 in the output images 301 to 304 represent the frame number of ch4.

In the example shown in FIG. 6, the entire image data of ch4 input image of the second frame is written in the frame memory area 12a at the timing t _A, as shown in FIG. 7, the second frame at time t _A ch4 Even if all the image data of the input image is not written in the frame memory area 12a, the output image 301 can be generated using the ch4 input image of the second frame. Some image data 306 of ch4 input image of the second frame that is not yet written at time t _A is, if not read immediately after the timing t _A, read as part image data of the output image 301 near the timing t2 If so, it is possible to generate the output image 301 using the input image of ch4 of the second frame even in the situation as shown in FIG.

Next, consider a case where an output image is generated from each input image using the second LUT so that a display image as shown in FIG. 20 is displayed. Reference should be made to FIG. 8 to be compared with FIG. 21, focusing on only the image corresponding to ch3. FIG. 8 shows the relationship between the input timing of the ch3 input image and the output timing of the output image. The input images of ch3 in the first, second, third, fourth, and fifth frames are sequentially supplied to the image processing circuit 11, and they are supplied to the ch3 frame memory (that is, the frame memory M ₃ [1] to M ₃ [4]) is cyclically stored (the input image of the first frame is not shown). The image processing circuit 11 generates the vertical synchronization signal Vsync2 for the output images 311, 312, 313, and 314 at timings t11, t12, t13, and t14, respectively.

In the example shown in FIG. 8, at the timing t11, all image data of the input image of ch3 of the first frame is held in the frame memory area 12a, while the image data of the input image of ch3 of the second frame is the frame. The data is being written to the memory area 12a. Since it is necessary to draw an image based on the ch3 input image in the upper right area of the display screen immediately after the timing t11, the ch3 input image of the first frame is applied to the upper right area of the output image 311. On the other hand, the corresponding region of ch3 in the all-around bird's-eye image to be drawn in the left region of the output image 311 is located below the output image 311 and the image data of ch3 for this region is actually stored in the frame memory region. in start timing t _B read from 12a, the entire image data of ch3 input image of the second frame is written in the frame memory area 12a (Incidentally, the timing t _B is the timing between the timings t11 and t12 is there).

  Therefore, the image processing apparatus 11 can also use the input image of ch3 of the second frame for the corresponding region of ch3 in the all-around bird's-eye view image included in the output image 311. In addition, the ch3 input images of the first frame and the second frame are mixed. The same applies to the output images 312 to 314. Thus, if the latest information can be presented to the driver only with the image of ch3 at the bottom of the screen, there is a safety merit. In FIG. 8, the numerical value described in the portion corresponding to ch3 in the output images 311 to 314 represents the frame number of ch3.

  In order to generate the output image as described above, basically the following determination process is performed. For simplification of description, consider a case where an output image is generated by fitting an image obtained by cutting out a partial rectangular area of the input image of ch4 into a predetermined area of the output image as it is. FIG. 9 shows the relationship between the input image and output image of ch4 corresponding to this situation. As a specific example, an image in a rectangular area having a line segment connecting position (100, 100) and position (300, 300) as a diagonal line is cut out from the input image of ch4, and the cut-out image is moved to position (50, 100) is assumed to be inserted into the output image. The cut-out image is fitted into a rectangular area in the output image having a line segment connecting the position (50, 100) and the position (250, 300) as a diagonal line. In this case, if the pixel signal at the position (100, 100) of the latest frame is written and readable at the time of reading the pixel signal of the pixel at the position (50, 100) on the output image, the latest An input image of ch4 of the frame can be output.

  In general, when an image of a pixel at the position (Xin, Yin) of the input image is displayed at the position (Xout, Yout) of the output image, the horizontal line (Yin + α) of the frame of interest at the time of outputting the pixel signal of the horizontal line of Yout. Are written in the frame memory area 12a, an output image can be generated using the input image of the frame of interest. Here, α represents the latency (delay time) related to the processing itself in the circuit, represented by the access time to the SDRAM, by the number of lines. The value of α is determined in advance depending on the processing speed of the system. If the circuit does not use intra-frame interpolation (intra-field interpolation when the video signal is an interlace video signal), it is sufficient to set α to 1. Hereinafter, it is assumed that α = 1.

Therefore, in the example shown in FIG. 9, since Yout = 50 and (Yin + α) ≈ (100 + 1) = 101, each pixel signal of the 101st horizontal line of the frame of interest at the output time of the pixel signal of the 50th horizontal line. Is written in the frame memory area 12a, an output image can be generated using the input image of the frame of interest. That is, attention is paid to the frame memory (M ₄ [3] in the example of FIG. 10) in which the image data of the latest input image is to be written among the ch4 frame memories M ₄ [1] to M ₄ [4]. Then, it is only necessary to determine whether or not each pixel signal of the 101st horizontal line of the frame memory of interest is written at a timing when each pixel signal of the 50th horizontal line is to be output. Then, if written, an output image is generated using the image data in the focused frame memory. If not written, the frame memory (in FIG. 10) holds the input image of the previous frame. In the example, an output image is generated using the image data in M ₄ [2]).

  For simplification of explanation, the example in which the cutout image is directly inserted into the output image has been described. However, when the image data of the input image is subjected to lens distortion correction, bird's eye conversion and / or image enlargement / reduction processing, The relationship between the pixels of the input image and the pixels of the output image becomes very complicated. Note that the bird's-eye conversion means conversion for obtaining a bird's-eye image by projecting an input image onto the ground, and the all-around bird's-eye image is also obtained using the bird's-eye conversion.

  FIG. 11 schematically shows table data of the first LUT for obtaining the all-around bird's-eye view image corresponding to FIG. FIG. 11 shows LUT table data suitable for an image size of (640 × 480), and the LUT table data includes (640 × 480) pieces of input pixel position information. Each input pixel position information includes information indicating the horizontal position and vertical position of the pixel and a channel number. The input pixel position information at the address (p, q) on the LUT is input pixel position information for the pixel at the position (p, q) on the output image (p and q are 0 ≦ p <Xsize and 0 ≦ q <An integer satisfying Ysize).

  For example, the input pixel position information (600, 80, ch1) is stored at the address (0, 0) on the LUT in FIG. 11, which is the pixel at the position (600, 80) in the input image of ch1. Is to be read out as the pixel signal of the pixel at the position (0, 0) in the output image. Further, for example, the input pixel position information (10, 78, ch2) is stored at the address (1, 0) on the LUT in FIG. 11, and this indicates that “the position (10, 78) in the input image of ch2”. The pixel signal of the pixel should be read out as the pixel signal of the pixel at the position (1, 0) in the output image. Further, for example, the input pixel position information (601, 79, ch1) is stored in the address (0, 1) on the LUT in FIG. The pixel signal of the pixel should be read out as the pixel signal of the pixel at the position (0, 1) in the output image.

  In the example shown in FIG. 9, the difference in line position between the horizontal line on the input image and the horizontal line on the output image with respect to the horizontal line has always been 50 lines, but FIG. When an LUT as shown is used, the value of (Yin-Yout) representing this difference varies depending on the position of the pixel on the output image. In order to display the latest image, it is necessary to accurately grasp the value of (Yin−Yout).

Therefore, (Yin−Yout) when assuming that the pixel at the position (Xin, Yin) on the input image is associated with the position (Xout, Yout) on the output image by LUT is set as ΔY, Thus, all ΔY values are obtained for each channel in a lump. Then, the maximum value ΔY _MAX of ΔY values is obtained for each channel, and the pixel signal of the (Yout_top + ΔY _MAX + α) th horizontal line is written at the read timing of the pixel signal of the first horizontal line where the block is displayed. The latest frame memory is selected as the frame memory for generating the output image. Yout_top represents the number of the first horizontal line on which the above chunk is displayed. Thereby, the video delay can be minimized.

The significance of ΔY _{MAX and the} like will be described with specific examples. In the example shown in FIG. 9, [Delta] Y _MAX is 50 for ch1 for [Delta] Y are all 50 for ch1. On the other hand, the number of the first horizontal line on the output image on which an image based on the input image of ch1 is displayed is Yout_top = 50. Therefore, (Yout_top + ΔY _MAX + α) for ch1 is 101 from 50 + 50 + 1 = 101. Accordingly, the pixel signal of the 101 (= Yout_top + ΔY _MAX + α) th horizontal line is written at the read timing of the pixel signal of the first horizontal line on which the input image of ch1 is displayed, that is, the 50 (= Yout_top) th horizontal line. The latest frame memory is selected as the frame memory for generating the output image.

In the example shown in FIG. 11, ΔY for the address (0, 0) on the LUT is 80 from (Yin−Yout) = 80−0, and ΔY for the address (0, 1) on the LUT is (Yin−Yout) = 79-1 to 78. These two ΔY are ΔY with respect to ch1. Regarding ch1, ΔY is similarly obtained for other addresses on the LUT, and the maximum value of ΔY thus obtained is set as ΔY _MAX for ch1. On the other hand, when the LUT shown in FIG. 11 is used, an image based on the input image of ch1 is displayed, and the first horizontal line number Yout_top on the output image is 0 (input at the address (0, 0) on the LUT). (The pixel position information is for ch1). Therefore, if ΔY _MAX for ch1 is 120, (Yout_top + ΔY _MAX + α) for ch1 is 121 from 0 + 120 + 1 = 121. ΔY _MAX and Yout_top are similarly determined for each of ch2, ch3, and ch4. When the LUT of FIG. 11 is used, only Yout_top for ch4 takes a value other than zero.

Hereinafter, a represent a [Delta] Y _MAX for ch1~ch4 in ΔY _MAX1 ~ΔY _MAX4, to be representative of the Yout_top for ch1~ch4 in Yout_top ₁ ~Yout_top _4. The value of ΔY _MAX1 ~ΔY _MAX4 and Yout_top ₁ ~Yout_top ₄ for each LUT formed in the LUT region 12b of FIG. 3 is defined, by specifying the LUT to be used to form an output image, which is the designated the value of ΔY _MAX1 ~ΔY _MAX4 and Yout_top ₁ ~Yout_top ₄ based on the table data in the LUT are determined automatically. Table data in each LUT is because it is intended to be defined in advance, the value of ΔY _MAX1 ~ΔY _MAX4 and Yout_top ₁ ~Yout_top ₄ for each LUT may have a be predetermined.

  As examples relating to the output image generation processing described above, first to third examples will be exemplified below. The matters described in one embodiment can be applied to other embodiments as long as no contradiction arises.

<< First Example >>
First, the first embodiment will be described. FIG. 12 is a partial block diagram of the driving support system that also shows the internal configuration of the image processing circuit 11 according to the first embodiment. The image processing circuit 11 according to the first embodiment includes each part referred to by reference numerals 21 to 25 in FIG. The RAM unit 12 is accessed from the MPU 16 in addition to the write address generation circuit 21, the LUT address generation circuit 23, and the read address generation circuit 24 (the state of access by the MPU 16 is not shown). Therefore, data reading / writing with respect to the RAM unit 12 is actually executed under the arbitration control by the arbitration circuit 25. However, for the sake of simplification of explanation, the following explanation is performed by ignoring the existence of the arbitration circuit 25. And

  Although the number of channels is 4, the operation of the image processing circuit 11 for the data of each channel is the same among the four channels. Therefore, the variable i representing the channel number is introduced, and the explanation is simplified using i as appropriate (i is 1, 2, 3, or 4). For example, the decoder DECi means the decoder DEC1 when considering the channel ch1, and means the decoder DEC2 when considering the channel ch2.

The write address generation circuit 21 is currently shown based on the vertical synchronization signal Vsync and horizontal synchronization signal Hsync from the decoder DECi of FIG. 3 and a pixel clock (not shown) synchronized with the output interval of the pixel signal from the decoder DECi. 3 generates an address (hereinafter referred to as a write address) on the RAM unit 12 to which a pixel signal (pixel signal forming the image data Din) sent from the decoder DECi of 3 is to be written. The write address is an address on the frame memories M _i [1] to M _i [4] in the RAM unit 12.

The pixel signal from the decoder DECi in FIG. 3 is written to the address specified by the write address in the frame memories M _i [1] to M _i [4]. The write address generation circuit 21 also manages in which frame memory of the frame memories M _i [1] to M _i [4] the pixel signal input at this time is to be stored. The write address is also given to the read address generation circuit 24.

  The video output address generation circuit 22 is driven asynchronously with the video signal Vin, generates and outputs a vertical synchronization signal Vsync2 and a horizontal synchronization signal Hsync2 for an output image, and is synchronized with the vertical synchronization signal Vsync2 and the horizontal synchronization signal Hsync2. The generated video output address is generated, and the video output address is sent to the LUT address generation circuit 23.

  The LUT address generation circuit 23 selects one LUT out of a plurality of LUTs stored in the LUT area 12b of the RAM unit 12 according to the LUT selection information sent from the MPU 16. The selected LUT is hereinafter referred to as “designated LUT”. Then, the LUT address generation circuit 23 generates an address on the designated LUT (hereinafter referred to as LUT address) corresponding to the video output address. This LUT address is sent to the RAM unit 12, data at the LUT address on the designated LUT is read from the RAM unit 12 as LUT information, and this LUT information is sent to the read address generation circuit 24. The LUT information is the same as the input pixel position information such as (600, 80, ch1) shown using FIG.

A read address generating circuit 24, a timing relationship between the output timing of the image data input timing and the output image of the image data of the input image is created according to ΔY _MAX1 ~ΔY _MAX4 and Yout_top ₁ ~Yout_top ₄ to the specified LUT Based on the read memory switching information, a frame memory storing the latest input image that can be used to generate an output image is selected for each channel. The input timing of the image data of the input image is specified by the write address. The output timing of the image data of the output image is specified based on the vertical synchronization signal Vsync2 and the horizontal synchronization signal Hsync2, or based on the video output address.

The selected frame memory is hereinafter referred to as “designated frame memory”. That is, based on the timing relationship and the read memory switching information, one of the ch1 frame memories M ₁ [1] to M ₁ [4] is selected as the designated frame memory for ch1, and for ch2 One of the frame memories M ₂ [1] to M ₂ [4] is selected as the designated frame memory for ch2, and one of the frame memories M ₃ [1] to M ₃ [4] for ch3 is selected. one was chosen as the designated frame memory for ch3, selects one of the frame memory _{M 4 [1] ~M 4 [} 4] for the ch4 as specified frame memory for ch4. This selection is performed through switching determination of the designated frame memory based on the read memory switching information (details will be described later).

  The read address generation circuit 24 generates a read address on the designated frame memory based on the LUT information in order to read out the pixel signal at the channel and the pixel position designated by the LUT information from the designated frame memory. The pixel signal written to the read address is output to the encoder 13 as a pixel signal of a pixel at a position corresponding to the video output address of the output image.

  Hereinafter, unless otherwise specified, it is assumed that the designated LUT is the first LUT and the table data of the first LUT is the same as that shown in FIG.

  Regarding video output, the entire period includes an effective video period in which pixel signals at each position (0, 0) to (Xsize-1, Ysize-1) on the output image are actually output, and a blanking period. And an invalid video period (see FIG. 4B). Since the video output address output from the video output address generation circuit 22 is an address including an invalid video period, the horizontal component of the video output address (in other words, the component indicating the number of the vertical line) and the vertical component ( In other words, the component representing the number of the horizontal line) may indicate a value greater than or equal to Xsize and Ysize.

  In the present embodiment and each embodiment described later, the maximum values of the horizontal component and the vertical component of the video output address including the invalid video period from the video output address generation circuit 22 are Xsize 'and Ysize', respectively. That is, the video output address generation circuit 22 has (0,0), (1,0), (2,0),..., (Xsize ′, 0), (0,1), (1,1). , (2, 1),..., (Xsize ′, Ysize ′) are generated in this order. Here, Xsize '> Xsize and Ysize'> Ysize.

  The position (x, y) described above represents the coordinates of the pixel actually drawn on the output image. A coordinate system in which pixels actually drawn on the output image are arranged is called an effective video coordinate system. On the other hand, a coordinate system including an invalid video period is called an invalid video coordinate system. The effective video coordinate system is included in the invalid video coordinate system. The coordinates on the invalid video coordinate system are represented by (x ′, y ′). Hereinafter, the coordinates (x, y) on the effective video coordinate system are also referred to as effective video coordinates, and the coordinates (x ′, y ′) on the invalid video coordinate system are also referred to as invalid video coordinates. Now, the invalid video coordinate (x ′, y ′) corresponding to the valid video coordinate (0, 0) is (20, 20), and the actual pixel on the output image from the invalid video coordinate (20, 20). Are arranged.

  The coordinates one horizontal line before the valid video coordinates (0, 0) correspond to the invalid video coordinates (20, 19). When the invalid video coordinates (20, 19) are expressed by the valid video coordinates, (0, -1) The following will be described as it is. For simplification of description, when the video output address output from the video output address generation circuit 22 is an address belonging to the invalid video period (for example, (0, −1)), the LUT address generation circuit 23 Assume that the LUT address (0, 0) is output.

  The timing relationship between the pixel signal read from the frame memory through the generation of the LUT address based on the video output address and the vertical synchronization signal Vsync2 and horizontal synchronization signal Hsync2 is between the arbitration circuit 25 and the encoder 13. It is assumed that adjustment is performed by a provided FIFO (First In, First Out) (not shown). In addition, a synchronization addition circuit (not shown) that receives the video signal Vout is provided in the previous stage of the encoder 13, and the synchronization addition circuit adds a synchronization signal of an invalid video period to the video signal Vout and outputs signals other than the synchronization signal. A signal level is determined.

  The description of the pixel signal reading operation is made concrete by giving specific numerical values. The video output address generation circuit 22 considers the latency including the access time to the SDRAM, and generates a video output address corresponding to the pixel one horizontal line earlier than the timing at which the focused pixel is actually displayed. For example, when the pixel signal at the position (0, 0) of the output image is to be output to the encoder 13, the video output address generation circuit 22 sends the video output address (0, −1) to the LUT address generation circuit 23. The LUT address generation circuit 23 generates an LUT address corresponding to the video output address (0, −1). This LUT address is an address on the designated LUT in which input pixel position information (that is, (600, 80, ch1); see FIG. 11) corresponding to the position (0, 0) on the output image is stored. Actually, the LUT address is an address in the entire memory space of the SDRAM, but as described with reference to FIG. 11, the input pixel position information corresponding to the position (0, 0) on the output image is stored. Since the address on the current LUT is regarded as (0, 0), the LUT address is (0, 0).

Data at the LUT address (0, 0) is read as LUT information. This LUT information is input pixel position information (600, 80, ch1) corresponding to the position (0, 0) on the output image (see FIG. 11). Now, assuming that the designated frame memory for ch1 is the frame memory M ₁ [3], the read address generation circuit 24, based on this LUT information, the pixel signal of the pixel at the position (600, 80) in the input image of ch1. The address on the frame memory M ₁ [3] in which is written is generated as a read address. The pixel signal read according to this read address is output to the encoder 13 as a pixel signal at the position (0, 0) of the output image. By executing the above operation for each position on the output image, the image data Dout representing the entire output image is output to the encoder 13, and the output image is displayed on the display screen.

FIG. 13 shows the configuration of the read memory switching information. The read memory switching information includes switching determination timing information that defines switching determination timing and switching determination condition information that defines switching determination conditions for each channel. The switching determination timing information for chi is (Yout_top _i −α), and the switching determination condition information for chi is (Yout_top _i + ΔY _MAXi ) (as described above, i is 1, 2, 3, or 4). The read memory switching information is set for each LUT, and when the designated LUT is switched from one LUT to another LUT, the read memory switching information adapted from the MPU 16 to the switching destination LUT during the vertical blanking period at the time of switching. Is sent to the read address generation circuit 24.

  The read address generation circuit 24 targets each channel and determines whether or not to switch the frame memory from which the input image of the target channel is read at the switching determination timing defined by the switching determination timing information. This determination is performed by determining whether or not the switching determination condition for the target channel is satisfied.

A more specific description will be given focusing on ch4 (assuming the situation of FIG. 10). (Yout_top ₄ -α) th horizontal line of the output image, i.e., (Yout_top ₄ -α) represented by the switching determination timing information ch4 th output pixel signals of the horizontal lines to the encoder 13 reads from the RAM 12 The timing to be performed is the switching determination timing for ch4. In order to recognize the switching determination timing, the read address generation circuit 24 refers to the vertical synchronization signal Vsync2, the horizontal synchronization signal Hsync2, and the video output address as appropriate.

At this switching determination timing, the read address generation circuit 24 determines whether to switch the frame memory from which the ch4 input image is to be read based on the write address. If the designated frame memory of ch4 before this determination is the frame memory M ₄ [2], the image data of the latest input image of ch4 is being written from the frame memory M ₄ [2] to the designated frame memory. It is determined whether or not to switch to the frame memory M ₄ [3] that is assumed to be.

Decision on this switching, whether the most recent (Yout_top ₄ + ΔY _MAX4) in the input image of th pixel signals of the horizontal lines of ch4 is, in the switching judgment timing of ch4, already written in the frame memory M ₄ [3] This is done by determining whether or not. That, (Yout_top ₄ + ΔY _MAX4) th pixel signals of horizontal lines represented by the switching determination condition information ch4 is, in the switching judgment timing of ch4, whether already written in the frame memory M ₄ [3] This determination is made regarding the switching. Then, if it is determined based on the write address that it has already been written, the frame memory M ₄ [3] can be used to form an output image, so the ch4 command frame memory is used as the frame memory M ₄ [2]. Is switched to the frame memory M ₄ [3], and if it is not determined that it has already been written, the ch4 command frame memory remains the frame memory M ₄ [2].

  As described above, the read address generation circuit 24 takes into account the timing relationship between the input timing of image data from each camera and the output timing of image data of the output image, and designates it for use in forming an output image for each channel. A frame memory is selected from a plurality of frame memories. At this time, this selection operation is performed at a switching determination timing (in other words, selection timing) set individually for each channel. As can be understood from the above description, the read memory switching information including the switching determination timing information is determined depending on how and where the image based on the input image of each channel is arranged on the output image. The determined contents are expressed by LUT.

The switching determination condition is dependent on ΔY _MAX1 ~ΔY _MAX4. On the other hand, automatically ΔY _MAX1 ~ΔY _MAX4 determines if LUT table data defining the correspondence between each pixel on each pixel and the output image on the input image is Sadamare. Therefore, it can be interpreted that the switching determination of the designated frame memory based on the switching determination condition is executed based on the table data.

  When the first LUT for obtaining a display image as shown in FIG. 18 is a designated LUT, the number of switching determination timings for each channel is one for one output image. Depending on the case, it may be more than one. For example, when the second LUT for obtaining a display image as shown in FIG. 20 is the designated LUT, two areas in which an image based on the input image of ch3 is arranged are provided in one output image. Therefore, the number of ch3 switching determination timings can be two for one output image. In this case, a switching determination condition is set for each of the two switching determination timings. Then, at each switching determination timing, whether the corresponding switching determination condition is satisfied or not is determined, and the designated frame memory for ch3 is selected based on the determination result.

<< Second Example >>
In the first embodiment, the switching determination of the designated frame memory is performed in line units to reduce the delay in video output. However, if the circuit scale is allowed to increase, the switching determination may be performed in pixel units. Is possible. If the switching determination of the designated frame memory is performed on a pixel basis, the delay reduction effect is further enhanced. An embodiment in which the switching determination of the designated frame memory is performed on a pixel basis will be described as a second embodiment. The second embodiment corresponds to an embodiment obtained by modifying a part of the first embodiment. For this reason, only the differences from the first embodiment will be described.

First, paying attention to each LUT in the LUT area 12b in FIG. 3, ΔXY is obtained instead of ΔY described above. Specifically (see FIG. 11), the pixel at the position (Xin, Yin) on the input image (that is, the invalid video coordinates (Xin ′, Yin ′) including the invalid video period) is displayed on the output image by the LUT. (Xin + Yin · Xsize ′ − (Xout + Yout · Xsize ′) when considered to be associated with the position (Xout, Yout) (that is, the invalid video coordinates (Xout ′, Yout ′) including the invalid video period) ) As ΔXY, and all the values of ΔXY are obtained for each channel in a lump on the LUT. Then, the maximum value ΔXY _MAX of ΔXY values is obtained for each channel, and the pixel signal of the pixel of (XYout_top + ΔXY _MAX + α ′) is written at the read timing of the pixel signal of the first pixel where the block is displayed. The frame memory is selected as a frame memory for generating an output image.

  XYout_top represents the pixel number of the first pixel in which the above-mentioned block is displayed. When the position of the pixel (in other words, the coordinate) is (x, y), that is, the pixel including the invalid video period. When the coordinates are (x ′, y ′), it is defined as XYout_top = x ′ + y ′ · Xsize ′. α ′ represents the latency (delay time) of the processing inside the circuit, represented by the access time to the SDRAM, by the number of pixels. The value of α ′ is determined in advance depending on the processing speed of the system.

The significance of ΔXY and the like will be described with specific examples. Assuming that Xsize ′ = 720 and Ysize ′ = 525, the LUT shown in FIG. 11 is taken as an example. ΔXY for the address (0,0) on the LUT is 58200 from (Xin + Yin · Xsize ′ − (Xout + Yout · Xsize ′)) = 600 + 80 · 720− (0 + 0 · 720) = 58200 and the address (0,1) on the LUT ΔXY with respect to is (Xin + Yin · Xsize ′ − (Xout + Yout · Xsize ′)) = 601 + 79 · 720− (0 + 1 · 720) = 56761 to 56761. These two ΔXY are ΔXY with respect to ch1. Similarly, ΔXY is obtained for other addresses on the LUT related to ch1, and the obtained maximum value of ΔXY is set to ΔXY _MAX for ch1.

On the other hand, when the LUT shown in FIG. 11 is used, an image based on the input image of ch1 is displayed, and the pixel number XYout_top of the first pixel on the output image is 0 (input at the address (0, 0) on the LUT). (The pixel position information is for ch1). Therefore, if it is [Delta] xy _MAX 60000 for temporarily ch1, 'it is, (60000 + α (XYout_top + ΔXY MAX + α)' for ch1 becomes). ΔXY _MAX and XYout_top are similarly determined for each of ch2, ch3, and ch4.

Hereinafter, a represent a [Delta] xy _MAX for ch1~ch4 in ΔXY _MAX1 ~ΔXY _MAX4, to be representative of the XYout_top for ch1~ch4 in XYout_top ₁ ~XYout_top _4. The value of ΔXY _MAX1 ~ΔXY _MAX4 and XYout_top ₁ ~XYout_top ₄ for each LUT formed in the LUT region 12b of FIG. 3 is defined, if Sadamare designation LUT to be used to form an output image, the designated LUT the value of ΔXY _MAX1 ~ΔXY _MAX4 and XYout_top ₁ ~XYout_top ₄ based on the table data in parentheses are automatically determined. Table data in each LUT is because it is intended to be defined in advance, the value of ΔXY _MAX1 ~ΔXY _MAX4 and XYout_top ₁ ~XYout_top ₄ for each LUT may have a be predetermined.

  The internal configuration of the image processing circuit 11 according to the second embodiment is the same as that of FIG. 12, and the operation of each part in the image processing circuit 11 is basically the same as that of the first embodiment. However, the read memory switching information is different from that of the first embodiment in order to perform the pixel unit switching determination.

FIG. 14 shows the configuration of read memory switching information according to the second embodiment. The read memory switching information includes switching determination timing information that defines switching determination timing and switching determination condition information that defines switching determination conditions for each channel. The switching determination timing information for chi is (XYout_top _i −α ′), and the switching determination condition information for chi is (XYout_top _i + ΔXY _MAXi ) (as described above, i is 1, 2, 3, or 4). . The read memory switching information is set for each LUT, and when the designated LUT is switched from one LUT to another LUT, the read memory switching information adapted from the MPU 16 to the switching destination LUT during the vertical blanking period at the time of switching. Is sent to the read address generation circuit 24.

  The read address generation circuit 24 targets each channel and determines whether or not to switch the frame memory from which the input image of the target channel is read at the switching determination timing defined by the switching determination timing information. This determination is performed by determining whether or not the switching determination condition for the target channel is satisfied.

A more specific description will be given focusing on ch4 (assuming the situation of FIG. 10). (XYout_top ₄ -α ') of the output image th pixel, i.e., (XYout_top ₄ -α represented by the switching determination timing information ch4' outputs a pixel signal of) th pixel to the encoder 13 reads from the RAM 12 The timing to be performed is the switching determination timing for ch4. In order to recognize the switching determination timing, the read address generation circuit 24 refers to the vertical synchronization signal Vsync2, the horizontal synchronization signal Hsync2, and the video output address as appropriate.

The read address generation circuit 24 determines at this switching determination timing whether or not to switch the frame memory from which the input image of ch4 is to be read. If the designated frame memory of ch4 before this determination is the frame memory M ₄ [2], the image data of the latest input image of ch4 is being written from the frame memory M ₄ [2] to the designated frame memory. It is determined whether or not to switch to the frame memory M ₄ [3] that is assumed to be.

Decision on this switching, the pixel signals of the latest in the input image (XYout_top ₄ + ΔXY _MAX4) th pixel ch4 is, in the switching determination timing ch4, whether already written in the frame memory M ₄ [3] This is done by judging. That is, the determination pixel signal (XYout_top ₄ + ΔXY _MAX4) th pixels represented by the switching determination condition information ch4 is, in the switching determination timing ch4, whether already written in the frame memory M ₄ [3] By doing so, a judgment regarding this switching is made. Then, if it is determined based on the write address that it has already been written, the frame memory M ₄ [3] can be used to form an output image, so the ch4 command frame memory is used as the frame memory M ₄ [2]. Is switched to the frame memory M ₄ [3], and if it is not determined that it has already been written, the ch4 command frame memory remains the frame memory M ₄ [2].

In the first embodiment, sets the switching determination timing in units of lines, the corresponding maximum value of the difference [Delta] Y position of the horizontal line between the input and output image (ΔY _MAX1 ~ΔY _MAX4) switching determination of the designated frame memory based on ( In other words, in the second embodiment, the switching determination timing is set in units of pixels, and the corresponding pixel position difference ΔXY between the input and output images is the maximum value. (in other words, selection of the designated frame memory) switching determination of the designated frame memory based on (ΔXY _MAX1 ~ΔXY _MAX4) is performed.

  As described in the first embodiment, depending on the designated LUT, the number of switching determination timings for a certain channel can be two or more for one output image. For example, when the second LUT is a designated LUT, the number of ch3 switching determination timings can be two for one output image. In this case, a switching determination condition is set for each of the two switching determination timings. Then, at each switching determination timing, whether the corresponding switching determination condition is satisfied or not is determined, and the designated frame memory for ch3 is selected based on the determination result.

<< Third Example >>
In the first and second embodiments, the designated frame memory is selected in the image processing circuit 11, but some of the functions performed by the image processing circuit 11 of the first and second embodiments are as follows. Needless to say, the MPU 16 serving as the arithmetic processing means can be used. As an embodiment embodying this, a third embodiment will be described.

  FIG. 15 is a partial block diagram of the driving support system that also shows the internal configuration of the image processing circuit 11 according to the third embodiment. The image processing circuit 11 according to the third embodiment includes each part referred to by reference numerals 21 to 23, 24a, 25 and 26 in FIG. Also in the third embodiment, description will be made ignoring the presence of the arbitration circuit 25.

  The operations of the write address generation circuit 21, the video output address generation circuit 22 and the LUT address generation circuit 23 of FIG. 15 are the same as those shown in the first embodiment. The LUT information read from the RAM unit 12 according to the LUT address generated by the LUT address generation circuit 23 is sent to the read address generation circuit 24a.

  Frame memory designation information is supplied from the MPU 16 to the read address generation circuit 24a. The frame memory designation information is information that defines a designated frame memory for each channel. The read address generation circuit 24a identifies the designated frame memory of each channel according to the frame memory designation information, and reads the pixel signal at the channel and pixel position designated by the LUT information from the designated frame memory based on the designated frame. A read address on the memory is generated. The pixel signal written to the read address is output to the encoder 13 as a pixel signal of a pixel at a position corresponding to the video output address of the output image.

  In order for the MPU 16 to create the frame memory designation information, the MPU 16 needs to accurately grasp the input timing of the image data of the input image and the output timing of the image data of the output image. Therefore, in the configuration shown in FIG. 15, the write address generated by the write address generation circuit 21 is given to the register 26, and the write address at the time when the vertical synchronization signal Vsync 2 is generated is latched in the register 26 and given to the MPU 16. On the other hand, when the vertical synchronization signal Vsync2 is generated, an interrupt request for the MPU 16 is generated. As a result, the MPU 16 grasps the interrupt request occurrence time as the occurrence time of the vertical synchronization signal Vsync2, and simultaneously recognizes the write address at that time. That is, the MPU 16 recognizes the timing relationship between the input timing of the image data of the input image and the output timing of the image data of the output image.

  The MPU 16 recognizes the read memory switching information as shown in FIG. 13 adapted to the designated LUT in advance. Then, the MPU 16 recognizes the switching determination timing using a timer (not shown) built in the MPU 16 and determines whether the switching determination condition is satisfied or not at the switching determination timing. The frame memory designation information is updated according to the determination result (if the switching determination timing is the same as when Vsync2 is generated, it is not necessary to use a timer).

The operation of the MPU 16 for ch4 will be described assuming that the designated LUT is the first LUT and the read memory switching information shown in FIG. 13 is determined. (Yout_top ₄ -α) th horizontal line of the output image, i.e., (Yout_top ₄ -α) represented by the switching determination timing information ch4 th output pixel signals of the horizontal lines to the encoder 13 reads from the RAM 12 The timing to be performed is the switching determination timing for ch4. The MPU 16 recognizes the switching determination timing by setting a timer based on the generation time of Vsync2 so that an interrupt request is generated by the timer at the switching determination timing. FIG. 16 shows the state of the interrupt request by the timer.

  The MPU 16 then determines whether to switch the frame memory from which the ch4 input image is to be read based on the write address given from the register 26 at this switching determination timing. The determination method itself is the same as that described in the first embodiment, except that the determination subject is changed to the MPU 16. The determination result is reflected in the frame memory designation information.

As described in the first embodiment, depending on the designated LUT, the number of switching determination timings for a certain channel can be two or more for one output image. For example, when the second LUT is a designated LUT, the number of ch3 switching determination timings can be two for one output image. In this case, a switching determination condition is set for each of the two switching determination timings. Then, at each switching determination timing, the MPU 16 may determine whether the corresponding switching determination condition is satisfied or not, and select a designated frame memory for ch3 based on the determination result. For example, when a first 2LUT as specified LUT, for the ch3, timing represented by a generation timing of Vsync2 (Yout_top ₃ -α) is a switching determination timing, the interrupt request by the timer in the latter timing It should be generated (see FIG. 17).

  Moreover, although the example which performs the switching determination of the designation | designated frame memory per line was shown, the matter described in 2nd Example is also applicable to 3rd Example. That is, if the circuit scale is allowed to increase, the switching determination can be performed on a pixel basis as shown in the second embodiment.

  Further, when the switching determination of the designated frame memory is performed mainly by hardware as in the first or second embodiment, the time management becomes accurate and the delay in video output can be reduced well. . On the other hand, in the third embodiment, since the MPU 16 is involved in the real-time control, an error may occur in time management depending on the processing priority order of the interrupt by Vsync2 or the interrupt by the timer. For this reason, it can be said that the effect of reducing the delay in video output is slightly smaller than that of the first embodiment, but there is an advantage that the circuit configuration of the read address generation circuit can be simplified.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 5 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the above example, the number of frame memories provided for one channel is 4, but the number can be any number of 2 or more.

[Note 2]
In the above example, LUT (table data) is prepared as correspondence information that defines the correspondence between each pixel on the input image and each pixel on the output image, and an output image is formed from each input image using the LUT. However, the use of the LUT is not essential if an output image equivalent to the case of using the LUT can be obtained. For example, an output image may be obtained by performing geometric transformation according to the correspondence information on each input image.

[Note 3]
The driving support system in FIG. 3 can be realized by hardware or a combination of hardware and software. In particular, the function of each part in the image processing apparatus 11 shown in FIG. 12 or 15 can be realized by hardware, software, or a combination of hardware and software.

[Note 4]
Although the embodiment in the case where the present invention is applied to the driving support system for the vehicle has been described above, the present invention is not limited to the vehicle application. That is, the present invention can be applied to various apparatuses and systems (such as a monitoring system) that obtain an output image by performing image processing on input images from a plurality of cameras that are asynchronous with each other. Applicable to all. Depending on the contents of the image processing, the time required for the image processing increases. In this case, the above-described latency (α or α ′) can be set to a large value.

  Further, the configuration in which the video signal from the camera is input to the image processing circuit via the decoder has been described in the above embodiment, but the input / output mode of the video signal according to the present invention is limited to that in the above embodiment. Not. For example, even if the video signal from the camera is received via a network, the present invention can be applied as long as the video signal can be once decoded and stored in the frame memory. However, in this case, assuming that there is a possibility that the input video signal may be interrupted in the middle of the frame, it is necessary to wait for the reception of the pixel signals of all the pixels used for display among the pixels forming the input image. It is necessary to display.

  Therefore, when the video signal reception via the network is applied to the first embodiment, for example, the pixel signal of the pixel on the horizontal line obtained by adding one line to the last horizontal line to which the pixel used for display belongs is received. Sometimes, a switching determination of a designated frame memory (that is, a frame memory used for display) is performed. Similarly, when the video signal reception via the network is applied to the second embodiment, for example, when the pixel signal of the pixel next to the last pixel used for display is received, the switching determination of the designated frame memory is performed. Like that. In the third embodiment, since the switching determination timing is predicted using a timer, video signal reception via the network cannot be used as it is in the third embodiment. When the video signal reception via the network is applied to the third embodiment, an interrupt request is generated to the MPU 16 when the last pixel used for display is received, and the MPU 16 corresponding to the interrupt request is generated. It is necessary to determine whether to switch the designated frame memory during the interrupt routine executed in the program.

[Note 5]
For example, it can be considered as follows. An image processing apparatus that generates an output image from an input image acquired from each camera is mainly formed by the image processing circuit 11, the RAM unit 12, and the MPU 16 of FIG. In the first or second embodiment, the selection means for selecting the designated frame memory is included in the image processing circuit 11, but in the third embodiment, the selection means is included in the MPU 16.

It is the top view which looked at the vehicle to which the driving assistance system concerning the embodiment of the present invention is applied from the upper part. It is the figure which looked at the vehicle of FIG. 1 from the diagonally left front. 1 is an overall configuration block diagram of a driving support system according to an embodiment of the present invention. It is a figure which shows the structure of the input image and output image of the image processing circuit of FIG. It is a figure which shows the frame memory group provided in the RAM part of FIG. FIG. 4 is a diagram for explaining a method for generating an output image by the driving support system of FIG. FIG. 4 is a diagram for explaining a method for generating an output image by the driving support system of FIG. FIG. 4 is a diagram for explaining a method for generating an output image by the driving support system in FIG. 3, and is a diagram illustrating a relationship between an input timing of an input image and an output timing of the output image. FIG. 4 is a diagram for explaining a method for generating an output image by the driving support system of FIG. 3, and is a diagram illustrating a state in which a cut-out image from the input image is fitted in the output image. It is a figure which shows a mode that image data is written in the frame memory group with respect to ch4. It is a figure which shows the table data of LUT formed in the LUT area | region of FIG. FIG. 4 is a partial block diagram of the driving support system of FIG. 3, showing the internal configuration of the image processing circuit according to the first embodiment of the present invention. FIG. 13 is a diagram illustrating a configuration of read memory switching information applied to the read address generation circuit of FIG. 12 according to the first embodiment of the present invention. FIG. 13 is a diagram illustrating a configuration of read memory switching information applied to the read address generation circuit of FIG. 12 according to the second embodiment of the present invention. FIG. 4 is a partial block diagram of the driving support system in FIG. 3 according to a third embodiment of the present invention and also showing an internal configuration of an image processing circuit. It is a figure which shows the mode of the interruption generation | occurrence | production by the MPU built-in timer concerning 3rd Example of this invention. It is a figure which shows the mode of the interruption generation | occurrence | production by the MPU built-in timer concerning 3rd Example of this invention. It is a figure which shows the example of an output image (display image). FIG. 19 is a diagram illustrating a relationship between input timing of an input image and output timing of an output image in FIG. 18 according to the related art. It is a figure which shows the other example of an output image (display image). FIG. 21 is a diagram illustrating a relationship between input timing of an input image and output timing of an output image in FIG. 20 according to the related art.

Explanation of symbols

CA1 to CA4 Camera 100 Vehicle 11 Image processing circuit 12 RAM section 12a Frame memory area 12b LUT area 16 MPU

Claims (9)

A plurality of frame memories that receive image data of input images sequentially picked up by each of the first to n-th internal synchronous cameras and each store image data of one input image include the internal synchronous camera An input memory unit for each (n is an integer of 2 or more),
Based on the timing relationship between the input timing of the image data from each of the first to n-th internal synchronous cameras and the output timing of the image data of the output image, it is used for forming the output image for each internal synchronous camera. Selecting means for selecting a designated frame memory as a designated frame memory;
Output means for generating and outputting image data of the output image by synthesizing the input images stored in each designated frame memory, and an image processing apparatus comprising:
The selection means includes
Each input image is reflected in the output image, based on the position on the output image, the selection timing is set individually for each internal synchronization camera,
The image processing apparatus, wherein the designated frame memory is selected at the selection timing for each internal synchronization camera. When i is an integer of 1 to n,
The selection means includes
Paying attention to the frame memory to store the image data of the latest input image from the i-th internal synchronous camera,
A determination is made as to whether or not the image data of the focused frame memory can be used at a timing when the image data of the input image from the i-th internal synchronization camera is to be output as the image data of the output image;
The image processing apparatus according to claim 1, wherein the designated frame memory for the i-th internal synchronization camera is selected based on the determination result. The output means is stored in each designated frame memory based on predetermined correspondence information that defines a correspondence relationship between each pixel on the input image and each pixel on the output image from each internal synchronization camera. Generating image data of the output image from image data of the input image;
The image processing apparatus according to claim 2, wherein the selection unit performs the determination based on information determined from the correspondence information. The difference in line position between the horizontal line on the input image from the i-th internal synchronization camera and the horizontal line on the output image corresponding to the horizontal line and the maximum amount of the difference are determined by the correspondence information,
The image processing apparatus according to claim 3, wherein the selection unit sets the selection timing in units of horizontal lines and performs the determination based on the maximum amount of the difference. The difference in pixel position between the pixel on the input image from the i-th internal synchronization camera and the pixel on the output image corresponding to the pixel and the maximum amount of the difference are determined by the correspondence information,
The image processing apparatus according to claim 3, wherein the selection unit sets the selection timing in units of pixels and performs the determination based on the maximum amount of difference. The selection means sets a plurality of the selection timings for at least one of the first to n-th internal synchronization cameras, and the command to the internal synchronization camera in which a plurality of selection timings are set. 6. The image processing apparatus according to claim 1, wherein the frame memory is selected at each of the plurality of selection timings. Arithmetic processing means having the selection means and a built-in timer for generating an interrupt request,
7. The arithmetic processing unit operates the timer so that the interrupt request is generated at the selection timing, and selects the designated frame memory when the interrupt request is generated. An image processing apparatus according to any one of the above. First to nth internal synchronous cameras attached to the vehicle (n is an integer of 2 or more);
An image processing apparatus according to any one of claims 1 to 7,
A driving support system, wherein the output image is displayed on the display means by supplying image data of the output image generated by the image processing device to the display means. A vehicle, wherein first to nth internal synchronous cameras are attached (n is an integer of 2 or more), and the image processing apparatus according to any one of claims 1 to 7 is installed.

Images