JP2008167348A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for superimposing both an image, which was subjected to image processing, and an original imaged image without causing a dislocation gap between them. <P>SOLUTION: In an identification ID embedding unit 13, an identification ID for showing time when an image was imaged with a camera 30 is embedded in the image. In a detection processing unit 16, the image, which includes a detecting object, is subjected to image processing. In a detected result superimpose processing portion 24, the image subjected to image processing by using the detection processing unit 16 and the image, which was imaged with the camera 30 at a time concerning the identification ID embedded in the image by the identification ID embedding unit 13 are superimposed. Then, they can be superimposed with one-to-one correspondence at a time, which the identification ID indicates, without using average processing time for superimpose of the image, so both the image subjected to image processing and the original imaged image are superimposed without causing a dislocation gap between them. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus that performs image processing on a captured image including a detection target and superimposes the captured image on the captured image.

Conventionally, it has been proposed to detect a pedestrian in an automobile night vision device. For example, in Patent Document 1, an infrared camera capable of photographing the surroundings of the host vehicle, an image processing unit that determines a detection target that affects the traveling of the host vehicle from an image captured by the infrared camera, and an image processing unit When a detection object is detected by the above, a video image taken by an infrared camera and a highlighted image such as a pedestrian detection frame indicating the detection object are simultaneously superimposed on a display device arranged in front of the driver's seat of the host vehicle. There has been proposed a vehicle periphery display device provided with a display control unit for displaying the information. In such an apparatus, the average processing time of the process of determining the detection target and superimposing the emphasized video indicating the detection target is obtained, and the emphasized video is displayed on the captured image delayed in display to the driver by the average processing time. Is performed.
JP 2004-364112 A

  However, in the conventional technology as described above, if a difference occurs between the actual processing time for determining the detection target and superimposing the emphasized video indicating the detection target and the average processing time, the detection target is indicated. There is a problem that the emphasized video is superimposed on a captured image that is different from the image obtained by actually capturing the detection target, and the image display of the display device is shifted or uncomfortable.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus capable of superimposing both images without causing a gap between the image subjected to image processing and the original captured image. Is to provide.

  The present invention provides an imaging unit that captures a surrounding image, an identification ID embedding unit that embeds an identification ID indicating the time when the image was captured by the imaging unit, and an image in which the identification ID is embedded by the identification ID embedding unit. An image processing apparatus comprising: processing means for performing image processing; and superimposition processing means for superimposing images having identification IDs at the same time before and after the processing means performs image processing. is there.

  According to this configuration, the identification ID embedding unit embeds the identification ID indicating the time when the image was captured by the imaging unit in the image, and the processing unit performs image processing on the image in which the identification ID is embedded by the identification ID embedding unit. And the superimposing processing unit superimposes the images before and after the processing unit performs image processing and having identification IDs related to the same time, so that the superimposition of the images does not depend on the average processing time. Since the superimposition can be performed in a one-to-one correspondence with the time indicated by the identification ID, both can be superimposed without causing a gap between the image subjected to image processing and the original captured image.

  In this case, the imaging unit captures a surrounding image for each frame, and when the processing unit processes the image is longer than the threshold, the superimposition processing unit superimposes the frame at the time when the imaging unit captured the image. It is preferable to further include frame interpolation means for performing frame interpolation between frames at the time to be performed.

  According to this configuration, when the time for which the processing unit processes the image is equal to or greater than the threshold, the frame interpolation unit includes the frame at the time when the imaging unit captures the image and the frame at the time when the superimposition processing unit superimposes the image. Since the frame interpolation is performed during this period, even if the processing time of the processing means is prolonged, it is possible to prevent the image displayed by the frame interpolation from feeling uncomfortable.

  According to the image processing apparatus of the present invention, it is possible to superimpose both images without causing a shift between the image-processed image and the original captured image.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating a configuration of the object detection device according to the present embodiment. The object detection apparatus according to the present embodiment is an image processing apparatus for detecting an object by pattern matching and highlighting it on a driver in an in-vehicle night vision apparatus or the like. As shown in FIG. 1, the object detection device 1 of the present embodiment includes a video distribution / detection processing device 10, a video drawing device 20, a camera (imaging means) 30, an image capture device 31, a spare frame memory 40, and a monitor 50. It has.

  The camera 30 is a monocular camera or a stereo camera using a semiconductor element such as a CCD or a CMOS, and acquires a forward image for each frame at a period of 10 to 100 ms. If importance is attached to detection of pedestrians and the like at night, a near-infrared camera and a far-infrared camera excellent in night vision performance can be applied as the camera 30. The frame image acquired by the camera 30 is sent to the image capture device 31 and converted into a data format that can be processed by the video distribution / detection processing device 10.

  The video distribution / detection processing device 10 is configured using microcomputer hardware and software such as an ECU (Electric Control Unit), and includes a detection execution determination unit (detection execution determination unit) 11, an identification ID provision unit 12, and an identification. An ID embedding unit (identification ID embedding unit) 13, a video distribution unit 14, an identification ID decoding unit 15, a detection processing unit (processing unit) 16, and a detection result transmission unit 17 are included.

  The detection execution determination unit 11 is used to determine whether or not the detection processing unit 16 is performing detection processing when the camera 30 captures an image. For example, when the camera 30 captures a frame image at a period of 33 ms, if the detection processing unit 16 performs the detection processing every three frames (100 ms) because of the processing speed, the detection processing unit 16 performs an inspection. Since the detection processing of other frame images cannot be performed while the processing is being performed, the detection execution determination unit 11 determines whether or not to perform the detection processing of other frame images. Whether or not the detection processing unit 16 is performing the detection process is determined based on the information from.

  The identification ID assigning unit 12 is for generating an identification ID indicating the time when the image is captured by the camera 30 and whether the image is a detection target. If the detection execution determination unit 11 determines that the detection processing unit 16 is performing the detection process, an identification ID indicating that the detection processing unit 16 is not a detection target is generated. If the detection processing unit 16 is not the detection process, the detection execution determination is performed. When the unit 11 determines, an identification ID indicating that it is a detection target is generated.

  The identification ID embedding unit 13 is for embedding the identification ID in the image. FIG. 2 is a diagram illustrating the identification ID embedded in the pedestrian detection image. In the example of FIG. 2, a time stamp 101 indicating the time when the image is captured and a detection target determination value 102 indicating whether or not the image is a detection target are given to the upper right corner of the pedestrian detection image 100.

  The video distribution unit 14 is for distributing the image to the identification ID decoding unit 15 and the spare frame memory 40. The identification ID decoding unit 15 is for reading the identification ID stored in the image in order to store the identification ID in the detection result after the detection process. The detection processing unit 16 surrounds a detection target such as a pedestrian with a pedestrian detection frame or the like on the image determined to be detected by the identification ID decoding unit 15 and displays a predetermined symbol. This is for performing detection processing (enhanced image processing). The detection result transmission unit 17 is for transmitting the image subjected to the detection process to the video drawing device 20.

  The spare frame memory 40 is for acquiring an image in which the identification ID is embedded from the video distribution unit 14 of the video distribution / detection processing apparatus 10 and storing it for a predetermined time.

  Similar to the video distribution / detection processing device 10, the video drawing device 20 is configured using microcomputer hardware and software such as an ECU, and includes a result waiting unit 21, a superimposing image correspondence unit 22, a frame interpolation unit ( A frame interpolation unit) 23, a detection result superimposing unit (superimposing unit) 24, and an image display unit 25. The video distribution / detection processing device 10, the video drawing device 20, and the call frame memory 40 may be integrated.

  The detection result standby unit 21 is for acquiring an image subjected to the detection process from the detection result transmission unit 17 of the video distribution / detection processing device 10. The superimposing image correspondence unit 22 is for associating images provided with an identification ID including the time stamp 101 at the same time from the detection result waiting unit 21 and the spare frame memory 40.

  The frame interpolation unit 23 is configured so that the camera 30 includes the detection target when the detection processing unit 16 of the video distribution / detection processing device 10 processes the image including the detection target for a predetermined threshold time or more. This is for performing frame interpolation between the frame at the time when the captured image is captured and the frame at the time when the detection result superimposing unit 24 superimposes the image.

  The detection result superimposing processing unit 24 uses the image processed by the detection processing unit 16 of the video distribution / detection processing device 10 and the identification ID embedded in the image by the identification ID embedding unit 13 of the video distribution / detection processing device 10. This is to superimpose an image captured by the camera 30 at such a time. The image display unit 25 outputs the image superimposed by the detection result superimposition processing unit 24 to the monitor 50 for display.

  The monitor 50 is for displaying an image on which a pedestrian detection frame or the like from the image display unit 25 is displayed to the user. In the case of a vehicle-mounted device, generally, a liquid crystal monitor or a navigation monitor can be applied to the monitor 50.

  Next, with reference to FIGS. 3-7, operation | movement of the target object detection apparatus 1 of this embodiment is demonstrated. FIG. 3 is a flowchart showing an image distribution thread according to this embodiment. As shown in FIG. 3, the image capture device 31 acquires an image from the camera 30 (S101). The detection execution determination unit 11 of the video distribution / detection processing device 10 determines whether or not the detection processing unit 16 of the video distribution / detection processing device 10 is performing detection processing based on flag monitoring or the presence or absence of an interrupt (S102). .

  When the detection processing unit 16 is performing the detection process, a new image detection process is impossible. Therefore, the identification ID assigning unit 12 of the video distribution / detection processing apparatus 10 acquires the current time, and the detection target determination value. = ID is created as OFF (S103, S104). If the detection processing unit 16 is not in the detection process, a new image detection process is possible, so the identification ID assigning unit 12 acquires the current time and creates an identification ID with the detection target determination value = ON (S103). , S105).

  The identification ID embedding unit 13 of the video distribution / detection processing apparatus 10 embeds the identification ID in a predetermined position of the image as shown in FIG. 2 (S106). The video distribution unit 14 of the video distribution / detection processing apparatus 10 asynchronously transmits (outputs) the created image to the spare frame memory 40 and stores the image in the spare frame memory 40 (S107). Sequentially, the same processing is repeated.

  FIG. 4 is a flowchart showing the detection thread according to the present embodiment. As shown in FIG. 4, the identification ID decoding unit 15 of the video distribution / detection processing device 10 reads the identification ID stored in the image in order to store the identification ID in the detection result after the detection processing (S201). . The detection processing unit 16 of the video distribution / detection processing device 10 performs detection processing such as surrounding a detection target object such as a pedestrian with a pedestrian detection frame or displaying a predetermined symbol, and outputs the detection result data of the target object. The identification ID is stored (S202). The detection result transmission unit 17 of the video distribution / detection processing apparatus 10 transmits (outputs) the detection result to the video drawing apparatus 20 (S203).

  FIG. 5 is a flowchart showing a detection result waiting thread according to the present embodiment. As shown in FIG. 5, the detection result waiting unit 21 of the video drawing device 20 waits for reception of the detection result of the object from the detection result transmission unit 17 of the video distribution / detection processing device 10 (S301).

  If the detection result has not arrived, the detection result waiting unit 21 repeats step S301 and performs a later-described frame interpolation determination thread (S302). On the other hand, when the detection result arrives, the superimposing image correspondence unit 22 of the video drawing device 20 stores the identification ID stored in the detection result of the object and the image accumulated in the spare frame memory 40. Verification with the identified ID is performed (S303).

  The detection result superimposing processing unit 24 of the video drawing device 20 superimposes the object detection result on the image whose identification ID matches (S304). The image display unit 25 of the video drawing device 20 outputs the superimposed or unprocessed image to the monitor 50 (S305). In this case, the information generated in the frame insertion process or the frame deletion process is also reflected in the frame interpolation determination thread described later.

  FIG. 6 is a flowchart showing the frame insertion process of the frame interpolation determination thread according to this embodiment. In this case, the detection thread is waiting for the end of the detection process, and during the detection process, the standby frame memory 40 currently stores an image waiting for display processing, including the image being detected. As shown in FIG. 6, when the arrival of the detection result is delayed in step S302 of the detection result waiting thread, the frame interpolation unit 23 of the video drawing device 20 measures the image output timing at the frame rate of the camera 30 (S401). . If a frame is inserted before the start of the current detection process, it is necessary to delete the frame. Therefore, the frame interpolation unit 23 performs a frame deletion process to be described later (S402).

On the other hand, when frame insertion has not been performed before the start of the current detection process, and when frame insertion has not been performed during one detection process (S402, S403), the frame interpolating unit 23 sets a spare Two consecutive frames of the image already stored in the frame memory 40 (time series, tentatively assumed as F _n−1 and F _n ) are inspected (S404). The frame interpolation unit 23 acquires and refers to the identification ID from the time-series continuous two frames F _n−1 and F _n waiting for video display, and the “detection target determination value” is the time-series continuous two frames F _n−1 , F _n , It determines whether or not it is from "OFF" to "oN" in F _n (S405). In other words, in order to insert a new frame between the frame waiting for display (the detection target determination value is OFF) immediately before the frame currently being detected (the detection target determination value is ON), A process of confirming whether such a combination of frames exists in the spare frame memory 40 is performed.

If the “detection target determination value” is not changed from “OFF” to “ON”, the frame interpolation unit 23 issues a command to the image display unit 25 to designate the frame F _n−1 as a drawing target. The display unit 25 displays the frame F _n−1 on the monitor 50 (S406, S407, S305). That is, the display is performed according to the time series (standby order) without particularly inserting a frame.

On the other hand, when the “detection target determination value” is changed from “OFF” to “ON”, the frame interpolation unit 23 performs a frame interpolation process between the frame F _n−1 and the frame F _n to generate a new one. A frame image NF _n is generated and inserted (S406). Here, N means the number of increase / decrease frames. The frame interpolation unit 23 increments the increase / decrease frame number N (S409), issues a command to the frame NF _n to be drawn, and the image display unit 25 displays the frame NF _n on the monitor 50. (S410, S305).

If frame insertion is performed during one detection process in step S403 described above, the frame interpolation unit 23 determines that the latest image whose “detection target determination value” is ON, that is, the currently detected image, Frame interpolation processing is performed between the frame NF _n and a new frame image NF _{n + 1} is created (S411). The frame interpolation unit 23 increments the number N of increase / decrease frames (S409), issues a command to the frame NF _{n + 1} to be drawn, and the image display unit 25 displays the frame NF _{n + 1} on the monitor 50. (S412, S305).

  FIG. 7 is a flowchart showing the frame deletion process of the frame interpolation determination thread according to this embodiment. In this embodiment, the frame deletion process is performed so that the capacity of the spare frame memory 40 does not become insufficient. However, if the capacity of the spare frame memory 40 is sufficiently large, it is not necessary to perform the frame deletion process. As shown in FIG. 7, when frame insertion is performed before the start of the current detection process in step S402 of the frame insertion process, the frame interpolation unit 23 determines that the latest frame Fdn (“detection target determination value” is ON) The image frame currently being detected) and the previous frame Fdo (detection target image that has already been displayed) are acquired (S501). The frame interpolation unit 23 determines the number M of image frames stored in the spare frame memory 40 between the frame Fdn and the frame Fdo from the time stamp of the identification ID, and counts the number M of frames (S502).

  The frame interpolation unit 23 compares the increase / decrease frame number N with the frame number M between the frame Fdn and the frame Fdo, and determines whether M> N (S503). That is, the frame interpolation unit 23 compares the total number N of insertions with the number M of display waiting frames, and if M is larger, deletes one frame from the display waiting frames. If M> N is not satisfied, the frame interpolation unit 23 cannot delete the frame, and performs step S404 of the frame insertion process (S505, S404).

  On the other hand, if M> N, the frame interpolation unit 23 deletes one frame of the latest frame (the display timing is later) from among the frames between the frame Fdn and the frame Fdo, and M = It is assumed that M−1 and N = N−1 (S505). That is, the number of display waiting frames is reduced by 1, and the inserted history number N is reduced by 1. N is reduced in order to store how many frames are to be deleted in the next detection process.

  The frame interpolation unit 23 issues a command to the drawing unit 25 in order from the next frame of the frame Fdo to the drawing target, and the image display unit 25 displays the image on the monitor 50 in order from the next frame of the frame Fdo (S506, S305).

  In the present embodiment, the identification ID embedding unit 13 embeds an identification ID indicating the time when the image was captured by the camera 30 in the image, and the detection processing unit 16 performs image processing on the image including the detection target. And the detection result superimposing processing unit 24 superimposes the images before and after the detection processing unit 16 performs the image processing and having the identification IDs related to the same time. Therefore, it is possible to superimpose them so as to correspond one-to-one with the time indicated by the identification ID, so that both can be superimposed without causing a gap between the image subjected to image processing and the original captured image. .

  That is, as shown in FIG. 8, conventionally, when the detection processing result is superimposed on the camera video to obtain a delayed display video, the average detection processing time assuming the average detection processing time PT and the captured image stored in the memory is assumed. Only PT was delayed in advance. Therefore, even when displaying the pedestrian detection frame 300 for the pedestrian 200 that is the detection target, if the processing is delayed from the average detection processing time PT, a synchronization shift occurs, and the detection results are superimposed on different images. There was a case. On the other hand, in the present embodiment, as shown in FIG. 9, the image can be superimposed so as to correspond to the time indicated by the identification ID on a one-to-one basis without depending on the average processing time. Superimposition processing can be performed. Therefore, when the apparatus of this embodiment is applied to pedestrian detection, the pedestrian detection frame is not displaced, and the driver can easily grasp the position of the pedestrian on the video. Furthermore, in this embodiment, since the identification ID is directly written, it is not necessary to transmit / receive extra data for synchronization. As a result, the state transition becomes simple and there is no need to increase communication data.

  In this embodiment, when the time for which the detection processing unit 16 processes an image is equal to or greater than the threshold, the frame interpolation unit 23 superimposes the frame at the time when the camera 30 captures the image and the detection processing unit 16 superimposes the image. Since the frame interpolation is performed between the frames at the time to be performed, even if the processing time of the detection processing unit 16 is prolonged, it is possible to prevent the image displayed by the frame interpolation from feeling uncomfortable. Furthermore, in this embodiment, the frames are deleted without a sense of incongruity, and the drawing of the detection results is matched, so that the capacity of the spare frame memory 40 is insufficient and can be prevented from overflowing.

  Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications can be made. For example, in the above-described embodiment, the case where the present invention is applied to pedestrian detection in an on-vehicle night vision device has been mainly described, but the present invention is not limited to this and is also applied to other uses. Is possible.

It is a block diagram which shows the structure of the target object detection apparatus which concerns on this embodiment. It is a figure which shows identification ID embedded in the pedestrian detection image. It is a flowchart which shows the image distribution thread | sled which concerns on this embodiment. It is a flowchart which shows the detection thread | sled which concerns on this embodiment. It is a flowchart which shows the detection result waiting thread which concerns on this embodiment. It is a flowchart which shows the frame insertion process of the frame interpolation judgment thread concerning this embodiment. It is a flowchart which shows the frame deletion process of the frame interpolation judgment thread concerning this embodiment. It is a figure which shows the relationship between the conventional camera image | video and a delay display image | video. It is a figure which shows the relationship between the camera image and delayed display image of this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Object detection apparatus, 10 ... Video distribution / detection processing apparatus, 11 ... Detection execution determination part, 12 ... Identification ID provision part, 13 ... Identification ID embedding part, 14 ... Video distribution part, 15 ... Identification ID decoding part , 16 ... detection processing unit, 17 ... detection result transmission unit, 20 ... video drawing device, 21 ... detection result standby unit, 22 ... superimposition image correspondence unit, 23 ... frame interpolation unit, 24 ... detection result superposition processing unit, DESCRIPTION OF SYMBOLS 25 ... Image display part, 30 ... Camera, 31 ... Image capture device, 40 ... Preliminary frame memory, 50 ... Monitor, 100 ... Pedestrian detection image, 101 ... Type stamp, 102 ... Detection object discriminating value, 200 ... Pedestrian, 300: Pedestrian detection frame.

Claims (2)

An imaging means for capturing a surrounding image;
An identification ID embedding unit that embeds an identification ID indicating the time when the image was captured by the imaging unit in the image;
Processing means for performing image processing on the image in which the identification ID is embedded by the identification ID embedding means;
Superimposition processing means for superimposing images having identification IDs related to the same time before and after the processing means performs image processing;
An image processing apparatus. The imaging means captures a surrounding image for each frame,
When the time for which the processing unit processes the image is equal to or greater than a threshold, frame interpolation is performed between the frame at the time when the imaging unit captures the image and the frame at the time when the superimposition processing unit superimposes the image. The image processing apparatus according to claim 1, further comprising a frame interpolation unit for performing the operation.

Images