JP5344618B2 - MOBILE BODY TRACKING DEVICE, TRACKING METHOD, AND COMPUTER PROGRAM - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving object tracking device, a tracking method and a computer program, for precisely tracking a moving destination of a moving object without needing much calculation, even when sunshine changes, a sunny place and a shadow area are included, contract is reduced, and moving objects partially overlap in a captured image. <P>SOLUTION: A vehicle tracking device 1 extracts a positive sample including a vehicle from the t-th image, extracts a plurality of pixel pairs from the positive sample, extracts a plurality of negative samples from the periphery of the positive sample, extracts each of the plurality of pixel pairs from each negative sample, and obtains the pixel pairs, in which the accuracy of determination between the positive samples and the negative samples is high, as pair features based on the size relation of the luminance values of the pixel pairs. The vehicle tracking device 1 extracts a plurality of image regions as moving destination candidates from the (t+1)-th image, and determines the moving destination candidate, in which the size relation of the pixel pairs extracted from the moving destination candidates is the most similar to the pair features of the positive samples, as the moving destination of the moving object. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a moving body tracking device, a tracking method, and a computer program for tracking a moving destination of a moving body in a plurality of images taken in time series.

  For smooth traffic and accident prevention, etc., a communication system in which infrastructure devices installed on the road side and vehicles (on-vehicle devices) exchange information by road-to-vehicle communication, especially in places that are blind spots from vehicle drivers. Studies are underway on a system that can provide information on existing vehicles or pedestrians from the infrastructure device to the vehicle. In this system, it is desired that not only information such as the number of vehicles passing on the road and the average speed but also the position and movement of each vehicle are accurately detected and provided from the infrastructure device to the vehicle. For this reason, the infrastructure device needs to be equipped with sensors for detecting vehicles or pedestrians on the road, and taking into consideration the performance such as the detection range, product life, and cost, the captured image captured by the camera is used. A method of detecting a vehicle or a pedestrian is a promising method.

  In the method using a camera, in order to accurately measure the moving speed and moving direction of a vehicle or a pedestrian, a technique for accurately tracking the destination of the vehicle or the pedestrian in the image captured by the camera is required. is there. This tracking technology includes sunshine changes due to changes in weather, mixed sun and shade areas in captured images, low contrast in captured images, and partial vehicle or pedestrians due to overlapping of vehicles or pedestrians, etc. There is a problem that the tracking accuracy is lowered due to the influence of hidden hiding, and a tracking method that can solve this problem is demanded. In addition, for practical application and spread of the above system, it is also important to reduce the amount of calculation required for image processing and to reduce the price of the apparatus.

  A block matching method is widely known as a technique for tracking a vehicle, a pedestrian, or the like (hereinafter referred to as a moving body) based on a captured image. In the block matching method, an image region including a moving object to be tracked at a certain time is extracted from a captured image as a template, and the image region most similar to the template is searched for in the captured image at the subsequent time. This is a method of using a moving object as a moving destination. Note that whether or not the image areas are similar is determined by summing up the absolute values of the differences between the luminance values of the pixels included in the template and the luminance values of the pixels included in the image area to be determined, so-called SAD ( Sum of Absolute Difference) is generally used, and it can be determined that the template and the image region are more similar as the SAD value is smaller.

  In addition, a gradient method is widely known as another technique for tracking a moving object based on a captured image. The gradient method is a method of determining feature points of a moving object to be tracked and tracking the feature points with a plurality of images taken in time series. Based on the change in the luminance value of the pixel in the x direction, y direction, and time direction at the position of the feature point, the moving direction and moving amount of the moving body in the image can be calculated.

  In Non-Patent Document 1, a Mean-Shift tracking method is proposed. In the Mean-Shift tracking method, a color histogram is acquired as a feature from an image region that includes a moving object that is copied in an initial image, and an image region that has the best color histogram feature from a subsequent image is a position that has a centroid position that matches. As a result, the image area is set as the movement destination of the moving object. In the Mean-Shift tracking method, since the moving body is tracked based on the characteristics of the luminance distribution, it is possible to track the moving body of various shapes, and the advantage that it is robust against partial overlapping of the moving bodies. There is.

  In Non-Patent Document 2, tracking of a moving object is regarded as an identification problem between the moving object and the background, and a classifier is generated by extracting features from the moving object and the background. A method for detecting the destination of the movement has been proposed. In this tracking method, since tracking can be performed using various features such as the color and shape of the moving object as a clue, it is possible to track the moving object with high accuracy by learning the classifier.

Dorin Comaniciu, Visvanathan Ramesh, Peter Meer, “Real-time tracking of non-rigid objects using mean using the Mean Shift method shift) ", in proc. CVPR 2000, Vol.2, pp.142-149, 2000 Helmut Grabner, Michael Grabner, Horst Bischof, “Real-Time Tracking via On-Line Boosting”, Proc. BMVC, vol .1, p.47-56, 2006

  However, the above-described block matching method has a problem that when the tracking target moving body moves from the shade to the sun, the SAD value becomes very large, and it is easy to mistrack the road surface in the shade. In addition, when a part of the tracking target moving body is hidden behind another moving body, there is a problem that the value of SAD becomes very large and mistracking is likely to occur. The gradient method also has a problem that mistracking is likely to occur when a part of the moving object is hidden, as in the block matching method, and that a large movement of the moving object is difficult to detect.

  Further, the Mean-Shift tracking method described in Non-Patent Document 1 has a problem that it is difficult to specify a precise movement destination when the contrast between the tracking target moving body and its surroundings is low. Further, the Mean-Shift tracking method has a problem that black and white images (monochrome images) have low discrimination due to histograms and are likely to cause erroneous tracking. In addition, the tracking method using the classifier described in Non-Patent Document 2 has a problem that the sample amount and the calculation amount required for learning of the classifier are enormous, and the calculation amount of the identification process using the classifier is large. is there.

  The present invention has been made in view of such circumstances, and the object of the present invention is to change the sunlight due to changes in the weather, etc., the mixture of the sun and shade areas, the low contrast of the captured image, and the moving object. Even when partial overlap occurs, the moving object tracking can accurately track the moving destination of the moving object and can track the moving object without requiring a large amount of calculation. To provide an apparatus, a tracking method, and a computer program.

  The mobile tracking device according to the present invention is a mobile tracking device that tracks a destination of a mobile in the image based on a plurality of images captured in time series. A tracking target sample extracting means for extracting, as a tracking target sample, an image area of a predetermined size including a moving object from the original image, and an image area of a predetermined size around the tracking target sample from the tracking source image. A plurality of surrounding sample extracting means, a tracking target pixel combination extracting means for extracting a plurality of pixel combinations of a plurality of pixels from the tracking target sample extracted by the tracking target sample extracting means, and the tracking target pixel combination extracting means In accordance with the extraction position of the pixel combination extracted from the surrounding sample, a plurality of sets of images are extracted from the surrounding sample extracted by the surrounding sample extracting means. A surrounding pixel combination extracting unit that extracts a combination, a magnitude relationship between pixel values included in the pixel combination extracted by the tracking target pixel combination extracting unit, and a pixel combination included in the pixel combination extracted by the surrounding pixel combination extracting unit The tracking target sample and the surrounding sample can be determined from a determination unit that determines the magnitude relationship of the pixel values and a plurality of pixel combinations extracted by the tracking target pixel combination extraction unit according to a determination result of the determination unit. From the selection means for selecting a plurality of pixel combinations, the image area extraction means for extracting a plurality of image areas from the tracking destination image of the plurality of images, and the tracking target sample of the pixel combination selected by the selection means From the image areas extracted by the image area extraction means, a plurality of sets of A pixel combination extraction unit that extracts each elementary combination; a determination unit that determines a magnitude relationship of pixel values of each pixel included in the pixel combination extracted by the pixel combination extraction unit; and a pixel combination selected by the selection unit And a destination determination unit that determines a destination image area of the moving body based on a magnitude relationship and a magnitude relationship relating to a pixel combination of each image area extracted by the image area extraction unit. .

  The mobile tracking device according to the present invention is characterized in that the tracking target pixel combination extraction unit extracts a pixel combination of a plurality of pixels having a pixel value difference equal to or larger than a threshold value.

  In the mobile tracking device according to the present invention, the tracking target pixel combination extracting unit extracts a plurality of pixel combination groups including a predetermined number of pixel combinations, and the selecting unit includes the tracking target pixel. The pixel combination group having the highest discrimination accuracy of the tracking target sample and the surrounding sample is selected from a plurality of pixel combination groups extracted by the combination extraction unit.

  Further, in the mobile tracking device according to the present invention, the selection unit selects a pixel combination having high discrimination accuracy of the tracking target sample and the surrounding sample from a plurality of pixel combinations extracted by the tracking target pixel combination extraction unit. It is characterized in that a predetermined number is selected from the top.

  Further, in the mobile tracking device according to the present invention, the image area extraction unit extracts image areas of different sizes according to the extraction positions in the tracking destination image, and the image area extraction unit The image processing apparatus further includes an adjusting unit that adjusts the size of the extracted image area to the size of the sample to be tracked, and the pixel combination extracting unit extracts a pixel combination from the image area adjusted by the adjusting unit. It is characterized by.

  In the mobile tracking device according to the present invention, the image area extraction unit extracts image areas of different sizes according to the extraction positions in the tracking destination image, and the pixel combination extraction unit includes: The pixel extraction position relating to the pixel combination is adjusted according to the ratio of the size of the image region extracted by the image region extraction means and the size of the tracking target sample.

  The mobile tracking device according to the present invention further includes calculation means for calculating edge strength of an image, and the tracking source image and the tracking destination image are images in which pixel values are converted into edge strength by the calculation means. It is characterized by being.

  Further, the tracking method according to the present invention is a tracking method for tracking a moving destination of a moving object in the image based on a plurality of images taken in time series. From the tracking source image, a plurality of image regions of a predetermined size around the tracking target sample are extracted as surrounding samples, and a plurality of extracted tracking target samples are extracted from the tracking source image. A plurality of pixel combinations of a plurality of pixels are extracted, and according to the extraction position of the pixel combination extracted from the tracking target sample, a plurality of pixel combinations are extracted from the surrounding samples and extracted from the tracking target sample. Magnitude relationship of pixel values of pixels included in the pixel combination and pixel values of pixels included in the pixel combination extracted from the surrounding samples Each of the magnitude relations is determined, and a plurality of pixel combinations that can distinguish the tracking target sample and the surrounding sample are selected from a plurality of pixel combinations extracted from the tracking target sample according to a determination result of the magnitude relation, A plurality of image regions are extracted from the tracking destination image of the plurality of images, and a plurality of sets are obtained from each image region extracted from the tracking destination image according to the extraction position of the selected pixel combination from the tracking target sample. Each pixel combination is extracted, the magnitude relationship between the pixel values of each pixel included in the extracted pixel combination is determined, and the magnitude relationship relating to the selected pixel combination and the magnitude relationship relating to the pixel combination of each extracted image region are determined. Based on this, an image area to which the moving body is moved is determined.

  The tracking method according to the present invention extracts an image area of a predetermined size as a tracking target sample from the tracking source image, and uses an image area of a predetermined size around the tracking target sample as the surrounding sample from the tracking source image. Extract a plurality of pixel combinations based on a plurality of pixels from the extracted tracking target sample, and extract a plurality of pixel combinations from the surrounding samples according to the extraction position of the pixel combination extracted from the tracking target sample Determining the magnitude relationship between the pixel values of the pixels included in the pixel combination extracted from the sample to be tracked and the magnitude relationship between the pixel values of the pixels included in the pixel combination extracted from the surrounding samples. In response to the tracking target sample, a plurality of pixel combinations extracted from the tracking target sample are used. A plurality of pixel combinations capable of discriminating the pull and the surrounding sample are selected, a plurality of image regions are extracted from the tracking destination image, and the tracking destination is selected according to the extraction position of the selected pixel combination from the tracking target sample. A plurality of pixel combinations are extracted from each image region extracted from the image, the magnitude relationship between the pixel values of each pixel included in the extracted pixel combination is determined, and the magnitude relationship and the extracted pixel combination are extracted. The image area of the movement destination of the sample to be tracked is determined based on the magnitude relationship relating to the pixel combination of each image area.

  The computer program according to the present invention is a computer program for causing a computer to execute a moving body tracking step for tracking a moving destination of a moving body in the image based on a plurality of images taken in time series. A moving body tracking step of extracting, from the tracking source image of the plurality of images, an image area of a predetermined size including the moving body as a tracking target sample; from the tracking source image, surrounding the tracking target sample; A step of extracting a plurality of image areas of a predetermined size as surrounding samples, a step of extracting a plurality of pixel combinations of a plurality of pixels from the extracted tracking target sample, and an extraction position of the pixel combination extracted from the tracking target sample To extract a plurality of pixel combinations from the surrounding samples. And a magnitude relationship between the pixel values of the pixels included in the pixel combination extracted from the tracking target sample and a magnitude relationship between the pixel values of the pixels included in the pixel combination extracted from the surrounding samples, Selecting a plurality of pixel combinations capable of discriminating the tracking target sample and the surrounding sample from a plurality of pixel combinations extracted from the tracking target sample according to the determination result, and tracking of the plurality of images A step of extracting a plurality of image regions from the previous image, and a plurality of sets of pixel combinations from each image region extracted from the tracking destination image according to the extraction position of the selected pixel combination from the tracking target sample, respectively. Extracting the pixel value of each pixel included in the extracted pixel combination A step of determining a small relationship, and a step of determining an image region to which the moving object is moved based on the size relationship related to the selected pixel combination and the size relationship related to the pixel combination of each extracted image region. It is characterized by that.

In the present invention, an image region of a predetermined size including the moving body is extracted as a tracking target sample from a tracking source image among a plurality of images obtained by capturing the moving body in time series. In addition, a plurality of image regions that are shifted by several pixels around the tracking target sample in the tracking source image, for example, up, down, left, and right of the tracking target sample are extracted as surrounding samples. Each surrounding sample may include a part of the moving body.
Next, a plurality of sets of pixel combinations (for example, pixel pairs of two pixels) are extracted from the extracted tracking target sample, and the magnitude relationship between the pixel values (luminance values, etc.) of the pixels included in each pixel combination is extracted. judge. Extraction of the pixel combination can be performed by, for example, selecting a plurality of pixels randomly from the image region of the tracking target sample, but may be performed by other methods. Similarly, in a plurality of extracted surrounding samples, a plurality of pixel combinations are extracted according to the extraction position of the pixel combination extracted in the tracking target sample, and the magnitude relationship between the pixel values of the pixels included in each pixel combination is extracted. get. Extraction of the pixel combination from the surrounding sample can be performed from the same position as the extraction position of the pixel combination extracted from the tracking target sample, for example.
Next, the size relationship between the plurality of pixel combinations extracted from the tracking target sample and the size relationship between the plurality of pixel combinations extracted from the plurality of surrounding samples are compared to determine the tracking target sample and the surrounding sample. A plurality of pixel combinations to be obtained are selected. The selected plurality of pixel combinations can be used as information (feature amount) representing the characteristics of the moving object. By using not only the pixel value itself but also the magnitude relationship between the pixel values of the pixel combination as the feature quantity of the moving object, it is robust against changes in sunlight, mixing of sun and shade, low contrast, and partial overlapping of moving objects. is there. Further, since only the magnitude relationship between pixel values needs to be determined, the amount of calculation is small compared to a method for learning a discriminator.

  A combination of a plurality of pixels selected from the tracking source image is used as a feature amount of the moving object to be tracked. By searching for an image area, the image area obtained as a result of the search can be set as a moving destination of the moving object. Specifically, a plurality of image areas are extracted from the tracking destination image, and similarly, extraction of a plurality of pixel combinations from the extracted plurality of image areas and determination of the magnitude relation of pixel values of pixels included in these pixel combinations are performed. Do each. Compares the magnitude relationship between the pixel values of a plurality of pixel combinations selected as feature quantities of the moving object and the magnitude relationship between the pixel values of a plurality of pixel combinations extracted from a plurality of image areas in the tracking destination image. An image area including the largest number of pixel combinations having the same relationship can be set as a moving destination of the moving object.

  In the present invention, when a pixel combination is extracted from the tracking target sample, a plurality of pixels having a difference equal to or greater than a predetermined threshold are extracted as the pixel combination. Thereby, it is possible to extract a pixel combination having a high possibility of discriminating between the tracking target sample and the surrounding sample, and to improve tracking accuracy of the moving object.

In the present invention, when selecting a plurality of pixel combinations as the feature amount of the moving object, a plurality of pixel combination groups including a predetermined number of pixel combinations are extracted from the tracking target sample. Each pixel combination group is used to discriminate between the tracking target sample and a plurality of surrounding samples, and by selecting the pixel combination group having the highest discrimination accuracy, a plurality of pixel combinations included in this pixel combination group are selected. It can be a feature amount.
In this way, by extracting a plurality of pixel combination groups and selecting the pixel combination group having the highest discrimination accuracy from these, it is possible to accurately discriminate the moving object from the background by the selected pixel combination. Thus, it is possible to more accurately determine the image area that is the moving destination of the moving object from the tracking destination image.

In the present invention, when selecting a plurality of pixel combinations as the feature amount of the moving object, the number is larger than the number (predetermined number) necessary for the subsequent processing (several times to several tens of times the predetermined number). Pixel combinations are extracted from the sample to be tracked. From these many pixel combinations, a predetermined number of pixel combinations having high discrimination accuracy between the tracking target sample and the plurality of surrounding samples are selected from the top.
In this way, by selecting a predetermined number of pixel combinations with high discrimination accuracy from a large number of pixel combinations, it is possible to accurately discriminate between a moving object and a background by the selected pixel combinations, and from the tracking destination image It is possible to more accurately determine the image area that is the destination of the moving object.

For example, when a moving object such as a vehicle or a pedestrian is imaged with a camera installed on the road, the size of the moving object in the captured image changes according to the distance from the camera to the moving object. Therefore, in the present invention, when a plurality of image areas are extracted from the tracking destination image, image areas having different sizes are extracted according to the extraction positions. Thereby, even when the moving body moves in a direction approaching the camera and when the moving body moves in a direction away from the camera, the moving body can be accurately tracked.
However, if the size of the image area to be extracted is different, it is difficult to compare with the feature amount of the tracking target sample (the relationship between the pixel values of the pixel combinations). Adjust to.

  Similarly, in the present invention, when a plurality of image areas are extracted from the tracking destination image, image areas having different sizes are extracted according to the extraction positions. Further, when extracting a pixel combination from image areas having different sizes, the position of the pixel to be extracted as the pixel combination is adjusted according to the ratio between the size of the image area and the size of the tracking target sample. As a result, pixels at the same position as the tracking target sample can be extracted from image regions of different sizes. The pixel value relationship in the pixel combination of each image region and the pixel value in the pixel combination of the tracking target sample The moving object can be tracked with high accuracy by comparing the magnitude relation of the two.

  In the present invention, an image in which pixel values are converted into edge strength (hereinafter referred to as an edge image) is generated using an image filter that calculates the edge strength of the image. As described above, when tracking a moving object based on a pixel combination, an edge image is used as an image such as a tracking source image and a tracking destination image. Depending on the installation environment of the camera, that is, the environment around the moving body, the edge image may be able to capture the characteristics of the moving body. In such a case, tracking accuracy can be improved by tracking the moving object based on the edge image.

  In addition, the tracking process based on the feature amount described above is not only performed when the moving destination of a moving object is tracked from an image captured by a camera or the like, but also in image processing such as a moving image compression process, for example, The present invention can also be applied to the process of searching for the movement destination of the image area, and can increase the accuracy and speed of such image processing.

  In the case of the present invention, the image region having the closest feature amount from the plurality of image regions extracted from the tracking destination image is searched by using the magnitude relationship between the pixel values of the pixel combination extracted from the tracking source image as the feature amount of the moving object. By using this image area as the destination of the moving object, the moving object is characterized by the magnitude relationship of the pixel values, so that the change in sunlight, the mixture of the sun and the shade, the low contrast, the partial overlap of the moving object, etc. Even when this occurs, the moving destination of the moving object can be accurately tracked from the images captured in time series. In addition, since only the magnitude relationship between pixel values needs to be compared, the amount of calculation required for tracking a moving object can be reduced. Therefore, it is possible to provide a low-cost apparatus that can accurately track the moving body.

It is a schematic diagram for demonstrating the process which a vehicle tracking device performs. It is a block diagram which shows the structure of a vehicle tracking apparatus. It is a schematic diagram which shows an example of the image which the camera imaged in time series. It is a schematic diagram for demonstrating the extraction process of a positive sample and a negative sample. It is a schematic diagram for demonstrating the acquisition method of a pair characteristic. It is a flowchart which shows the procedure of the acquisition process of a pair characteristic. It is a flowchart which shows the procedure of the acquisition process of a pair characteristic. It is a schematic diagram for demonstrating the extraction method of a movement destination candidate. It is a flowchart which shows the procedure of the tracking process of a movement destination. 10 is a flowchart illustrating a procedure of pair feature acquisition processing according to Modification 1; 10 is a schematic diagram for explaining a method of extracting a pixel pair from a movement destination candidate according to Modification 2. FIG. It is a block diagram which shows the structure of the vehicle tracking apparatus which concerns on the modification 3.

  Hereinafter, the present invention will be specifically described with reference to the drawings showing embodiments thereof. In the present embodiment, the configuration of a vehicle tracking device that tracks a vehicle on a road as a moving body will be described. However, the tracking target moving body is not limited to a vehicle. For example, a pedestrian, a two-wheeled vehicle, a ship, It may be other various objects such as an airplane or an animal, as long as its position changes with time in at least an image captured by a camera.

<Vehicle tracking device>
FIG. 1 is a schematic diagram for explaining processing performed by the vehicle tracking device. FIG. 2 is a block diagram showing the configuration of the vehicle tracking device. In the figure, a vehicle tracking device 1 is a device that tracks a vehicle traveling on a road, and performs processing for tracking the destination of the vehicle based on images (that is, moving images) captured in time series by the camera 3. .

  The camera 3 captures a predetermined area near the road in a state in which imaging conditions such as a predetermined height, depression angle, and rotation angle are set so as to capture a predetermined area including the road (an imaging area indicated by a one-dot chain line in FIG. 1). It is installed at the point. The camera 3 continuously captures images on the road (in time series) and gives the captured images to the vehicle tracking device 1. The vehicle tracking device 1 detects a vehicle from one image captured by the camera 3, tracks the vehicle by searching for the movement destination of the vehicle in a later image (tracking destination image), and wirelessly communicates with other vehicles. To provide information on tracking vehicles.

  For example, in FIG. 1, a camera 3 is installed to image one road at the intersection of two roads, and the vehicle tracking device 1 tracks the vehicle C1 existing in the imaging area of the camera 3. The vehicle tracking device 1 determines the traveling direction and traveling speed of the vehicle C1 from the tracking result of the vehicle C1, and notifies these information to the vehicles C2 and C3 existing on the other road by wireless communication. As a result, at an intersection with poor visibility, the driver of the vehicles C2 and C3 can recognize that another vehicle C1 is traveling on the intersecting road, and can perform safe driving such as slow driving or temporary stop. .

  The vehicle tracking device 1 includes a control unit 11, an image input unit 12, an A / D conversion unit 13, an image memory 14, a communication unit 15, a storage unit 16, a ROM (Read Only Memory) 17, and the like. The vehicle tracking device 1 is installed near the road together with the camera 3 and is connected to the camera 3 via a cable for inputting and outputting video signals. In FIG. 2, the vehicle tracking device 1 and the camera 3 are separate devices, but the present invention is not limited to this, and a configuration in which both are integrated may be used.

  Specifically, the control unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) and a storage element such as a ROM (Read Only Memory). By executing a computer program stored in advance, the operation of each part of the vehicle tracking device 1 can be controlled and various processes can be performed. In particular, in the present embodiment, the control unit 11 executes a vehicle tracking program 40 stored in advance in the ROM 17 so as to track the movement destination of the vehicle based on the images captured by the camera 3 in time series. I do.

  The camera 3 is a so-called video camera that captures a moving image. For example, the camera 3 captures a moving image by capturing an image of 240 × 320 pixels per frame by 30 frames per second, and captures the captured image every frame. Output to the vehicle tracking device 1. The camera 3 outputs an image obtained by imaging to the vehicle tracking device 1 as an analog image signal (however, the camera 3 may be configured to output a digital image signal). This is input to the image input unit 12 of the vehicle tracking device.

  FIG. 3 is a schematic diagram illustrating an example of an image captured by the camera 3 in time series. FIG. 3A illustrates an image captured at a previous time, and FIG. 3B illustrates an image captured at a later time. The image is shown. By installing the camera 3 in the vicinity of the road under predetermined conditions (conditions such as an installation position and an installation direction), the camera 3 can capture the road and its surroundings as shown in FIG. Further, the camera 3 captures 30 frames per second, so that a vehicle traveling on the road can be captured over a plurality of images. In the example shown in FIG. The vehicle is imaged over two images. In the present embodiment, the size of the captured image is 240 × 320 pixels, but is not limited to this.

  The image input unit 12 of the vehicle tracking device 1 has a connection terminal for connecting an image signal input / output cable, and acquires an image signal input from the camera 3 and outputs it to the A / D conversion unit 13. To do. The A / D conversion unit 13 converts the analog image signal input from the image input unit 12 into a digital signal, and stores the converted digital signal in the image memory 14 as image data. A captured image input from the camera 3 via the image input unit 12 is stored in the image memory 14 as image data in units of one frame in synchronization with the frame rate (image capturing interval) of the camera 3. The image memory 14 is configured by a memory element such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), and has a storage capacity capable of storing a large number of image data.

  The communication unit 15 includes a narrow-range communication function, a mid-range communication function such as a wireless local area network (LAN) such as a UHF (Ultra High Frequency) band or a VHF (Very High Frequency) band, a mobile phone, and a PHS (Personal Handyphone System). ), Wide-area communication functions such as multiplexed FM (Frequency Modulation) broadcasting or Internet communication. The communication unit 15 receives predetermined vehicle information (for example, vehicle position information, vehicle speed information, vehicle type information, operation status of a wiper, and operation of a vehicle lamp such as a headlight) from vehicles existing in and near the imaging region of the camera 3. Information indicating the situation). In addition, the communication unit 15 transmits information to vehicles in the vicinity in order to provide information related to the vehicle detected from the captured image of the camera 3. Further, the communication unit 15 can transmit / receive predetermined information to / from a server device installed in a traffic control center or the like and another vehicle tracking device.

  The storage unit 16 includes a memory element such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), or a magnetic storage device such as a hard disk. The storage unit 16 receives data received by the communication unit 15 and a vehicle described later. Various data and the like generated in the calculation process in which the control unit 11 performs the tracking process are stored.

  The ROM 17 is configured by a nonvolatile memory element such as an EEPROM (Electrically Erasable Programmable ROM) or a flash memory, and stores various computer programs necessary for the operation of the vehicle tracking device 1 including the vehicle tracking program 40 in advance. The vehicle tracking program 40 causes the control unit 11 to function as a vehicle detection means 41, a positive sample extraction means 42, a negative sample extraction means 43, a pair feature acquisition means 44, a movement destination candidate extraction means 45, a movement destination determination means 46, and the like. The code is included. The codes of these means can be realized as subroutines, functions, subprograms, or the like of the vehicle tracking program 40.

  The vehicle detection means 41 by the vehicle tracking program 40 performs processing for detecting a vehicle existing in this image from one image captured by the camera 3. As a vehicle detection method by the vehicle detection means 41, for example, a vehicle classifier is generated by a SVM (Support Vector Machine) method based on feature quantities acquired in advance from a large number of vehicle samples, and this classifier is used. Vehicles can be detected. However, the vehicle detection method is not limited to this, and various existing techniques can be used.

  In addition, the vehicle tracking device 1 and the vehicle tracking program 40 perform processing for tracking a vehicle detected from an image captured by the camera 3, but the vehicle detection processing is not necessarily performed by itself. Or the structure which acquires the process result of the vehicle detection which another program performed, and performs the tracking process of a vehicle may be sufficient (that is, the structure which the vehicle tracking device 1 is not provided with the vehicle detection means 41 may be sufficient). . For this reason, in this Embodiment, the description about the detail of a vehicle detection process is abbreviate | omitted.

  The vehicle tracking device 1 and the vehicle tracking program 40 according to the present embodiment acquire a feature quantity “pair feature” newly found by the inventor of the present invention for the vehicle detected from the captured image of the camera 3, and this pair A process for tracking the destination of the vehicle is performed by specifying the imaging position of the vehicle from the subsequent images (tracking destination images) in time series based on the characteristics. Details of the vehicle tracking method based on the pair feature will be described below.

<Vehicle tracking method based on pair characteristics>
In the vehicle tracking method of the present embodiment, an image region including a vehicle to be tracked is extracted from the t-th image V _t (tracking source image) captured by the camera 3 in time series, and is most similar to this image region. The vehicle is tracked by searching the image area from the (t + 1) -th image V _{t + 1} (tracking destination image). Note that as the information such as the position and size of the vehicle to be tracked in the t-th image V _t, the vehicle detection result by the vehicle detection means 41 may be used as described above, or the t−1-th image V _The result of vehicle tracking based on _t-1 may be used.

In the vehicle tracking process, first, the extracts one image region to be tracked including vehicle from t th image V _t as a positive sample (tracked sample) I _t, a plurality of surrounding positive sample I _t (F negative samples (ambient samples) J _t ¹ of the individual), ..., processing to extract J _t ^F. Incidentally, the positive sample I _t and negative samples J _t ^1, ..., extraction of J _t ^F is the processing performed respectively by the positive sample extraction unit 42 and the negative sampling means 43 of the vehicle tracking program 40.

Figure 4 is a positive sample I _t and negative samples J _t ^1, ..., it is a schematic diagram for explaining the process of extracting J _t ^F. In 4 (a) shows, the image shown as an example in Figure 3 of the above (a), Aru the image area to be extracted as a positive sample I _t indicated by a rectangular frame. The positive sample extraction means 42 acquires the processing result of the tracking process based on the (t−1) -th image V _t-1 or the processing result of the vehicle detection process by the vehicle detection means 41, and according to these processing results, the positive sample I The extraction position and extraction size of _t are determined. The size of the vehicle in the image captured by the camera 3 in order to vary according to the position of the vehicle may vary depending on the sizes extraction position of the positive sample I _t.

Then, F-number of negative samples J _t ¹ from the surrounding positive sample I _t, ..., J _t ^F negative sample extraction means 43 extracts. Each negative sample J _t ¹ negative sample extracting unit 43 extracts, ..., J _t ^F is a positive sample is an image area of the same size as I _t, in the captured image from the positive sample I _t vertically and horizontally to the predetermined pixel A plurality of negative samples J _t ¹ ,..., J _t ^F can be extracted from the range. For example, when the negative samples J _t ¹ ,..., J _t ^F are extracted by shifting one pixel at a time from the range of 5 pixels vertically (left and right) from the positive sample I _t (11 pixels × 11 pixels), F = 11 × 11-1 = 120 negative samples J _t ¹ ,..., J _t ^F can be extracted from around the positive samples I _t .

Positive samples I _t and negative samples J _t ^1, ..., after extracting J _t ^F, these positive samples I _t and negative samples J _t ^1, ..., as a feature quantity that can distinguish between J _t ^F, positive samples pair characteristic acquisition unit 44 from I _t is to acquire a pair of features. FIG. 5 is a schematic diagram for explaining a pair feature acquisition method.

To obtain the pair feature, first extracts a predetermined number of pixel pair from the positive sample I _t (pixel combinations). Pixel pair, among the positive samples I _t of two pixels extracted at random, the difference between the luminance value (pixel value) is more than the threshold value T _p. As an example, in FIG. 5 (a) shows the five pixel pair extracted from the positive sample I _t of 8 pixels × 8 pixels, two pixels connected by solid line arrows is a pixel pair. In FIG. 5A, pixels having a large luminance value in the pixel pair are indicated by white squares, and pixels having a low luminance value are indicated by hatched squares. Each pixel pair has a label indicating that the luminance value of the first pixel p (the pixel on the start point side of the arrow in the drawing) is larger than the luminance value of the second pixel q (the pixel on the end point side of the arrow in the drawing). “1” is attached, and a label “−1” is attached to a pixel whose luminance value is smaller than that of the second pixel q.

Details are as follows. Positive samples I _t and negative samples J _t ^1, ..., if the size of the J _t ^F is W pixels × H pixels, positive samples I _t and negative samples J _t ^1, ..., of each pixel included in J _t ^F The coordinates are expressed as grid points of the grid Γ shown in the following equation (1-1). Thereby, an image area of W pixels × H pixels can be regarded as a luminance function on the grid Γ.

Here, a positive sample I _t regarded as grid gamma, any pair extracted from the lattice point on the grid gamma (p, q) with respect to, ppf (p → q; T p) of the following values (1 -2) It defines with a formula. In yet (1-2) equation, I (p) and I (q) is the luminance value of the pixel p and the pixel q in the positive sample _{I t, T p (> 0} ) is a threshold of the luminance difference. (1-2) equation _{ppf (p → q; T p} ) represents the difference between the pixel pair of the positive sample I _t (p, q).

Extraction of a pixel pair in a positive sample I _t is, ppf in (1-2) below; may be extracted pair (p, q) so that _{(p → q T p) ≠} Φ. Since the pixel pairs that can be extracted from one positive sample I _t exist an infinite number limits the number of pixel pair to be extracted into N (N sets). Here, a set RP _s of pixel pairs (p, q) randomly extracted according to the policy s is defined by the following equation (1-3).

After extracting the N pixel pairs from positive samples I _t as described above, in the same manner, the negative sample J _t ^1, ..., respectively, from J _t ^F is extracted N pixels pairs. Positive samples I _t and negative samples ^{_{J t 1, ..., J t}} F is the same size, negative samples J _t ^1, ..., extracted pixel pair from J _t ^F, the pixel pair extracted from the positive sample I _t This is done by extracting a pixel pair at the same position relative to (see FIG. 5B). Here, for the pixel pair extracted from the negative samples J _t ¹ ,..., J _t ^F , a label b (p → q) is defined by the following equation (1-4). In yet (1-4) equation, J (p) and J (q) is negative samples J _t ¹ one, ..., a luminance value of the pixel p and the pixel q in J _t ^F. The label b (p → q) represents the magnitude relationship of the pixel pair (p, q) of the negative samples J _t ¹ ,..., J _t ^F.

In the above (1-2) equation, the pixel pair extracted from the positive sample I _t (p, q), ppf when the luminance value of the pixel p is greater than the threshold value T _p than the luminance value of the pixel q (p → q; T _p ) = 1, and ppf (p → q; T _p ) = − 1 when the luminance value of the pixel _p is smaller than the luminance value of the pixel q by a threshold value T _p or more. Similarly, in the above formula (1-4), the negative sample ^{_{J t 1, ..., J t}} F extracted pixel pair from (p, q) for, when the luminance value of the pixel p is greater than the luminance value of the pixel q b (p → q) = 1, and b (p → q) = − 1 when the luminance value of the pixel p is smaller than the luminance value of the pixel q. ppf (p → q; T _p ) and b (p → q) are values representing the magnitude relationship between the luminance values of the pixel p and the pixel q, and are a positive sample I _t and a negative sample J _t ¹ _,. ppf for ^F in the corresponding pixel pair (p, q); by comparing the value of _(p → q T p) and b (p → q), positive sample I _t and negative samples J _t ^1, ..., J _t It can be used as an index for judging whether or not ^F is similar. Therefore, for each pixel pair (p, q) included in RP _s , an individual similarity r (p, q, J) is defined by the following equation (1-5).

According to the (1-5) equation, ppf (p → q; T p) = b If (p → q) is satisfied, i.e., the luminance value of a pixel pair extracted from the positive sample I _t (p, q) , And the magnitude relationship of the luminance values of the pixel pairs (p, q) of the corresponding negative samples J _t ¹ ,..., J _t ^F match, the individual similarity of the pixel pair (p, q) The value of r (p, q, J) can be “1”. The _{ppf (p → q; T p} ) ≠ b For (p → q), i.e. the magnitude relationship between the luminance values of the positive sample pixel pair extracted from I _t (p, q), the corresponding negative samples J _t ¹ ,..., If the magnitude relationship of the luminance values of the pixel pair (p, q) of J _t ^F does not match, the value of the individual similarity r (p, q, J) of the pixel pair (p, q) is set to “ 0 ".

Therefore, negative samples J _t ¹ with a set RP _s positive sample I _t and one of the extracted pixel pair (p, q), ..., when determining the J _t ^F, the positive sample I _t and one negative The similarity c _s (I, J, RP _s ) of the samples J _t ¹ ,..., J _t ^F is defined by the following equation (1-6).

Therefore, (1-6) equation similarity c _s (I, J, RP _s) so that the value of is smaller, by selecting the pixel pair (p, q), positive sample I _t and negative samples J _A pixel pair (p, q) that can discriminate _t ¹ ,..., J _t ^F with higher accuracy can be acquired. A plurality of pixel pairs (p, q) and their magnitude relationships (values of ppf (p → q; T _p )) thus obtained can be used as “pair features”.

Next, a positive sample I _t and negative samples J _t ^1, ..., illustrating a method of selecting a characteristic pixel pair that can determine J _t ^F (pair feature). Pair feature positive sample I _t and negative samples J _t ^1, ..., is selected to maximize the discrimination criterion between J _t ^F. This discrimination criterion is determined from the following three conditions.
The value of the pair feature is binary (+ v or −v). The number of positive samples is one. The number of negative samples (F) is large. From these conditions, positive samples p and negative samples n ^i. Is defined by the following equation (2-1).

The number of negative samples is sufficiently large, assuming F >> 1 is satisfied, the average value mu _T of the whole, substantially equal to the average value mu _n of the negative samples. Therefore, if the number of negative samples with n ⁱ = −v is m, the following equation (2-2) is established.

Further, the overall variance σ _T ² and the interclass variance σ _B ^{2 of} the positive sample and the negative sample are expressed by the following equations (2-3) and (2-4), respectively.

Therefore, the above discrimination criterion is represented by the ratio between the inter-class variance σ _B ² of the positive sample and the negative sample and the overall variance σ _T ² and is defined by the following equation (2-5).

  This expression (2-5) indicates that minimization of negative sample variance corresponds to maximization of the discrimination criterion. Therefore, when the variance of the negative sample is replaced using the above equation (2-2), the following equation (2-6) is obtained.

In this equation (2-6), the negative sample variance has a minimum value of 0 when m = 0 or F = m. However, since m = 0 means that all negative samples have the same value as the positive sample, it is impossible to discriminate between positive samples and negative samples. Therefore, when m = F, the discrimination criterion is maximized to 1, and the similarity between the positive sample and the negative sample is minimized to 0. Further, minimizing the similarity c _s (I, J, RP _s ) shown in the above equation (1-6) is equivalent to maximizing the above discrimination criterion, and further, the following (3 It is equivalent to minimizing the total value C _min of the similarity expressed by the formula (1).

In the above equation (3), C _min , c _min and RP _min are selected based on a policy of minimizing the individual similarity of each pixel pair.

Here, three methods for selecting pixel pairs are conceivable.
1) A method of collecting pixel features by randomly generating pixel pairs until the similarity c _s becomes smaller than a certain threshold.
2) A method of randomly generating more than the required number of pixel pairs and selecting the required number of pixel pairs from those having a low similarity c _s .
3) A method in which a predetermined number of pixel pair sets are randomly generated and a set having the smallest similarity C _s is selected.
Although the third method is used in this embodiment, the present invention is not limited to this method, and the first or second method may be used.

6 and 7 are flowcharts showing the procedure of the pair feature acquisition process, which is a process performed by the control unit 11 of the vehicle tracking device 1 executing the vehicle tracking program 40. First, the control unit 11 of the vehicle tracking device 1 acquires the t-th image V _t (tracking source image) captured by the camera 3 and uses the result of the vehicle detection process or the result of the t−1-th vehicle tracking process. based identifies the position and size or the like of the tracked vehicle, and extracts an image region including a target object as a positive sample I _t (step S1). Then the control unit 11, from the extracted positive samples I _t, the difference between the brightness value N sets extracted threshold value T _p above two pixels (pixel pair) at random (step S2, Figure 5 (a)).

Next, the control unit 11, from the periphery of the positive samples I _t at t-th image V _t, the negative sample J _t was extracted one (see step S3, FIG. 4), the extracted negative samples J _t, step S2 at extracts the position of a pixel pair corresponding to pixel pair extracted from the positive sample I _t (see step S4, Figure 5 (b)). Next, the control unit 11 calculates the individual similarity r (p, q, J) for each pixel pair based on the above equation (1-5), and adds the individual similarity of the N pixel pairs. Thus, the similarity c _s (I, J, RP _s ) between the positive sample I _t and the negative sample J _t determined by the N pixel pairs is calculated based on the above equation (1-6) (step S5). ).

Note that the processing in steps S3 to S6 is processing that is repeatedly performed for each of ^F negative samples J _t ¹ ,..., J _t ^F extracted from the image V _t . After calculating the similarity c _s (I, J, RP _s ) of step S5 for each negative sample J _t , the control unit 11 calculates the total similarity of the calculated negative samples J _t (step S6). ). Thereafter, the control unit 11 determines whether or not the processing of the similarity calculation and the total calculation thereof has been completed for all of the ^F negative samples J _t ¹ ,..., J _t ^F (step S7). If the processing has not been completed for all the negative samples J _t ¹ ,..., J _t ^F (S7: NO), the processing returns to step S3 and the above-described processing of steps S3 to S6 is repeated.

The control unit 11, an extraction from positive samples I _t of the N pixel pair by performing a predetermined number of times repeatedly, and calculates the sum of the similarity each time. Therefore, when the control unit 11 finishes the processing for all the negative samples J _t ¹ ,..., J _t ^F (S7: YES), the total similarity calculated in step S6 is the similarity calculated previously. It is determined whether it is the minimum value compared with the total (step S8). When the sum of the similarities is the minimum value (S8: YES), the control unit 11 extracts the N pixel pairs extracted in step S2 (that is, N pixel pairs used for calculating the similarity). Is retained (step S9), the previously retained pixel pair is discarded, and the process proceeds to step S10. If the total similarity is not the minimum value (S8: NO), the control unit 11 advances the processing to step S10 without holding the N pixel pairs extracted in step S2.

Next, the control unit 11 determines whether or not the processes in steps S2 to S9 described above have been repeated a predetermined number of times (step S10). If the process has not been repeated a predetermined number of times (S10: NO), the process proceeds to step S2. And the processes in steps S2 to S9 are repeated. When the processes of steps S2 to S9 are repeated a predetermined number of times (S10: YES), the control unit 11 ends the pair feature acquisition process. Incidentally, in the iterative process of steps S2 to S9, eventually the N pixel pair held at step S9 becomes a pair, wherein the positive sample I _t, used in subsequent processing.

After acquiring the pair feature of the vehicle to be tracked from the t-th image V _t captured by the camera 3, the vehicle tracking device 1 determines the destination of the tracking target in the t + 1-th image V _{t + 1} captured by the camera 3. Explore. In the destination search process, first, the vehicle tracking device 1 extracts a number of image areas that are destination candidates from the t + 1-th image V _{t + 1} . This process is a process performed by the movement destination candidate extraction means 45 of the vehicle tracking program 40.

FIG. 8 is a schematic diagram for explaining a method of extracting a destination candidate. The vehicle tracking device 1 sequentially extracts rectangular image areas as movement destination candidates from the t + 1-th image V _{t + 1} captured by the camera 3 while moving one pixel or several pixels in the horizontal and vertical directions ( (See FIG. 8 (a)). The size of the image area to be extracted at this time is determined according to the extraction position. In the example shown in the figure, an image area of a small size is extracted on the upper side of the image V _{t + 1} , and an image area of a large size is extracted on the lower side. Since the size change of the image area to be extracted is determined corresponding to the perspective in the image captured by the camera 3, it is appropriate for each camera 3 depending on the installation position and imaging direction of the camera 3 on the road. Size change can be set.

Further, in the illustrated example, the range for extracting the image area of the destination candidate is limited to only on the road in the image V _{t + 1} . In this way, by excluding the range where the vehicle is unlikely to move and limiting the extraction range of destination candidates, the amount of calculation required for the vehicle tracking process can be reduced. The range of the road in the image can be appropriately set in advance according to the installation position of the camera 3, the imaging direction, and the like.

Further, in the example shown in the figure, the extraction range of destination candidates is set on all roads in the image V _{t + 1} , but the present invention is not limited to this. For example, when it is known in advance that the vehicle moves from the upper side to the lower side of the image V _{t + 1} (the vehicle approaches the camera 3), the extraction range of the movement destination candidate is corrected in the t-th image V _t . it can be limited below the extracted location samples I _t. The moving direction of the vehicle can be determined based on, for example, the moving direction of the vehicle stipulated by the law of the road imaged by the camera 3. For example, the moving direction and moving speed of the vehicle from the result of previous vehicle tracking. Etc. can be predicted and determined.

Vehicle tracking apparatus 1, among the plurality of destination candidates extracted from t + 1 th image V _{t + 1,} the search based on the destination candidate that is most similar to a positive sample I _t of t th image V _t paired feature By doing so, the destination of the vehicle is searched. However, the size of the destination candidate extracting as described above are varied, since the size of the positive samples I _t and the size of the destination candidate is difficult of a determination different from the similar vehicle tracking device 1, t + 1 th the size of the destination candidates extracted from the image V _{t + 1} performs a process of adjusting to the same size as the positive samples I _t (see Figure 8 (b)). Vehicle tracking apparatus 1, when the size of the destination candidate is larger than the size of the positive samples I _t, performs image processing to reduce the moving destination candidate, and if the size of the destination candidate is less than a positive sample I _t, movement Image processing for enlarging the previous candidate may be performed.

After sizing the destination candidate, vehicle tracking apparatus 1, from the same position as the plurality of pixel pair extracted as a pair, wherein the positive sample I _t, extracts a pixel pair in a destination candidate. Positive sample I _t pair feature (pixel pair), using the pixel pair extracted from the destination candidate, and calculates a similarity between the destination candidate by the above (1-6) below and a positive sample I _t (That is, the negative sample may be replaced with the movement destination candidate in the expression (1-6)). Vehicle tracking device 1, for a plurality of destination candidates extracted from t + 1 th image V _{t + 1,} respectively calculates the degree of similarity between the positive sample I _t, the most similarity is larger destination candidate the tracked vehicle Determine the final destination of These processes are processes performed by the movement destination determination means 46 of the vehicle tracking program 40.

FIG. 9 is a flowchart showing the procedure of the destination tracking process, which is a process performed by the control unit 11 of the vehicle tracking device 1 executing the vehicle tracking program 40. First, the control unit 11 of the vehicle tracking device 1 acquires the t + 1-th image V _{t + 1} captured by the camera 3, and extracts one destination candidate from the image V _{t + 1} (step S21). see FIG. 8 (a)), extracts the size of the destination candidates is adjusted to the same size as the t-th positive samples I _t extracted from the image V _t of the reference (step S22, Figure 8 (b)).

Then the control unit 11, from the destination candidates adjust the size, so that the the extracted pixel pair as many and the same position as a pair, wherein the positive sample I _t, extracting a plurality of pixel pair (step S23), the positive based on the pixel pair of the destination candidate pixel pair of sample I _t, the similarity is calculated positive sample I _t and the destination candidate (step S24).

The control unit 11 repeatedly performs the extraction of a movement destination candidate from the t + 1-th image V _{t + 1} , the calculation of the similarity, and the like a plurality of times. Therefore, the control unit 11 determines whether or not the similarity calculated in step S24 is the maximum value compared with the similarity calculated previously (step S25). When the degree of similarity is the maximum (S25: YES), the control unit 11 holds information related to the movement destination candidate extracted in step S21 (for example, information such as the extraction position and extraction size of the movement destination candidate) (step S26). ) Discards previous information and proceeds to step S27. When the similarity is not the maximum (S25: NO), the control unit 11 discards the information related to the movement destination candidate without holding it, and advances the process to step S27.

Next, the control unit 11 determines whether or not the above-described steps S21 to S26 have been completed for all destination candidates to be extracted from the t + 1-th image Vt _{+ 1} (step S27). If the process has not been completed for the destination candidate (S27: NO), the process returns to step S21, and the above process is repeated. When the process is completed for all the movement destination candidates (S27: YES), the control unit 11 ends the movement destination tracking process.

Incidentally, in the repetition of the processing of step S21 to S27, the destination eventually according to the information held in step S26 is, the destination of the positive samples I _t of t th image V _t, i.e. the vehicle destination and It is determined. Further, the movement destination determined in the t + 1-th image V _{t + 1} can be used as the positive sample I _{t + 1} when the movement destination tracking process in the _{t + second} image V _{t + 2} is performed.

Here, when the vehicle tracking process is sequentially performed with a plurality of continuous images such as a t-th image V _t , a t + 1-th image V _{t + 1} , a t + second image V _{t + 2} ... The method can be used.
1) t th acquires the pair characteristics from positive samples I _t of the image V _t searches the destination at t + 1 th image V _{t + 1,} the destination t + 1 th image V _{t + 1} positive A method of repeatedly acquiring a pair feature and searching for a destination, such as obtaining a pair feature as a sample I _{t + 1} and searching for a destination in a _{t + second} image V _{t + 2} .
2) A pair feature is acquired from the positive sample I _{t of the} t-th image V _t , the destination is searched for in the t + 1-th image V _{t + 1, and the} positive sample I is detected in the t + 1-th image V _{t + 1} . without obtaining pairs features from _{t + 1,} it searches for a destination at t + 2 th image V _{t + 2} using the pair characteristics obtained from the positive sample I _t of t th image V _t ... that As described above, a method of searching for a destination in subsequent images using the pair feature acquired from the first image.

The vehicle tracking device 1 may perform vehicle tracking with a plurality of continuous images using any of the above-described methods. Which method is used depends on the tracking accuracy, processing speed, and device required for the vehicle tracking device 1. What is necessary is just to select suitably according to factors, such as cost. Comparing the above two methods, the first method is characterized by high tracking accuracy but low processing speed, and the second method has low tracking accuracy but high processing speed. In the second method, the tracking accuracy can be improved by correcting the pair feature acquired from the first image according to the tracking result of the movement destination. Fixed pair feature, for example extracted from t-th from the image V _t and the pixel pairs are extracted as a pair, wherein the positive sample I _t, t + 1 th image region which is to move the image V _{t + 1} pairs This can be done by comparing the feature and replacing a pixel pair that does not match in magnitude with a pixel pair that matches in magnitude.

Or more aspects of the vehicle tracking apparatus 1 and the vehicle tracking program 40 extracts a positive sample I _t containing vehicle from t th image V _t (tracking original image), a plurality of pixel pair from the positive sample I _t extracts the (pixel combinations), the negative sample J _t ¹ from the periphery a plurality of positive samples I _t, ..., extracts J _t ^F, the negative sample J _t ^1, ..., a plurality of pixels from J _t ^F Each pair is extracted, and the discrimination accuracy between the positive sample I _t and the negative samples J _t ¹ ,..., J _t ^F is high based on the magnitude relationship of the luminance value (pixel value) of the pixel pair (when discrimination is performed) A pixel pair having a low similarity is acquired as a pair feature, and the vehicle is tracked using the pair feature. This pair feature is a feature amount that focuses on the magnitude relationship between the luminance values of pixel pairs. For example, when the brightness of the image changes due to changes in sunlight, or when the vehicle moves from the sun to the sun, etc. Even if the brightness value of the vehicle inside changes, the magnitude relationship between the brightness values does not change, so that the vehicle can be tracked with high accuracy. Further, even when the image captured by the camera 3 is low in contrast, the vehicle can be tracked with high accuracy. Therefore, even if the image obtained by the camera 3 is not a color image but a monochrome image, Tracking can be done. Further, since a plurality of extracting the pixel pairs for one positive sample I _t, even in a case where a portion of the tracked vehicle is hidden another vehicle, pixel pair extracted from a portion which is not hidden another vehicle The vehicle can be tracked based on the information. In the vehicle tracking process using the pair feature, it is only necessary to compare the magnitude relationship of the luminance values of the pixel pairs to determine whether the magnitude relationship matches or does not match. Since the calculation amount required for processing is small, the vehicle tracking device 1 is not required to have a high processing capability, which can contribute to the cost reduction of the vehicle tracking device 1.

Further, the extraction of the N pixel pair from the positive sample I _t (pixel combination group) performed a plurality of times, a positive sample I _t and negative samples J _t ^1, ..., the highest accuracy of determining the J _t ^F (similar By selecting a set of N pixel pairs (lowest degree) as a pair feature, a pair feature having a higher degree of discrimination between the vehicle and the background can be acquired, and the tracking accuracy of the vehicle is improved. be able to.

Further, when extracting a plurality of pixel pair from the positive sample I _t, by extracting two pixels the difference is equal to or more than the threshold value T _p of the brightness value, as a candidate pixel pair is used as a pair, wherein the positive sample I _t And negative samples J _t ¹ ,..., J _t ^F can be extracted in advance, and the tracking accuracy of the vehicle can be improved.

Further, when extracting an image area as a movement destination candidate from the t + 1-th image V _{t + 1} (tracking destination image), an image area having a different size is extracted according to the extraction position in the image, and the extracted image area is by adjusting the same size as the positive samples I _t, it is possible to extract destination candidates suitable for the size of the moving object in the captured image which varies depending on the installation position and the imaging direction and the like of the camera 3 on the road, Vehicle tracking can be performed with high accuracy.

In the present embodiment, a configuration for performing vehicle tracking process using the pair feature based on the magnitude relationship between the two pixels extracted from the positive sample I _t of t th image V _t, limited to this rather, it extracted three or more pixels from the positive sample I _t, may be subjected to vehicle tracking process using the feature based on the magnitude relationship between the plurality of pixels. Furthermore, when extracting the pixel pair from the positive sample I _t, it is configured that the difference in luminance value is extracted two pixels above the threshold T _p, the threshold T _p is used at this time have a fixed value Alternatively, for example, the threshold value T _p may be changed according to the brightness or contrast of the image captured by the camera 3.

  In addition, although the vehicle is tracked as a moving body in the image, the tracking target is not limited to the vehicle, and another moving body such as a two-wheeled vehicle or a pedestrian may be tracked. Further, the moving body tracking device may not be configured to track the moving body based on the image captured by the camera 3 installed on the road. For example, a moving body tracking function is mounted on a digital camera or a video camera. The structure which tracks a mobile body based on the picked-up image of a portable camera etc. may be sufficient. In addition, the vehicle tracking device 1 is configured to perform the vehicle tracking process by executing the vehicle tracking program 40 by the control unit 11, that is, the configuration in which the vehicle tracking process is realized by software, but is not limited thereto. The body tracking process may be implemented with dedicated hardware. The moving body tracking process can also be provided as an IC (Integrated Circuit) chip. Further, the moving body tracking program is not only stored in a fixed recording medium such as the ROM 17 of the vehicle tracking apparatus 1, but also provided via a portable storage medium such as an optical disk or a memory card, a network, or the like. The moving object tracking process may be performed by executing a moving object tracking program on a general-purpose computer.

Further, not only the moving body in the two-dimensional image but also the moving body in the time-series three-dimensional image may be tracked. In this case, a positive sample, a negative sample, and the like are extracted as a three-dimensional region, and a pixel pair is extracted as two arbitrary points in the three-dimensional region. In addition, although a positive sample, a negative sample, a pixel pair, and the like are extracted from one tracking source image, the present invention is not limited to this. For example, 2 of t-th image V _t and t + 1-th image V _{t + 1} is used. One image may be used as a tracking source image, a positive sample and a negative sample may be extracted from each image, and a pixel pair may extract two points straddling two images. In this case, a movement destination candidate is extracted using two images of the t + 1th image Vt _{+ 1} and the t + 2nd image Vt _{+ 2} as tracking destination images, or the t + 2nd image Vt _{+ 2} and A destination candidate can be extracted using two images of the _{t + 3rd} image V _{t + 3} as tracking destination images. Further, when the tracking based image t th image V _t, it is configured to perform the tracking of the moving object to t + 1 th image V _{t + 1} as the tracking target image is not limited to this, t- A configuration may be adopted in which the moving object is tracked using the _first image V _t-1 as the tracking destination image.

  Further, the tracking method of the moving object using the pair feature is not only used for acquiring information such as a moving speed and a moving direction of a specific object such as a vehicle or a pedestrian from an image captured by a camera. It can be applied to image processing that requires processing to search for a specific location in the image, such as Moving Picture Expert Group) video compression processing. Can be

(Modification 1)
In the above-described embodiment, as shown in FIGS. 6 and 7, the vehicle tracking device 1 and the vehicle tracking program 40 randomly generate a predetermined number of pixel pair sets and select the one with the smallest similarity. By doing so, the pair feature is obtained. However, the present invention is not limited to this. The vehicle tracking device 1 and the vehicle tracking program 40 according to the modified example 1 randomly generate more pixel pairs than necessary, and acquire the pair features by selecting the necessary number of pixel pairs in descending order of similarity. Do.

FIG. 10 is a flowchart illustrating a procedure of pair feature acquisition processing according to the first modification, and is processing performed by the control unit 11 of the vehicle tracking device 1 executing the vehicle tracking program 40. Control unit 11 of the vehicle tracking device 1, first, the camera 3 is to obtain the t-th image V _t captured, extracts an image region including a target object as a positive sample I _t (step S41). Then the control unit 11, from the extracted positive samples I _t, the difference between the brightness values is M sets randomly extracts a threshold T _p or more pairs of pixels (step S42). However, when the number of pixel pairs necessary for the tracking process of the vehicle is N, the extraction number M in step S42 is a value sufficiently larger than N.

Next, the control unit 11, from the periphery of the positive samples I _t at t-th image V _t, to extract one negative sample J _t (step S43), extracted from a negative sample J _t that, in step S42 the positive samples It extracts the position of a pixel pair corresponding to pixel pair extracted from I _t (step S44). Next, the control unit 11 calculates the individual similarity r (p, q, J) for each pixel pair based on the above equation (1-5) (step S45).

The processing in steps S43 to S46 is processing that is repeatedly performed for each of ^F negative samples J _t ¹ ,..., J _t ^F extracted from the image V _t . After calculating individual similarity r in step S45 (p, q, J) to each negative sample J _t, the control unit 11 sums the already determined individual similarity of each negative sample J _t for each pixel pair (Step S46). Thereafter, the control unit 11, F-number of negative samples J _t ^1, ..., determines whether or not the process of calculating the sum of each individual similarity calculation and the pixel pairs for all J _t ^F (Step S47) If the processing has not been completed for all negative samples J _t ¹ ,..., J _t ^F (S47: NO), the processing returns to Step S43, and the processing of Steps S43 to S46 described above is performed. Repeat.

Further, when the processing is completed for all the negative samples J _t ¹ ,..., J _t ^F (S47: YES), the control unit 11 uses the total value of the individual similarities calculated by repeating step S46 as a pixel pair. Comparison is made every time, and from the M pixel pairs extracted in step S42, pixel pairs having the smallest similarity are selected from the top to N pairs (step S48), and the pair feature acquisition process is terminated. N sets of pixel pair selected in step S48 becomes a pair, wherein the positive sample I _t, used in subsequent processing.

The vehicle tracking device 1 and the vehicle tracking program 40 according to Modification 1 having the above-described configuration extract in advance M pixel pairs that are larger than the required number N from the positive samples It of the t-th image V _t , and M pixels positive samples I _t and negative samples J _t ¹ from the pair, ..., with the configuration for selecting the N sets of pixel pairs that can be more accurately determined and J _t ^F from the upper, while reducing the calculation amount, the accuracy A good mobile tracking process can be realized.

(Modification 2)
In the above-described embodiment, as illustrated in FIG. 8, when extracting the destination candidate image area from the t + 1-th image V _{t + 1} , image areas having different sizes are extracted according to the extraction positions. , it is configured to adjust the image area to the same size as the positive samples I _t, is not limited thereto. The vehicle tracking device 1 according to the modification 2 is configured to adjust the extraction position of the pixel pair extracted from the movement destination candidate according to the size of the movement destination candidate extracted from the t + 1-th image V _{t + 1} .

FIG. 11 is a schematic diagram for explaining a pixel pair extraction method from movement destination candidates according to the second modification. For example, a positive sample I _t extracted from t th image V _t is the image area of 8 pixels × 8 pixels, representing each pixel in the image area at coordinates (x, y), the transverse direction of the image area Is the x direction, the vertical direction is the y direction, the upper left pixel is the coordinate (1, 1), and the lower right pixel is the coordinate (8, 8). Further, to a pixel pair as a pair features from the positive sample _{I t (1,3) → (3,8} ), and that extracts the pixel pair (6,2) → (5,5) (Fig. 11 (a)).

If the image area of the same size is extracted as the destination candidate from t + 1 th image V _{t + 1} positive samples I _t is a pixel from the destination candidate (1,3) → (3,8) What is necessary is just to extract a pair and the pixel pair of (6,2)-> (5,5), and determine similarity. In contrast, if the size of the destination candidates extracted from t + 1 th image V _{t + 1} is different from the size of the positive samples I _t, vehicle tracking apparatus 1, the destination candidate as the size of the positive samples I _t Size The coordinates of the pixel pair extracted from the movement destination candidate are adjusted according to the ratio.

For example, since when the image region of 16 × 16 pixels as a destination candidate is extracted, the size of the destination candidates is two-fold, respectively vertically and horizontally relative to the size of the positive samples I _t, the destination candidate The coordinates of the pixel pair to be extracted are each doubled in the x direction and the y direction. That is, a pixel pair of (2, 6) → (6, 16) and a pixel pair of (12, 4) → (10, 10) are extracted from the movement destination candidates of 16 pixels × 16 pixels (FIG. 11). (See (b)).

Although not shown, when the size of the destination candidate is smaller than the size of the positive samples I _t is also, for example, if the destination candidate is vertically and horizontally half times the size of the positive samples I _t, The coordinates of the pixel pair to be extracted may be halved in the x direction and the y direction (fractions are rounded down or rounded off).

The vehicle tracking device 1 according to the modified example 2 having the above configuration differs depending on the extraction position in the image when extracting an image region as a movement destination candidate from the t + 1-th image V _{t + 1} (tracking destination image). extracting an image region of size, depending on the ratio between the size and the size of the movement destination candidate of the positive samples I _t, by adjusting the extraction position of the pixel pairs to be extracted from the moving destination candidate, the camera 3 on the road It is possible to extract destination candidates suitable for the size of the moving body in the captured image that changes depending on the installation position, the imaging direction, and the like, and the vehicle can be accurately tracked.

(Modification 3)
In the above-described embodiment, the vehicle tracking device 1 is configured to acquire the magnitude relationship related to the pixel pair based on the luminance value of each pixel of the image captured by the camera 3, but is not limited thereto. FIG. 12 is a block diagram illustrating a configuration of the vehicle tracking device 1 according to the third modification. The vehicle tracking device 1 according to the modification 3 is different from the vehicle tracking device 1 shown in FIG. 2 in that the vehicle tracking program 40 includes an edge image generation unit 47.

The edge image generation means 47 calculates the gradient (edge strength) of the luminance value between each pixel in the captured image and the other pixels adjacent to this pixel with respect to the captured image of the camera 3. An edge image with the pixel value of each pixel as the edge strength is generated. The generation of the edge image can be performed by image processing using, for example, a Sobel filter. The vehicle tracking device 1 uses the edge images generated by the edge image generation means 47 as the t-th image V _t , the t + 1-th image V _{t + 1} . Subsequent vehicle tracking processing is the same as the procedure described above.

  The vehicle tracking device 1 according to the third modified example having the above configuration is used when the edge image can capture the characteristics of the vehicle, for example, the image is unclear due to the installation environment of the camera 3 or the environment around the vehicle. In addition, an edge image can be generated from the captured image of the camera 3, and the vehicle tracking can be performed based on the pair feature based on the edge image, so that the tracking accuracy of the vehicle can be improved.

1 Vehicle tracking device (moving object detection device)
3 Camera 11 Control unit 12 Image input unit 13 A / D conversion unit 14 Image memory 15 Communication unit 16 Storage unit 17 ROM
40 Vehicle tracking program (computer program)
41 Vehicle detection means 42 Positive sample extraction means (tracking target sample extraction means)
43 Negative sample extraction means (ambient sample extraction means)
44 Pair feature acquisition means (tracking target pixel combination extraction means, surrounding pixel combination extraction means, determination means, selection means)
45 Moving destination candidate extraction means (image area extraction means, adjustment means)
46 Movement destination determination means (pixel combination extraction means, determination means, movement destination determination means)
47 Edge image generation means (calculation means)

Claims (10)

Based on a plurality of images captured in time series, in a mobile tracking device that tracks the destination of a mobile in the image,
A tracking target sample extracting means for extracting, as a tracking target sample, an image region of a predetermined size including a moving body from the tracking source image of the plurality of images;
A surrounding sample extracting means for extracting a plurality of image areas of a predetermined size around the tracking target sample as surrounding samples from the tracking source image;
A tracking target pixel combination extraction unit that extracts a plurality of pixel combinations of a plurality of pixels from the tracking target sample extracted by the tracking target sample extraction unit;
A surrounding pixel combination extracting unit that extracts a plurality of sets of pixel combinations from the surrounding samples extracted by the surrounding sample extracting unit according to the extraction position of the pixel combination extracted by the tracking target pixel combination extracting unit;
A determination unit that determines a magnitude relationship between pixel values of pixels included in the pixel combination extracted by the tracking target pixel combination extraction unit and a magnitude relationship between pixel values included in the pixel combination extracted by the surrounding pixel combination extraction unit; ,
In accordance with the determination result of the determination unit, a selection unit that selects a plurality of pixel combinations capable of determining the tracking target sample and the surrounding sample from a plurality of pixel combinations extracted by the tracking target pixel combination extraction unit;
Image region extraction means for extracting a plurality of image regions from the tracking destination image of the plurality of images;
Pixel combination extraction means for extracting a plurality of sets of pixel combinations from each image area extracted by the image area extraction means in accordance with an extraction position of the pixel combination selected by the selection means from the tracking target sample;
Determining means for determining a magnitude relationship between pixel values of each pixel included in the pixel combination extracted by the pixel combination extracting means;
Based on the magnitude relationship related to the pixel combination selected by the selection means and the size relationship related to the pixel combination of each image area extracted by the image area extraction means, a destination to determine the destination image area of the moving object A moving body tracking device comprising: determining means. The mobile tracking apparatus according to claim 1, wherein the tracking target pixel combination extracting unit extracts a pixel combination of a plurality of pixels having a pixel value difference equal to or greater than a threshold value. The tracking target pixel combination extracting unit is configured to extract a plurality of pixel combination groups including a predetermined number of pixel combinations,
The selecting means is configured to select a pixel combination group having the highest discrimination accuracy of the tracking target sample and the surrounding sample from a plurality of pixel combination groups extracted by the tracking target pixel combination extracting means. The moving body tracking device according to claim 1 or 2. The selecting means is configured to select, from a plurality of pixel combinations extracted by the tracking target pixel combination extracting means, a predetermined number of pixel combinations having high discrimination accuracy of the tracking target sample and the surrounding samples. The mobile body tracking device according to claim 1 or 2, wherein The image area extraction means is configured to extract image areas of different sizes according to the extraction position in the tracking destination image,
An adjustment unit for adjusting the size of the image region extracted by the image region extraction unit to the size of the tracking target sample;
The mobile object tracking device according to any one of claims 1 to 4, wherein the pixel combination extracting unit extracts a pixel combination from an image region adjusted by the adjusting unit. . The image area extraction means is configured to extract image areas of different sizes according to the extraction position in the tracking destination image,
The pixel combination extracting unit adjusts the extraction position of the pixel related to the pixel combination in accordance with the ratio of the size of the image region extracted by the image region extracting unit and the size of the tracking target sample. The mobile body tracking device according to any one of claims 1 to 4. A calculation means for calculating the edge strength of the image;
The moving body tracking according to any one of claims 1 to 6, wherein the tracking source image and the tracking destination image are images in which pixel values are converted into edge intensity by the calculation unit. apparatus. In a tracking method for tracking a moving destination of a moving object in the image based on a plurality of images captured in time series,
From the tracking source image of the plurality of images, an image region of a predetermined size including a moving object is extracted as a tracking target sample,
From the tracking source image, a plurality of image areas of a predetermined size around the tracking target sample are extracted as surrounding samples,
Extract multiple sets of pixel combinations from multiple extracted pixels from the sample to be tracked,
According to the extraction position of the pixel combination extracted from the tracking target sample, a plurality of pixel combinations are extracted from the surrounding sample,
Determining the magnitude relationship between the pixel values of the pixels included in the pixel combination extracted from the tracking target sample and the magnitude relationship between the pixel values included in the pixel combination extracted from the surrounding samples,
According to the determination result of the magnitude relationship, a plurality of pixel combinations that can discriminate the tracking target sample and the surrounding samples are selected from a plurality of pixel combinations extracted from the tracking target sample,
Extracting a plurality of image regions from the tracking destination image of the plurality of images,
According to the extraction position of the selected pixel combination from the tracking target sample, each of a plurality of pixel combinations is extracted from each image region extracted from the tracking destination image,
Determine the magnitude relationship of the pixel values of each pixel included in the extracted pixel combination,
A tracking method comprising: determining an image area to which the moving object is moved based on a magnitude relation relating to a selected pixel combination and a magnitude relation relating to a pixel combination of each extracted image area. From the tracking source image, extract an image area of a predetermined size as a tracking target sample,
From the tracking source image, a plurality of image areas of a predetermined size around the tracking target sample are extracted as surrounding samples,
Extract multiple sets of pixel combinations from multiple extracted pixels from the sample to be tracked,
According to the extraction position of the pixel combination extracted from the tracking target sample, a plurality of pixel combinations are extracted from the surrounding sample,
Determining the magnitude relationship between the pixel values of the pixels included in the pixel combination extracted from the tracking target sample and the magnitude relationship between the pixel values included in the pixel combination extracted from the surrounding samples,
According to the determination result of the magnitude relationship, a plurality of pixel combinations that can discriminate the tracking target sample and the surrounding samples are selected from a plurality of pixel combinations extracted from the tracking target sample,
Extract multiple image areas from the tracking destination image,
According to the extraction position of the selected pixel combination from the tracking target sample, each of a plurality of pixel combinations is extracted from each image region extracted from the tracking destination image,
Determine the magnitude relationship of the pixel values of each pixel included in the extracted pixel combination,
A tracking method comprising: determining an image area to which the tracking target sample is moved based on a magnitude relation relating to the selected pixel combination and a magnitude relation relating to the pixel combination of each extracted image area. In a computer program for causing a computer to execute a moving body tracking step for tracking a moving destination of a moving body in the image based on a plurality of images captured in time series,
The moving object tracking step includes:
Extracting a predetermined size image area including a moving object as a tracking target sample from the tracking source image of the plurality of images;
Extracting a plurality of image areas of a predetermined size around the sample to be tracked as surrounding samples from the tracking source image;
Extracting a plurality of pixel combinations of a plurality of pixels from the extracted tracking target sample;
Extracting a plurality of sets of pixel combinations from the surrounding samples according to an extraction position of the pixel combination extracted from the tracking target sample;
Determining the magnitude relationship between the pixel values of the pixels included in the pixel combination extracted from the tracking target sample and the magnitude relationship between the pixel values included in the pixel combination extracted from the surrounding samples;
Selecting a plurality of pixel combinations capable of discriminating the tracking target sample and the surrounding samples from a plurality of pixel combinations extracted from the tracking target sample according to a determination result of the magnitude relationship;
Extracting a plurality of image regions from a tracking destination image of the plurality of images;
Extracting each of a plurality of sets of pixel combinations from each image region extracted from the tracking destination image according to an extraction position of the selected pixel combination from the tracking target sample;
Determining a magnitude relationship between pixel values of each pixel included in the extracted pixel combination;
And a step of determining an image area to which the moving object is moved based on a magnitude relation relating to the selected pixel combination and a magnitude relation relating to the extracted pixel combination of each image area.

Images