JP6847420B2 - Velocity estimator, velocity estimation method, and velocity estimation program - Google Patents

Description

The present invention relates to a speed estimation device, a speed estimation method, and a speed estimation program.

In recent years, an image processing device that analyzes the movement of a vehicle or the like based on an image captured by a wide area photographing device such as a fixed point camera or a surveillance camera has been known.

For example, Patent Document 1 describes an image processing device that calculates information (speed, speed vector, etc.) regarding the movement of a moving object in a captured image based on a time-series captured image from a fixed-point camera equipped with a stereo camera. Has been done. The image processing device disclosed in Patent Document 1 displays the captured time-series captured images on the display device together with information on motion. This image processing device obtains a display image including a time-series captured image of a vehicle moving in a fixed direction and a speed vector of the vehicle traveling in a fixed direction in the captured image. Create and display the created display image on the display device.

Japanese Unexamined Patent Publication No. 2013-127716

However, in the prior art, it is not possible to provide information on the movement of a vehicle or the like unless a stereo camera is provided as a wide area photographing device for photographing a wide area. Further, when the image information captured by the photographing device includes a plurality of roads in different traveling directions, it is difficult to analyze the movement of the vehicle for each traveling direction.

The present invention has been made to solve such a problem, and is a speed estimation device capable of estimating the speed of a vehicle or the like moving in the traveling direction of the road based on image information including the road. , Speed estimation methods, and speed estimation programs.

The speed estimation device according to the present invention is a region indicating a range of a road region included in the first image information, the second image information, and the first and second image information captured by the photographing unit at predetermined time intervals. A storage unit that stores information and angle range information, and a feature point extraction unit that extracts a plurality of first feature points from the first image information and extracts a plurality of second feature points from the second image information. A vector calculation unit that calculates a plurality of movement vectors starting from each of the plurality of first feature points and ending at a plurality of second feature points corresponding to each of the plurality of first feature points, and a plurality of movements. A speed calculation unit that extracts a specific movement vector whose start point or end point is included in the area information and whose direction is included in the angle range information, and calculates the speed of a vehicle traveling in the road area based on the specific movement vector. And an output unit that outputs the calculated speed.

Further, in the speed estimation device according to the present invention, the photographing unit captures a predetermined number of image information before photographing the first and second image information, and based on the predetermined number of image information, a predetermined number of images. A background image generation unit that generates a background image excluding a moving object image included in the information, a road area generation unit that generates area information based on the difference information between each of a predetermined number of image information and the background image, and a road area generation unit. It is preferable to provide.

Further, in the speed estimation device according to the present invention, in the speed calculation unit, the first difference information between the first image information and the background image and the second difference information between the second image information and the background image are similar. When it is determined whether or not the first difference information and the second difference information are similar, it is determined in the period from the acquisition of the first image information to the acquisition of the second image information. It is preferable to determine that the vehicle in the road area is not moving.

The speed estimation method according to the present invention is a speed estimation method by a computer connected to a photographing unit and having a storage unit, and the first image information and the second image information captured by the photographing unit at predetermined time intervals. The image information, the area information indicating the range of the road area included in the first and second image information, and the angle range information are stored in the storage unit, and a plurality of first feature points are extracted from the first image information. At the same time, a plurality of second feature points are extracted from the second image information, each of the plurality of first feature points is set as a start point, and a plurality of second feature points corresponding to each of the plurality of first feature points are set as end points. A plurality of movement vectors to be calculated are calculated, and a specific movement vector whose start point or end point is included in the area information and whose direction is included in the angle range information is extracted from the plurality of movement vectors, and the road area is based on the specific movement vector. Includes calculating the speed of the vehicle traveling in and outputting the calculated speed.

The speed estimation program according to the present invention has a first image information and a second image information, a first image, and a second image information captured by the photographing unit at predetermined time intervals on a computer connected to the photographing unit and having a storage unit. The area information indicating the range of the road area included in the second image information and the angle range information are stored in the storage unit, a plurality of first feature points are extracted from the first image information, and the second image is displayed. A plurality of second feature points are extracted from the information, and a plurality of movement vectors having each of the plurality of first feature points as a start point and a plurality of second feature points corresponding to each of the plurality of first feature points as end points are obtained. From the plurality of movement vectors, the specific movement vector whose start point or end point is included in the area information and whose direction is included in the angle range information is extracted, and the speed of the vehicle traveling in the road area is extracted based on the specific movement vector. Is calculated, and the calculated speed is output.

The speed estimation device, the speed estimation method, and the speed estimation program according to the present invention make it possible to estimate the speed of a vehicle or the like moving in the traveling direction of the road based on the image information including the road.

It is a schematic diagram for demonstrating the outline of an image transmission system. It is a schematic diagram for demonstrating the outline of an image transmission system. It is a figure which shows an example of the schematic structure of the image transmission system 1. It is a figure which shows an example of the schematic structure of the speed estimation device 3. (A) is a diagram showing an example of image information, and (b) is a diagram showing an example of feature points extracted from the image information. (A) is a diagram showing an example of a movement vector calculated based on image information, and (b) is a diagram showing an example of a road region included in the image information. It is a figure which shows an example of the schematic structure of the speed estimation device 3. (A) to (b) are diagrams showing an example of the data structure of various tables. (A) to (b) are diagrams showing an example of the data structure of various tables. It is a figure which shows an example of the operation flow of speed estimation processing. It is a figure which shows an example of the operation flow of the calculation process.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

(Outline of this embodiment)
1 and 2 are schematic views for explaining an outline of an image transmission system. The image transmission system has at least a wide area photographing device and a speed estimation device.

As shown in FIG. 1, the wide area photographing apparatus is installed in a specific building or the like, and photographs the imaging target space in a predetermined imaging direction at predetermined time intervals. The predetermined time interval is 1/30 second, and the wide area photographing apparatus outputs image information obtained by photographing the imaging target space in the predetermined imaging direction every 1/30 second. The predetermined time interval may be 1 second, 1/5 second, 1/50 second, or the like.

When the speed estimation device acquires a plurality of image information output from the wide area photographing device, the speed estimation device stores the acquired plurality of image information according to the time series in which the images are photographed.

Hereinafter, an example of an outline of the speed estimation process based on a plurality of stored image information will be described with reference to FIG.

First, the speed estimation device extracts feature points from each of the first image information among the plurality of stored image information and the second image information captured after the first image information. When the first image information taken first among the plurality of image information is the first image information, the second image information among the plurality of image information is the second image information. Hereinafter, the feature points extracted from the first image information will be referred to as a first feature point, and the feature points extracted from the second image information will be referred to as a second feature point.

FIG. 2 (1-1) is an example of the first image information including the first feature point, and FIG. 2 (1-2) is an example of the second image information including the second feature point. is there. As shown in (1-1) of FIG. 2, a plurality of first feature points are extracted from the first image information, and as shown in (1-2) of FIG. 2, a plurality of second feature points are second. It is extracted from the image information of 2.

The speed estimation device identifies the second feature point corresponding to each of the plurality of first feature points from the plurality of second feature points. The speed estimation device calculates a movement vector for each of the plurality of first feature points, with each first feature point as a start point and each second feature point corresponding to each first feature point as an end point.

The movement vector is, for example, Optical Flow. The optical flow is calculated by the Lucas-Kanade method, the Horn-Schunk method, or the like.

FIG. 2 (2) is an example of the first image information including the movement vector.

As shown in (2) of FIG. 2, when the first feature point and the second feature point corresponding to the first feature point are separated, it is estimated that the first feature point is moving. The first image information and the second image information may include a vehicle traveling on a road, a flying object such as a bird or an aircraft, and a moving object such as a moving machine. In addition, when the color information, brightness, etc. of each pixel of each image information are different depending on the shooting conditions when the first image information and the second image information are taken, the state of the lens, etc., a moving object. A movement vector that does not indicate is sometimes estimated.

Therefore, the image transmission system of the present embodiment uses the area information indicating the range of the road area included in the first and second image information and the angle range information to move a movement vector related to the vehicle on the road. By extracting only, the speed of the vehicle with high accuracy is calculated ((3) in FIG. 1).

The road area is an image area corresponding to a road included in the image information. The area information is information that defines the polygonal shape of the road area, and is information that indicates a two-dimensional coordinate group that defines the polygonal shape. The area information in the example shown in FIG. 2 (3) is information that defines the shape of the image area corresponding to the road included in the first image area.

The angle range information is information that defines a predetermined angle set for each road area. The angle range information is the traveling direction of the road corresponding to the road area and a predetermined angle. The traveling direction is an angle composed of a predetermined axis (an axis parallel to either the vertical direction or the horizontal direction of the pixel in the image information) and the traveling direction. For example, the angle range information is a traveling direction (16 degrees), an angle (60 degrees), and the like.

The angle range information may be information about a first line segment and a second line segment constituting a predetermined angle. For example, the information about the first line segment is the information indicating the angle composed of the predetermined axis and the first line segment, and the information about the second line segment is the information about the predetermined axis and the second line segment. It is information indicating an angle composed of and. For example, the angle range information includes information on the first line segment (-14 degrees), information on the second line segment (+46 degrees), and the like. Further, the information about the first line segment may be a first vector (for example, a unit vector) parallel to the first line segment, and the information about the second line segment is parallel to the second line segment. It may be a second vector (eg, a unit vector).

In the speed calculation process, among the calculated movement vectors, the movement vector included in the area information indicating the range of the road area included in the first and second image information and whose direction is included in the angle range information is specified. Extracted as a moving vector.

In the example shown in FIG. 2 (3), first, among the movement vectors shown in FIG. 2 (2), a movement vector whose at least a part is included in the area information is extracted. Next, among the movement vectors included in the area information, the movement vector whose direction is within the range of the angle indicated by the angle range information is extracted as a specific movement vector.

Then, the speed estimation device calculates the average speed of the vehicle in the road area based on the extracted specific movement vector. For example, the speed estimator has taken the length of the extracted specific movement vector, the actual distance per pixel of the image information, and the time when the first image information was taken and the second image information. The speed corresponding to each specific movement vector is calculated based on the difference time from the time. Next, the speed estimation device calculates the average value of the calculated speeds as the average speed.

The speed estimation device outputs the average speed calculated for each road area. The output of the average speed is, for example, displaying a character string indicating the speed, an object image (gauge image or the like) indicating the speed on a display device, or outputting audio data or the like indicating the speed by a speaker or the like.

As described above, the speed estimation device can estimate the speed based on the movement vector corresponding to the area information corresponding to the road area included in the captured image information. This makes it possible to estimate the speed of the vehicle or the like for each traveling direction.

The above-mentioned explanations of FIGS. 1 and 2 are merely explanations for deepening the understanding of the contents of the present invention. Specifically, the present invention may be carried out in each of the embodiments described below, and may be carried out by various modifications without substantially exceeding the principles of the present invention. All such variations are included within the scope of the present invention and the present specification.

(Image transmission system 1)
FIG. 3 is a diagram showing an example of a schematic configuration of the image transmission system 1. The image transmission system 1 includes a wide area photographing device 2, a speed estimation device 3, a transmitting device 4, and a receiving device 5. The wide area imaging device 2 and the speed estimation device 3 are connected by a predetermined communication cable, and the wide area imaging device 2 transmits image information to the speed estimation device 3 via the predetermined communication cable. The wide area imaging device 2 may transmit image information to the speed estimation device 3 using a predetermined wireless communication. The predetermined wireless communication is, for example, short-range wireless communication between devices such as Bluetooth (registered trademark), and the wide area photographing device 2 and the speed estimation device 3 establish the wireless communication by the predetermined wireless communication, and the direct wireless communication is performed. Can communicate.

The wide area photographing device 2 is a fixed point camera installed in a specific building or the like. The wide area photographing device 2 may be a surveillance camera or the like installed in a specific building or the like. The wide area photographing device 2 is an example of an photographing unit.

The wide area photographing apparatus 2 includes an optical system having a predetermined angle of view, an image pickup device, and an image processing unit. The optical system is, for example, an optical lens, and a light flux from a subject is imaged on an image pickup surface of an image pickup device. A plurality of optical systems may be provided. For example, the wide area photographing apparatus 2 may be a stereo camera provided with two optical systems or a trinocular camera provided with three optical systems.

The image pickup device is a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, and outputs an image of a subject image formed on the image pickup surface. The image processing unit continuously creates and outputs image information in a predetermined file format at predetermined time intervals from images generated by the image sensor.

The wide area photographing apparatus 2 transmits a plurality of output image information to the speed estimation apparatus 3 at predetermined transmission time intervals. The predetermined transmission time interval is 10 seconds. The predetermined transmission time interval may be 5 seconds, 15 seconds, 30 seconds, or 1 minute. Further, the wide area photographing apparatus 2 may transmit the output image information to the speed estimation apparatus 3 each time the image information is output.

The speed estimation device 3 is a computer such as a personal computer (personal computer, PC) or a server. The speed estimation device 3 may be any information processing device as long as it has a function of acquiring image information from the wide area photographing device 2 and executing the speed estimation process.

The transmitter 4 is a transmitter or the like owned by a broadcaster or the like and for transmitting broadcast data. The transmitting device 4 generates broadcast data according to an operation of a broadcasting company or the like, and transmits the generated broadcasting data to the receiving device 5 via a relay station or the like. The transmitting device 4 may transmit the generated broadcast data to the receiving device 5 via a cable television (CATV) network.

Broadcast data includes image data, audio data, broadcast information data, and the like. The image data is, for example, video data of a broadcast program, still image data, or the like. Broadcast information data is graphic image data, character data, and / or control data described by a description language such as BML (Broadcast Markup Language). The graphic image data and the character data are displayed by a BML browser or the like executed by the receiving device 5. The control data is data for specifying control information such as display control. For example, control information for moving and displaying graphic image data or executing channel selection processing based on viewer's operation input information. It is data for identification. The broadcast data transmitted by the transmission device 4 of the present embodiment includes data related to the image information output by the wide area photographing device 2.

The receiving device 5 is a television receiver or the like installed in each home or the like, and is connected to the antenna unit 6. The receiving device 5 receives the broadcast data transmitted from the transmitting device 4 via the antenna unit 6, displays the image data included in the broadcast data, and outputs the audio data included in the broadcast data as audio.

(Configuration of speed estimation device 3)
FIG. 4 is a diagram showing an example of a schematic configuration of the speed estimation device 3.

The speed estimation device 3 has a function of acquiring image information from the wide area photographing device 2 and executing a speed estimation process. Therefore, the speed estimation device 3 includes a communication unit 31, a storage unit 32, an input unit 33, a display unit 34, and a processing unit 35.

The communication unit 31 includes a communication interface circuit for connecting the speed estimation device 3 to the wide area photographing device 2, and communicates with the wide area photographing device 2. The communication unit 31 supplies the image information output from the wide area photographing apparatus 2 to the processing unit 35.

The storage unit 32 includes a magnetic tape device, a magnetic disk device, an optical disk device, and the like. Further, the storage unit 32 may include a semiconductor memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 32 stores an operating system program, a driver program, a computer program, data, and the like used for processing in the processing unit 35. The computer program stored in the storage unit 32 is, for example, a speed estimation program for causing the processing unit 35 to execute the speed estimation process. The computer program may be a computer-readable portable recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a DVD-ROM (Digital Versaille Disc-Read Only Memory), using a known setup program or the like. It may be installed in the storage unit 32.

The data stored in the storage unit 32 includes a plurality of image information output from the wide area photographing apparatus 2, various tables described later, and the like. The storage unit 32 may temporarily store temporary data related to various processes such as speed estimation processing. In this case, the storage unit 32 may include a volatile RAM memory and store data temporarily output according to the execution of various processes.

The input unit 33 is a keyboard and a mouse. The input unit 33 may be a touch panel or the like. The user can input characters, numbers, symbols, etc. using the input unit 33. When the input unit 33 is operated by the user, the input unit 33 generates a signal corresponding to the operation. Then, the generated signal is supplied to the processing unit 35 as a user's instruction.

The display unit 34 is a liquid crystal display. The display unit 34 may be an organic EL (Electro-Luminescence) display or the like. The display unit 34 displays an image corresponding to the image data supplied from the processing unit 35, an image corresponding to the image data, and the like.

The processing unit 35 includes one or more processors and peripheral circuits thereof. The processing unit 35 is a CPU (Central Processing Unit) that comprehensively controls the overall operation of the speed estimation device 3. The processing unit 35, such as the communication unit 31 and the display unit 34, so that various processes of the speed estimation device 3 are executed by an appropriate procedure based on the program stored in the storage unit 32 and the operation of the input unit 33. Control the operation. The processing unit 35 executes processing based on a program (operating system program, driver program, application program, etc.) stored in the storage unit 32. In addition, the processing unit 35 can execute a plurality of programs (computer program programs, etc.) in parallel.

The processing unit 35 includes an acquisition unit 351, a generation unit 352, an image information extraction unit 353, a feature point extraction unit 354, a vector calculation unit 355, a vector extraction unit 356, a speed calculation unit 357, and a display processing unit 358. Each of these units included in the processing unit 35 is a functional module realized by a program executed by the processor included in the processing unit 35. Each of these units included in the processing unit 35 may be mounted on the speed estimation device 3 as firmware. The details of the acquisition unit 351, the generation unit 352, the image information extraction unit 353, the feature point extraction unit 354, the vector calculation unit 355, the vector extraction unit 356, the speed calculation unit 357, and the display processing unit 358 will be described later.

FIG. 5A is a diagram showing an example of image information. The image information shown in FIG. 5A is one of a plurality of image information output from the wide area photographing apparatus 2.

(Feature point)
FIG. 5B is a diagram showing an example of feature points extracted from image information.

The feature points are extracted from the image information using a predetermined feature point extraction method. The predetermined feature point extraction method is a SIFT (Scale-Invariant Feature Transfer) method, a SURF (Speeded Up Robot Features) method, or the like. Further, a corner detection method may be used as the feature point extraction method. In the corner detection method, corners and isolated points are detected as feature points based on the amount of change in the pixel value when the pixel is moved.

Hereinafter, the feature point extraction method by the SIFT method will be described. First, by executing DoG (Difference-of-Gaussian) processing on the image information, candidates for feature points in the image information are detected. In the DoG process, a plurality of DoG images are generated based on the image information, and pixels in which extreme values are detected from the generated plurality of DoG images are specified as feature point candidates.

Next, the feature points are extracted by deleting the candidates satisfying the predetermined deletion conditions from the feature point candidates. The predetermined deletion condition is that the candidate feature points are the same as the points detected by the corner detection method or the like. Further, the predetermined deletion condition may be a condition that the candidate pole value of the feature point is equal to or less than a predetermined numerical value. Candidates for the remaining feature points deleted in this way are extracted as feature points. For the feature point extraction method by the SIFT method, refer to, for example, Japanese Patent Application Laid-Open No. 2015-213615.

The feature point extraction process is executed for each of the plurality of image information stored in the storage unit 32.

(Movement vector)
FIG. 6A is a diagram showing an example of a movement vector calculated based on image information. The movement vector starts from the feature point extracted in the first image information of the plurality of image information, and in the second image information taken after the first image information of the plurality of image information. The extracted feature point is the end point.

When the first image information taken first among the plurality of image information is the first image information, the second image information taken out of the plurality of image information is the second image information. Is. Similarly, when the 20th captured image information of the plurality of image information is the first image information, the 21st captured image information of the plurality of image information is the second image information. is there.

The second image information is not limited to the image information taken after the first image information, but also the image information taken within a predetermined time (for example, 1 second) from the time when the first image information was taken. Good. For example, when the first image information taken first among the plurality of image information is the first image information, the sixth of the plurality of image information (for example, 1/6 of the first image information). The image information taken after a second) may be the second image information.

The movement vector calculation process is, for example, an optical flow calculation process. In the optical flow calculation process, for example, the Lucas-Kanade method, the Horn-Schunk method, the Gunnar Farneback method, and the like are used.

In the Lucas-Kanade method, the first feature point (or the first pixel) in the first image information is used as a starting point, and the second feature point in the second image information that can be regarded as the same as the specific point. The optical flow (motion vector) with (or the second pixel) as the end point is calculated. When the first feature point and the second feature point have the same brightness, the first feature point and the second feature point are considered to be the same. In this case, the first feature point and the optical flow can be expressed as in the equation (1).

Incidentally, I is the first characteristic point luminance in the first image information, x and y are two-dimensional coordinates of the first feature point in the first image information, v _x and v _y are optical flow (motion vector ).

When the equation (1) is Taylor-expanded, the equation (2) is obtained.

The following optical flow constraint equation (3) is obtained based on the equation (2).

Since the optical flow (motion vector) is not calculated from the equation (3), it is assumed that a virtual window is set around the first feature point, and the optical flow is the same in the window. Is calculated. For example, the optical flow is calculated by the following equation (4).

Regarding the calculation process of the optical flow by the Lucas-Kanade method, for example, B.I. D. Lucas and T. Kanade, "An iterative image registration technology with application to station vision", in Proc Imaging Understancing Workshop, pp. See 121-130, 1981.

(Area information)
FIG. 6B is a diagram showing an example of area information included in the image information.

In the example shown in FIG. 6B, the image information includes two road areas. The area information is information indicating a two-dimensional coordinate group that defines a polygonal shape that is the outer shape of the road area. Hereinafter, one of the two road areas will be referred to as a first road area, and the area information corresponding to the first road area will be referred to as a first area information. Further, the other of the two road areas is referred to as a second road area, and the area information corresponding to the second road area is referred to as a second area information.

The image information may include only one road area, or may include three or more road areas.

(Specific movement vector)
FIG. 7A is a diagram showing a specific movement vector corresponding to the first region information. The specific movement vector corresponding to the first area information is a movement vector included in the first area information. When at least one of the start point and the end point of the movement vector is included in the first area information, the movement vector is extracted as a specific movement vector. In the example shown in FIG. 7A, the movement vector whose start point is included in the first region information is extracted as a specific movement vector. Further, the direction of the specific movement vector corresponding to the first area information is included in the angle range information corresponding to the first area information.

FIG. 7B is a diagram showing a specific movement vector corresponding to the second road area. Similar to the specific movement vector corresponding to the first area information, the specific movement vector corresponding to the second area information is a movement vector included in the second area information, and the direction of the specific movement vector is the first. It is included in the angle range information corresponding to the area information of 1.

(Image information table T1)
FIG. 8A is a diagram showing an example of the data structure of the image information table T1. In the image information table T1, the image information ID (identification), the date and time, the file name of the image information, and the like are stored in association with each other for each image information.

The image information ID is an example of identification information for uniquely identifying each image information.

The date and time is information indicating the date and time when the image information was taken. When the wide area photographing device 2 captures the image information, the shooting date and time is associated with the image information, and the image information and the date and time associated with the image are output.

The file name of the image information is information indicating the path of the storage folder of the image information data stored in the storage unit 32, the name of the file, and the like.

(Road area table T2)
FIG. 8B is a diagram showing an example of the data structure of the road area table T2. In the road area table T2, the area ID, the area information, the traveling direction, the angle, and the like are stored in association with each other for each road area.

The area ID is an example of identification information for uniquely identifying each road area. The area information is information that defines the polygonal shape of the road area, and is information that indicates a two-dimensional coordinate group that defines the polygonal shape.

The traveling direction is information indicating the traveling direction of the road corresponding to the road area. The traveling direction is, for example, information indicating an angle formed by a predetermined axis and the traveling direction. The predetermined axis is an axis parallel to either the vertical direction or the horizontal direction of the pixel in the image information. The angle is information indicating an error angle in the traveling direction of the road.

(First speed estimation table T3)
FIG. 9A is a diagram showing an example of the data structure of the first speed estimation table T3. In the first speed estimation table T3, the specific movement vector and the like included in the first area information of each set when two consecutive image informations out of the plurality of image information are set are managed.

In the first speed estimation table T3, a set ID, an image information ID, a specific movement vector, an average speed, and the like are stored in association with each other for each set of two image information.

The set ID is an example of identification information for uniquely identifying each set.

The image information ID is an image information ID of two image information constituting each set. In the example shown in FIG. 9A, the image information IDs associated with the set ID “0235” are “0125656258” and “0125656259”. The former is an example of the image information ID of the first image information, and the latter is an example of the image information ID of the second image information.

The specific movement vector is information indicating a specific movement vector corresponding to the first region information among the movement vectors calculated based on the two image information constituting each set. In the example shown in FIG. 9A, the information indicating the specific movement vector is the information indicating the two-dimensional coordinates of the start point and the two-dimensional coordinates of the end point of each specific movement vector.

The average speed is the speed corresponding to each specific movement vector calculated for each specific movement vector. For example, the speed is calculated based on the length of the specific movement vector, the actual distance per pixel of the image information, and the difference time of the shooting times of the two image information constituting each set.

(Second speed estimation table T4)
FIG. 9B is a diagram showing an example of the data structure of the second speed estimation table T4. In the second speed estimation table T4, the specific movement vector and the like included in the second area information of each set when two consecutive image information among the plurality of image information are set are managed.

Similar to the first speed estimation table T3, the second speed estimation table T4 stores the set ID, the image information ID, the specific movement vector, the average speed, etc. in association with each other for each set of the two image information. Will be done.

(Speed estimation process)
FIG. 10 is a diagram showing an example of the operation flow of the speed estimation process.

First, the acquisition unit 351 acquires one or more image information output from the wide area photographing apparatus 2 together with the date and time, and stores the acquired image information in the storage unit 32 (step S101). In addition, the acquisition unit 351 newly assigns an image information ID corresponding to the stored image information, and sets the image information ID of the assigned image information, the date and time of acquisition together with the image information, and the file name of the image information in the image information table. Store in T1.

Next, the image information extraction unit 353 is photographed during the background calculation period (for example, 180 seconds from the most recently photographed time) from among the plurality of image information stored in the storage unit 32 in step S101. A predetermined number of image information is extracted as initial image information. Next, the generation unit 352 determines the moving object image included in each initial image information based on the extracted predetermined number of initial image information, and generates the background image information excluding the moving object image from the initial image information. (Step S102). For the background image information generation processing, see "Shinji Morita, Kazumasa Yamazawa, Yasuhiko Terazawa, Naokazu Yokoya," Network-compatible remote monitoring system using omnidirectional image sensors ", IEICE Journal of Electronics, Information and Communication Engineers, vol. 88, no. 5, pp. 864-875, 2005 ”. The generation unit 352 is an example of a background image generation unit.

Next, the image information extraction unit 353 extracts a set of image information from the plurality of image information taken after the background image calculation period (step S103). The image information extraction unit 353 newly assigns a set ID to the extracted set of image information, and sets the respective image information IDs of the set ID and the set of image information to the first speed estimation table T3 and the first speed estimation table T3. It is stored in the speed estimation table T4 of 2. The set of image information extracted in step S103 is a set of each of all the image information in the plurality of stored image information and the image information taken next. Further, the set of image information extracted in step S103 may be a set of each of a part of the stored image information and the image information captured next. ..

Next, the feature point extraction unit 354 extracts a plurality of feature points from the set of image information extracted in step S103 (step S104). The feature point extraction unit 354 extracts a plurality of first feature points from the first image information of the set of image information by using the predetermined feature point extraction method described above, and extracts a plurality of first feature points from the set of image information. A plurality of second feature points are extracted from the second image information.

Next, the vector calculation unit 355 calculates the movement vector based on each feature point of the set of image information (step S105). The vector calculation unit 355 calculates a movement vector (optical flow) based on the plurality of first feature points and the plurality of second feature points by using the Lucas-Kanade method or the like described above.

Next, the vector extraction unit 356 extracts a specific movement vector for each road area from the calculated movement vector (step S106). First, the vector extraction unit 356 extracts the area information, the traveling direction, and the angle of each road area from the road area information table T2. Next, the vector extraction unit 356 extracts the movement vector whose first feature point is included in each area information from the movement vectors calculated in step S105 for each area information. Next, the vector extraction unit 356 is a determination angle having an extracted angle and having the first feature point as the apex for each extracted movement vector, and is substantially bisected according to the extracted traveling direction. It is determined whether or not each of the extracted movement vectors is included in the determination angle. Next, the vector extraction unit 356 extracts the movement vector determined to be included in the determination angle as a specific movement vector. Then, the vector extraction unit 356 stores the specific movement vector extracted for the first area information in the first velocity estimation table T3, and stores the specific movement vector extracted for the second area information in the second speed estimation table T3. It is stored in the speed estimation table T3.

Next, the speed calculation unit 357 executes the calculation process (step S107), and ends a series of steps of the speed estimation process. The details of the calculation process will be described later.

Then, the display processing unit 358 displays the information regarding the vehicle speed calculated by the calculation process on the display unit 34 (step S108), and ends a series of steps. For example, the display processing unit 358 extracts a plurality of image information stored in the storage unit 32 from the storage unit 32 in chronological order. Next, the display processing unit 358 extracts the average velocity information associated with the image information ID of the extracted image information from the first velocity estimation table T3 and the second velocity estimation table T4. Then, the display processing unit 358 displays the extracted image information and the average speed information on the display unit 34 in chronological order.

The speed estimation device 3 may include an external output unit that outputs the calculated vehicle speed to the transmission device 4. In this case, the transmission device 4 transmits broadcast data including the speed of the vehicle output from the speed estimation device 3.

(Calculation processing)
FIG. 11 is a diagram showing an example of the operation flow of the calculation process. Calculation processing is executed for each of the first area information and the second area information. Hereinafter, the calculation process in the first area information will be described as an example.

First, the speed calculation unit 357 determines whether or not a specific movement vector is stored in the first speed estimation table T3 for each set ID (step S201).

When the speed calculation unit 357 determines that the specific movement vector is stored (step S201-Yes), the speed calculation unit 357 calculates the average speed based on the stored specific movement vector (step S202), and ends a series of processes. For example, the speed calculation unit 357 sets the length of the specific movement vector, the actual distance per pixel of the image information, the time when the first image information is taken, and the first image information for each stored specific movement vector. The speed is calculated based on the difference time from the time when the image information of 2 is taken. Then, the speed estimation device calculates the average value of the calculated speeds, and the calculated average value is displayed in the first speed estimation table T3 as the average speed associated with the set ID of the stored specific movement vector. Remember.

When the speed calculation unit 357 determines that the specific movement vector is not stored (step S201-No), the difference information between the image information and the background image constituting the set determined that the specific movement vector is not stored. Is generated (step S203). That is, the speed calculation unit 357 generates the first difference information between the first image information and the background image constituting the set determined that the specific movement vector is not stored, and the specific movement vector is not stored. The second difference information between the second image information and the background image constituting the set determined to be is generated.

Next, the speed calculation unit 357 determines whether or not the vehicle exists in the first area information (step S204). That is, the speed calculation unit 357 determines whether or not the first difference information and the second difference information are included in the first area information, and the first difference information and the second difference information are similar. When the speed calculation unit 357 determines that the first difference information and the second difference information included in the first area information are similar, the second image information is captured after the first image information is captured. It is determined that the vehicle in the first road area is not moving in the period until the above.

When the speed calculation unit 357 determines that the vehicle exists in the first area information (step S204-Yes), it estimates that the vehicle is stopped in the road area and determines that the specific movement vector is not stored. The average velocity "0 (zero)" is stored in the first velocity estimation table T3 in association with the pair ID of the pair (step S205), and a series of processes is completed.

When the speed calculation unit 357 determines that the vehicle does not exist in the first area information (step S204-No), it estimates that no vehicle is traveling in the road area, and the specific movement vector is stored. Information indicating that the vehicle does not exist as the average speed is stored in the first speed estimation table T3 in association with the set ID of the set determined not to be performed (step S206), and a series of processes is completed.

As described above, the image transmission system 1 can estimate the speed of a vehicle or the like moving in the traveling direction of the road based on the image information including the road.

(Modification example 1)
The generation unit 352 may generate the area information and the traveling direction and angle based on a predetermined number of initial image information and the generated background image information. For example, first, the generation unit 352 generates background subtraction information between each of a predetermined number of initial image information and background image information. Next, the generation unit 352 synthesizes the generated background subtraction information to generate a composite background image. Then, the generation unit 352 generates, as area information, a two-dimensional coordinate group of the polygonal shape formed by the outer shape of the image area having a predetermined area of the generated composite background image, and stores it in the road area table T2. To do. By automatically generating the area information in this way, it is possible to reduce the operational burden on the operator of the speed estimation device 3 when the shooting direction, angle of view, etc. of the wide area photographing device 2 are changed. Become. The generation unit 352 is an example of a road area generation unit.

(Modification 2)
Further, the vector extraction unit 356 may extract a specific movement vector using only the area information of each road region without using the traveling direction and the angle. This makes it possible to improve the extraction processing speed of the specific movement vector.

(Modification example 3)
Further, the background image information generation process executed by the generation unit 352 may be executed at a predetermined timing. For example, when the wide area photographing apparatus 2 is provided with an illuminance sensor, the background image information generation process may be executed at the timing when the illuminance measured by the illuminance sensor reaches a predetermined illuminance. Further, when the shooting direction of the wide area photographing device 2 can be changed manually or automatically, the background image information generation process may be executed at the timing when the shooting direction of the wide area photographing device 2 is changed. In this way, the speed estimation processing speed can be improved by executing the background image information generation processing at a predetermined timing.

It will be appreciated by those skilled in the art that various changes, substitutions and modifications can be made to this without departing from the spirit and scope of the invention.

1 Image transmission system 2 Wide area imaging device 3 Speed estimation device 31 Communication unit 32 Storage unit 33 Input unit 34 Display unit 35 Processing unit 351 Acquisition unit 352 Generation unit 353 Image information extraction unit 354 Feature point extraction unit 355 Vector calculation unit 356 Vector extraction Unit 357 Speed calculation unit 358 Display processing unit 4 Transmitter 5 Receiver 6 Antenna

Claims (3)

A storage unit for storing first image information and second image information captured at a predetermined time interval by the imaging unit,
A background image generation unit that generates a background image excluding moving object images included in the predetermined number of image information based on a predetermined number of image information captured before the first and second image information is captured. ,
A road that generates area information indicating the range of the road area included in the first and second image information and angle range information based on the difference information between each of the predetermined number of image information and the background image. Area generator and
A feature point extraction unit that extracts a plurality of first feature points from the first image information and extracts a plurality of second feature points from the second image information.
A vector calculation unit that calculates a plurality of movement vectors starting from each of the plurality of first feature points and ending at the plurality of second feature points corresponding to each of the plurality of first feature points.
A vector extraction unit that extracts a specific movement vector whose start point or end point is included in the area information and whose direction is included in the angle range information among the plurality of movement vectors.
When the specific movement vector is extracted, the speed of the vehicle traveling in the road area is calculated based on the specific movement vector, and when the specific movement vector is not extracted, the first image information and the said A speed calculation unit that determines whether or not there is a vehicle that is not moving in the road area based on the difference information between the second image information and the background image.
An output unit that outputs the calculated speed or the result of the determination,
A speed estimation device comprising. It is a speed estimation method by a computer that is connected to the photographing unit and has a storage unit.
Storing the first image information and second image information captured at a predetermined time interval by the imaging unit in the storage unit,
Based on a predetermined number of image information captured before the first and second image information is captured, a background image excluding the moving object image included in the predetermined number of image information is generated.
Based on the difference information between each of the predetermined number of image information and the background image, the area information indicating the range of the road area included in the first and second image information and the angle range information are generated.
A plurality of first feature points are extracted from the first image information, and a plurality of second feature points are extracted from the second image information.
A plurality of movement vectors having each of the plurality of first feature points as a start point and the plurality of second feature points corresponding to each of the plurality of first feature points as end points are calculated.
From the plurality of movement vectors, a specific movement vector whose start point or end point is included in the area information and whose direction is included in the angle range information is extracted.
When the specific movement vector is extracted, the speed of the vehicle traveling in the road area is calculated based on the specific movement vector, and when the specific movement vector is not extracted, the first image information and the said Based on the difference information between the second image information and the background image, it is determined whether or not there is a vehicle that is not moving in the road area .
Output the calculated speed or the result of the determination.
A speed estimation method characterized by including the fact that. To a computer that is connected to the photographing unit and has a storage unit
The first image information and the second image information captured by the photographing unit at predetermined time intervals, the area information indicating the range of the road area included in the first and second image information, and the angle range information. Is stored in the storage unit,
Based on a predetermined number of image information captured before the first and second image information is captured, a background image excluding the moving object image included in the predetermined number of image information is generated.
Based on the difference information between each of the predetermined number of image information and the background image, the area information indicating the range of the road area included in the first and second image information and the angle range information are generated.
A plurality of first feature points are extracted from the first image information, and a plurality of second feature points are extracted from the second image information.
A plurality of movement vectors having each of the plurality of first feature points as a start point and the plurality of second feature points corresponding to each of the plurality of first feature points as end points are calculated.
From the plurality of movement vectors, a specific movement vector whose start point or end point is included in the area information and whose direction is included in the angle range information is extracted.
When the specific movement vector is extracted, the speed of the vehicle traveling in the road area is calculated based on the specific movement vector, and when the specific movement vector is not extracted, the first image information and the said Based on the difference information between the second image information and the background image, it is determined whether or not there is a vehicle that is not moving in the road area .
A speed estimation program for executing the output of the calculated speed or the result of the determination.

Images