JP2001175872A - Device and method for image processing and recording medium with recorded software for image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide technology of image processing for measuring an optical flow with superior precision by optimizing an inputted image itself. SOLUTION: An optical flow field detection part 5 measures the optical flow according to the image inputted from an image pickup part 1. An optical flow field statistical processing part 6 performs statistical processing for the obtained optical flow field. An optimum frame rate calculation part 7 calculates an optimum frame rate to satisfy requested precision according to the result of the statistical processing by the processing part 6 and the requested precision inputted from a requested precision input part 3 if the optical flow does not reaches the requested precision. A frame rate determination part 8 determines a new frame rate according to the calculated optimum frame rate and requested frame rate. A frame rate control part 9 controls the image pickup part 1 according to the new frame rate.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement in a technique for measuring an optical flow from an image, and more particularly, to optimizing an input image itself.
The optical flow is measured with excellent accuracy.

[0002]

2. Description of the Related Art In recent years, with the development of digital technology, movements of surrounding objects are grasped from images taken by a video camera or the like for use in information processing such as a vehicle approach warning or automatic control for a vehicle. A technology to do this has been proposed. Such a technique is to grasp the movement of a captured object by measuring an optical flow between frames of a moving image, and is realized in the form of an image processing device or an image processing method.

Here, the optical flow is the speed at which each pixel moves on an image, and a vector representing this is called an optical flow vector or simply a flow vector. The field of the optical flow is called an optical flow field, and the field of the optical flow vector is called an optical flow vector field. Such an optical flow is obtained from each frame of a given image by a feature matching method (pattern matching), a spatiotemporal gradient method, or the like.

[0004] In such optical flow measurement, in order to improve the measurement accuracy of the optical flow of an object having a small relative velocity, conventionally, the reliability of the optical flow is evaluated, and based on the highly reliable optical flow, A method of improving the accuracy of the optical flow field by an interpolation method and a method of improving the reliability of a measurement result by statistical processing have been used. The interpolation method is
This is a method of estimating an intermediate value from a given value.

[0005]

However, the above-described method based on reliability is only a passive response to an input image, and information is obtained from a given input image with maximum accuracy. Was to draw. That is,
Since the input image information itself has not been improved at all, there is a limit in improving the measurement accuracy of the optical flow.

The present invention has been proposed in order to solve the above-mentioned problems of the prior art. The object of the present invention is to optimize an input image itself to realize an optical flow with excellent accuracy. It is an object of the present invention to provide an image processing technique to be measured, that is, an image processing apparatus and method, and a recording medium on which image processing software is recorded.

[0007]

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention comprises a means for inputting an image based on a frame rate, and an optical flow based on the input image. Means for measuring the optimal frame rate when the measured optical flow does not meet a predetermined criterion, based on the calculated optimal frame rate, Means for controlling the frame rate of a given image. According to a third aspect of the present invention, there is provided an image processing method according to the first aspect of the present invention, which includes a step of inputting an image based on a frame rate, and measuring an optical flow based on the input image. And calculating the optimal frame rate when the measured optical flow does not meet a predetermined criterion, based on the calculated optimal frame rate. Controlling the frame rate of an image. The invention of claim 5 captures the invention of claims 1 and 3 from the viewpoint of a recording medium on which computer software is recorded, and uses a computer from each frame of an image given based on a frame rate. In a recording medium recording image processing software for measuring an optical flow, the software causes the computer to measure an optical flow based on the input image, and the measured optical flow is determined in advance. When the criterion is not satisfied, an optimum frame rate is calculated, and the frame rate of a given image is controlled based on the calculated optimum frame rate. Claims 1, 3,
In the invention of the fifth aspect, the optimum frame rate is determined from the extracted optical flow vector field or the like, and the optimum imaging is performed by feeding back to the imaging unit. That is, when the optical flow represented by a flow vector or the like does not satisfy a predetermined criterion, the frame rate is reduced to increase the flow vector, and the measurement error in image quantization is reduced, so that the effective accuracy of the optical flow measurement is reduced. Improve. This enables more accurate information processing, for example, such as an improvement in collision prediction accuracy due to an improvement in relative speed measurement accuracy.

According to another aspect of the present invention, there is provided an image processing apparatus for sequentially capturing each frame of an image based on a frame rate, a unit for measuring an optical flow based on the captured image, and Means for inputting the required accuracy for the flow measurement, means for inputting the required frame rate, and when the measured optical flow is less than the required accuracy, the required accuracy is satisfied. Means for calculating an optimal frame rate; the calculated optimal frame rate; and the input required frame;
And means for determining a new frame rate based on the new frame rate, and means for controlling the imaging means based on the determined new frame rate. According to a fourth aspect of the present invention, there is provided an image processing method which captures the second aspect of the present invention from the viewpoint of a method, and includes an imaging step for sequentially capturing each frame of an image based on a frame rate, and an image processing method based on the captured image. Measuring the optical flow, inputting the required accuracy for the optical flow measurement, inputting the required frame rate, and measuring the measured optical flow to the required accuracy. If not, a new frame rate is calculated based on the steps for calculating an optimal frame rate to satisfy the required accuracy, the calculated optimal frame rate, and the input required frame. Determining a frame rate; and taking the image based on the determined new frame rate. A step for controlling the steps, characterized in that it comprises a. The invention of claim 6 captures the invention of claims 2 and 4 from the viewpoint of a recording medium on which computer software is recorded. From each frame of an image sequentially taken by the imaging unit based on the frame rate, Using a computer, in a recording medium on which image processing software for measuring an optical flow is recorded, the software causes the computer to measure an optical flow based on the captured image, and a request for measuring the optical flow is provided. When accepting the input of the accuracy, accept the input of the required frame rate, when the measured optical flow is less than the required accuracy,
An optimal frame rate for satisfying the required accuracy is calculated, and based on the calculated optimal frame rate and the input required frame, a new frame rate is determined and determined. The imaging unit is controlled based on the new frame rate. In the inventions of claims 2, 4, and 6,
Since the frame rate is controlled based on the required accuracy and the required frame rate in addition to the effects of 3 and 5, the optical flow measurement according to the situation such as the purpose of use or environmental change becomes easy.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention (referred to as embodiments) will be specifically described with reference to the drawings. It should be noted that, in the present embodiment, a portion related to data processing can be realized by controlling a computer by software, and the software in this case uses a conventional technology for a portion common to the conventional technology. The functions and effects of the present invention are realized by utilizing the physical hardware resources of the computer.

Further, in this case, various forms can be considered for the specific format of hardware and software, the range of processing by software, and the like, and a computer-readable recording medium on which such software is recorded is one aspect of the present invention. It is an aspect. Therefore, the present invention and the embodiments will be described below using virtual circuit blocks that realize the functions of the present invention and the embodiments.

[1. The present embodiment relates to an image processing apparatus (referred to as the present apparatus) for measuring an optical flow from a given image, and an image processing method executed on the present apparatus, and a recording medium on which image processing software is recorded. It can also be grasped as. First, as shown in the functional block diagram of FIG. 1, the present apparatus includes an imaging unit 1, an image processing system 2, a required accuracy input unit 3, and a required frame rate input unit 4, and performs optical flow measurement. Information such as a result is provided to the external information processing system 10.

The image pickup section 1 is an image pickup means for inputting by sequentially picking up each frame of an image based on a frame rate, and is realized by, for example, a video camera using a CCD. In addition, the frame rate in imaging can be controlled by a signal from outside the imaging unit 1.

The image processing system 2 is a control unit for performing data processing in the present apparatus, and based on the software, an optical flow field detection unit 5, an optical flow field statistical processing unit 6, an optimum frame It serves as a rate calculation unit 7, a frame rate determination unit 8, and a frame rate control unit 9.

Among them, the optical flow field detecting section 5 is a means for measuring an optical flow based on an input image. Specifically, the optical flow field detecting section 5 detects one or two or more optical flows. It is configured to measure and output as an optical flow field including a vector.

Further, in the present apparatus, the required accuracy, which is a predetermined reference, for measuring the optical flow,
The required frame rate (referred to as a required frame rate) can be designated by a user or another device inputting it from the outside. That is, the required accuracy input unit 3 is a unit for inputting the required accuracy, and the required frame rate input unit 4 is a unit for inputting the required frame rate.

An optical flow field statistical processing unit 6
Is, for example, a part for calculating the size of a flow vector having the largest size (called a maximum flow vector) or a flow vector to be focused (called a focused flow vector) by performing statistical processing of the obtained optical flow field. It is.

The optimum frame rate calculating section 7 calculates the optical flow based on the result of the statistical processing by the optical flow field statistical processing section 6 and the required precision input from the required precision input section 3. This is a means for determining whether or not the required accuracy is satisfied, and calculating an optimum frame rate (referred to as an optimum frame rate) for satisfying the required accuracy when the optical flow does not satisfy the required accuracy.

The frame rate determining section 8 and the frame rate control section 9 control the image pickup section 1 based on the optimum frame rate calculated by the optimum frame rate calculating section 7, so that the image processing system 2 Is a means for controlling the frame rate of the image given to the image.
Specifically, the frame rate determination unit 8 is a unit for determining a new frame rate based on the optimum frame rate calculated by the optimum frame rate calculation unit 7 and the required frame rate.

More specifically, in which case and how much priority is given between the optimum frame rate and the requested frame rate, a new frame rate is calculated based on the priorities. Should be determined. Further, the frame rate control unit 9 is means for controlling the imaging unit 1 based on the new frame rate determined by the frame rate determination unit 8.

Although the type and purpose of the external information processing system 10 are arbitrary, for example, a system that is mounted on a car and performs an approach warning of another car and automatic control of driving can be considered.

[2. Operation and Effect] In the present embodiment configured as described above, in the image processing system 2, the optimal frame rate is determined from the vector field of the extracted optical flow and the like, and the optimal frame rate is fed back to the imaging unit 1 to determine the optimal frame rate. This is for imaging. First, a processing procedure in the present embodiment is shown in FIG. In this figure,
Thick arrows indicate information such as an image originally processed by the apparatus and optical flows measured based on the image, and thin arrows indicate information related to frame rate control.

Note that the required accuracy and the required frame rate are input from the required accuracy input unit 3 and the required frame rate input unit 4 when the apparatus is started or when necessary thereafter. It is also conceivable to use the default value of.

In the procedure shown in FIG. 2, first, the imaging section 1 sequentially captures each frame of an image based on the frame rate and inputs the frame to the image processing system 2 (step 1). Then, in the image processing system 2, first, the optical flow field detection unit 5 measures the optical flow based on the input image (Step 2). Specifically, the optical flow field detection unit 5 measures and outputs an optical flow as an optical flow field including one or two or more optical flow vectors.

The optical flow field statistical processing unit 6
Performs the statistical processing of the obtained optical flow field (step 3) to obtain the maximum flow vector and the size of the flow vector of interest.

Further, the optimum frame rate calculating section 7 calculates the value of each vector of the optical flow based on the result of the statistical processing by the optical flow field statistical processing section 6 and the required precision inputted from the required precision input section 3. Evaluate accuracy. It is conceivable that the accuracy is evaluated based on, for example, a ratio of a maximum flow vector between each frame and a flow vector of interest that exceeds a predetermined reference value based on the required accuracy.

Then, the optimum frame rate calculator 7 calculates
It is determined whether or not the measured optical flows satisfy a predetermined value, here, required accuracy (step 4).
If the optical flow does not satisfy the required accuracy, an optimum frame rate for satisfying the required accuracy is calculated (step 5).

The frame rate determining section 8 and the frame rate control section 9 control the image pickup section 1 based on the optimum frame rate calculated by the optimum frame rate calculating section 7, so that the image processing system 2 To control the frame rate of the image provided to.

Specifically, the frame rate determining section 8
Based on the optimum frame rate calculated by the optimum frame rate calculation unit 7 and the requested frame rate,
A new frame rate is determined (step 6). Then, the frame rate control unit 9 controls the frame rate of the imaging unit 1 based on the new frame rate determined by the frame rate determination unit 8 (Step 7).

That is, when an optical flow represented by a flow vector or the like does not satisfy a predetermined criterion, a frame rate is reduced to increase a flow vector, and a measurement error in image quantization is reduced. Improve effective accuracy.

FIG. 3 shows optical flow vectors measured between frames of an image captured at a certain first frame rate before the control of the imaging unit 1 based on the new frame rate. It is a conceptual diagram. At the first frame rate, since the relative speed of the object O is small, the optical flow vector V31 between the frames F31 and F32 and the frames F32 and F33
Each of the intermediate optical flow vectors V32 is smaller in magnitude than a predetermined reference value based on the required accuracy.

In this case, for the same object O, after the control of the imaging unit 1 based on the new second frame rate,
FIG. 4 is a conceptual diagram showing optical flow vectors measured between frames of an image captured at the second frame rate. In this example, the second frame rate is lower than the first frame rate. That is, since the time interval between the frame F41 and the next frame F42 is larger than that in the example of FIG. 3, the optical flow vector V4 between the frames F41 and F42
1 becomes larger than a predetermined reference value based on the required accuracy, and the required accuracy is satisfied.

As a result, for example, more accurate information processing, such as an improvement in collision prediction accuracy due to an improvement in relative speed measurement accuracy, becomes possible.

In this embodiment, since the frame rate is controlled based on the required accuracy and the required frame rate, it is easy to measure the optical flow according to the situation such as the purpose of use and environmental changes.

As shown in FIG. 1, the optical flow field obtained by the optical flow field detection unit 5 is provided to an external information processing system 10 and used for various information processing such as relative speed measurement. Similarly, the result of the statistical processing can be provided from the optical flow field statistical processing unit 6 to the image processing system 2.

[3. Other Embodiments] The present invention is not limited to the above embodiments, but includes other embodiments as exemplified below. For example, the above-described embodiment is merely an example, and the specific numerical value and expression format of the frame rate, the type and format of the image, the expression format and measurement method of the optical flow, the content of the judgment criteria regarding the optical flow, and the calculation of the optimal frame rate Algorithm, a specific method for controlling the frame rate of an image given based on a new frame rate, and the like can be freely determined or changed.

For example, when changing the frame rate of a given image, in addition to changing the frame rate of the imaging unit itself such as a video camera, for example, by ignoring each frame of the input image for each frame, Alternatively, a control method such as halving the frame rate can be adopted.

[0037]

As described above, according to the present invention,
By optimizing the input image itself, it is possible to provide an image processing technique for measuring an optical flow with excellent accuracy, that is, an image processing apparatus and method, and a recording medium on which image processing software is recorded. The accuracy of information processing is also improved.

[Brief description of the drawings]

FIG. 1 is a functional block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a processing procedure in the embodiment of the present invention.

FIG. 3 illustrates optical flow vectors measured between frames of an image captured at a certain first frame rate before the control of the imaging unit based on a new frame rate in the embodiment of the present invention. Conceptual diagram.

FIG. 4 is a diagram illustrating an example in which an image of the same object is measured at a second frame rate between frames of an image captured at the second frame rate after control of the imaging unit based on the new second frame rate; The conceptual diagram showing the optical flow vector which was made.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 imaging unit 2 image processing system 3 required accuracy input unit 4 required frame rate input unit 5 optical flow field detection unit 6 optical flow field statistical processing unit 7 optimal frame rate calculation unit 8 frame rate determination unit 9: Frame rate control unit 10: Information processing system

Continued on the front page F-term (reference) 5C054 AA01 AA05 CA04 CC03 CH03 CH04 EA01 EJ00 FC13 FF02 HA05 5L096 BA02 BA04 CA02 CA14 HA04

Claims (6)

[Claims] A means for inputting an image based on a frame rate; a means for measuring an optical flow based on the input image; and a means for measuring the optical flow based on a predetermined reference. When less than, means for calculating an optimal frame rate; and means for controlling the frame rate of a given image based on the calculated optimal frame rate. Image processing apparatus. 2. An imaging unit for sequentially capturing each frame of an image based on a frame rate, a unit for measuring an optical flow based on the captured image, and a required accuracy for measuring the optical flow. Means for inputting, a means for inputting a required frame rate, and calculating an optimum frame rate for satisfying the required accuracy when the measured optical flow does not satisfy the required accuracy. Means for determining a new frame rate based on the calculated optimal frame rate and the input requested frame; and determining the new frame rate based on the determined new frame rate. Means for controlling the image pickup means based on the image processing means. A step of inputting an image based on a frame rate; a step of measuring an optical flow based on the input image; and a step of measuring the measured optical flow based on a predetermined reference. When less than, calculating an optimal frame rate; and, based on the calculated optimal frame rate, controlling the frame rate of a given image. Image processing method. 4. An imaging step for sequentially capturing each frame of an image based on a frame rate; a step for measuring an optical flow based on the captured image; and a precision required for measuring the optical flow. Inputting; inputting a required frame rate; calculating the optimal frame rate to satisfy the required accuracy when the measured optical flow does not meet the required accuracy. Determining a new frame rate based on the calculated optimal frame rate and the input required frame; and determining the new frame rate based on the determined new frame rate. Controlling the imaging step based on the Image processing how. 5. A recording medium on which image processing software for measuring an optical flow is recorded using a computer from each frame of an image given based on a frame rate, wherein the software is input to the computer. Measuring an optical flow based on the image, and when the measured optical flow does not satisfy a predetermined standard, calculate an optimal frame rate.Based on the calculated optimal frame rate, A recording medium on which image processing software is recorded, wherein the frame rate of a given image is controlled. 6. A recording medium recording image processing software for measuring an optical flow using a computer from each frame of an image sequentially captured based on a frame rate by an imaging unit, wherein the software is The computer is configured to measure an optical flow based on the captured image, to receive an input of required accuracy for the measurement of the optical flow, to receive an input of a required frame rate, and to determine the measured optical flow according to the request. When the accuracy is less than the predetermined frame rate, the optimum frame rate for satisfying the required accuracy is calculated. Based on the calculated optimum frame rate and the input required frame, a new frame rate is calculated. And the determined new frame Recording medium recording an image processing software, characterized in that to control the image pickup unit based on the rate.