JP7169894B2 - Image processing method, image processing device, and object recognition method - Google Patents

Description

The present invention relates to an image processing method, an image processing apparatus, and an object recognition method.

2. Description of the Related Art An image processing method for synthesizing a plurality of photographed frames is used in order to improve the image quality of image data composed of a plurality of consecutively photographed photographed frames. For example, in Japanese Patent Application Laid-Open No. 2002-200315, when generating a composite image based on the amount of change in the composite image, the number of composite images is set according to the size of the image. If the size of the image is large, the resources for generating the composite image can be reduced by reducing the number of images to be composited.

JP-A-2006-92450

However, when a composite image is generated by simply overlapping the photographed frames, the image quality is improved only in the non-moving parts of the observation target, which changes shape continuously due to movement, such as a walking human being. Moving parts are blurry.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing method, an image processing apparatus, and an object recognition method using the results of image processing that improve the image quality of an entire moving observation target.

An image processing method, an image processing apparatus, and an object recognition method according to an aspect of the present invention select a photographed frame in which an observation target has the same shape by measuring the movement of a plurality of measurement points in an image of image data, The gist is to generate a composite image by performing addition processing on these.

Advantageous Effects of Invention According to the present invention, it is possible to provide an image processing method, an image processing apparatus, and an object recognition method using the result of image processing that improve the image quality of the entire moving observation target.

1 is a schematic diagram showing the configuration of an image processing apparatus according to a first embodiment of the present invention; FIG. 1 is a graph showing human optical flow during walking; 1 is a graph showing the horizontal optical flow of a human foot during walking; 1 is a graph showing the vertical optical flow of a human foot during walking; Fig. 4 is a graph showing the horizontal optical flow of a human knee during walking; Fig. 4 is a graph showing the horizontal optical flow of a human hand during walking; Fig. 3 is a graph showing the vertical optical flow of the human head during walking; It is a schematic diagram which shows the synthesized image of a comparative example. 4 is a flow chart for explaining an image processing method according to the first embodiment of the present invention; FIG. 4 is a schematic diagram for explaining a method of measuring optical flow in the image processing method according to the first embodiment of the present invention (Part 1); FIG. 2 is a schematic diagram for explaining an optical flow measurement method in the image processing method according to the first embodiment of the present invention (No. 2); FIG. 4 is a schematic diagram showing an example of a processing result of the image processing method according to the first embodiment of the present invention; FIG. 5 is a schematic diagram showing the configuration of an image processing apparatus according to a second embodiment of the present invention; FIG. 11 is a schematic diagram showing the configuration of an image processing apparatus according to a third embodiment of the present invention;

Embodiments will be described below with reference to the drawings. In the description of the drawings, the same parts are denoted by the same reference numerals, and the description thereof is omitted.

(First embodiment)
The image processing apparatus 20 according to the first embodiment of the present invention shown in FIG. Image processing is performed for The imaging device 10 is, for example, a camera mounted on an automobile, a security camera, or the like. Image data captured by the imaging device 10 is transmitted to the image processing device 20 .

The image processing device 20 includes a measuring section 21 , a periodicity determining section 22 , a phase determining section 23 , an image storing section 24 and an image synthesizing section 25 . The image processing apparatus 20 further includes a recording device 26 having a measurement result recording device 261 and an image recording device 262 . Note that FIG. 1 shows an image data processing system including an imaging device 10, an image processing device 20, an image display device 30, and an object recognition device 40. As shown in FIG.

The measurement unit 21 measures movement over time at a plurality of measurement points set in an image of image data. For example, the motion of the measurement point is measured by comparing the images between the captured frames captured in succession in time. The measurement results are stored in the measurement result recording device 261 . For example, the measuring unit 21 measures an optical flow indicating motion of each part of the image of the image data.

The periodicity determination unit 22 determines whether or not there is periodicity in the movement of each of the measurement points. A result of the determination by the periodicity determination unit 22 is output to the phase determination unit 23 .

The phase determination unit 23 selects one of the measurement points determined to have periodicity in motion (hereinafter referred to as “periodic measurement points”) as the target measurement point Ps, and determines the motion of the target measurement point Ps. One of the phases of the operating cycle, which is the cycle, is set as the reference phase Ts. Further, the phase determination unit 23 determines whether each of the captured frames forming the image data is a reference phase frame captured at a time corresponding to the reference phase Ts, and extracts a plurality of reference phase frames.

The image storage unit 24 cuts out image regions around the target measurement point Ps from each of the plurality of reference phase frames. Images of image regions cropped from a plurality of reference phase frames are stored in image recorder 262 .

The image synthesizing unit 25 adds the image areas of the plurality of reference phase frames stored in the image recording device 262 to generate a synthesized image.

Note that the image area cut out from the reference phase frame is a certain range that includes the target measurement point Ps. Thereby, the image region is cropped from the reference phase frame to include the object having a portion that moves with the motion cycle. The range of this image area is set according to the distance from the imaging device 10 to the target measurement point Ps and the size of the observation target. For example, when a person is assumed as an observation target, the size of the person is identified using the distance from the imaging device 10, and the range of the image area is set so as to include the entire observation target.

A case where the measurement unit 21 measures the optical flow will be exemplified below. Movements of human feet, hands, knees, and head during walking have periodicity. For this reason, for example, as in the optical flow shown in FIG. 2, the leg position measurement point P1, the knee position measurement point P2, the hand position measurement point P3, and the head position measurement point P4 are periodic measurement points. be. The horizontal axis of FIG. 2 is the time, and the vertical axis is the velocity (flow velocity) of the measurement point. 3 to 7 show examples of optical flows at measurement points P1 to P4.

FIG. 3 shows an example of the horizontal optical flow of the image of the foot measurement point P1, and FIG. 4 shows an example of the vertical optical flow of the image of the measurement point P1. After landing, the foot stays on the ground for a while and then moves upwards. Therefore, the velocity at the moment the foot lands is zero. The speed at the measurement point P1 is maximum at the moment the foot touches the ground, and is minimum at the moment before the foot touches the ground.

FIG. 5 shows an example of the horizontal optical flow at the knee measurement point P2. The optical flow at the measurement point P2 has a first maximum value V1, which is the moment of maximum velocity when the knee connected to the leg moves, and a second maximum value V2, when the opposite leg moves.

FIG. 6 shows an example of the horizontal optical flow at the hand measurement point P3. The speed is maximum at the moment when the hand passes the lowest point.

FIG. 7 shows an example of the vertical optical flow at the measurement point P4 on the head. The maximum speed is the moment the knee is extended, and the minimum speed is the moment the knee is bent. It should be noted that the head and torso of a walking human being move in the same manner as the left and right movements of the hands and feet. Therefore, the period of movement of the hands and feet is twice the period of movement of the head.

If a composite image is generated by simply adding images of a walking human, a high-resolution image can be obtained for a portion that is not moving, as in the comparative example shown in FIG. Blurry. On the other hand, according to the image processing method according to the first embodiment, a high-resolution image including moving parts can be obtained.

The image processing method by the image processing apparatus 20 shown in FIG. 1 will be described below with reference to the flowchart shown in FIG. In addition, below, the case where a person who is walking is set as an observation target will be described as an example.

In step S<b>10 in FIG. 9 , image data to be processed captured by the imaging device 10 is transmitted to the image processing device 20 . Thereafter, image processing is performed on the image data in step S20.

In step S<b>21 , the measurement unit 21 measures motion at a plurality of measurement points of the image data using a plurality of captured frames sent from the imaging device 10 . For example, the measurement unit 21 measures an optical flow that indicates the motion of each part of the image of the captured frame. Optical flow measurement is performed by extracting feature points that can be easily tracked, such as corners and points of an image included in a photographed frame, as optical flow measurement points, and tracking the movement of the measurement points over a plurality of photographed frames.

Tracking of measurement points between a certain shooting frame (hereinafter referred to as "original shooting frame") and a shooting frame shot next to the original shooting frame (hereinafter referred to as "next shooting frame") , for example, as follows: First, the measurement unit 21 cuts out an image range having a size including a predetermined number of pixels from the original captured frame, centering on the measurement point. Then, in the next photographed frame, the measuring unit 21 calculates a plurality of pixels included in a predetermined search range centered on the position of the measurement point of the original photographed frame, and the image range of the same size as the original photographed frame. Crop a small image area. Next, the measurement unit 21 correlates each of a plurality of image ranges cut out from the next captured frame with the image range cut out from the original captured frame, and selects the center of the image range with the highest correlation for the next captured frame. Coordinates of the measurement point in the frame. The movement of the measurement points between the captured frames obtained in this manner is referred to as an optical flow.

For example, as shown in FIG. 10, the measurement unit 21 cuts out an image range A from the original captured frame so that the measurement point P is located at the central pixel S. Then, the measurement unit 21 sets nine pixels including the pixel S and pixels adjacent to the pixel S as the search range B in the next imaging frame, as shown in FIG. 11 . Next, the measurement unit 21 cuts out nine image ranges having the same size as the image range A centering on each pixel included in the search range B from the next captured frame. Then, the measurement unit 21 selects an image range having the largest correlation with the image range A from among the nine image ranges cut out from the next captured frame, and places the center of the selected image range in the next captured frame. Coordinates of the measurement point.

The measurement unit 21 outputs to the periodicity determination unit 22 the history of the optical flow of the measurement points from the time when the image processing was started to the set past. The set period up to the past is arbitrary, but at least a period of time that allows confirmation of the periodicity of the movement of the assumed observation target is set.

In step S22, the periodicity determination unit 22 determines the periodicity of the movements of all measurement points from the optical flow history, and selects measurement points having periodicity. For example, when the observation target is a walking human, the human walks by moving the left and right limbs alternately, so that the movement is cyclically and continuously deformed. For this reason, a subject such as a car that has high rigidity and does not deform can be distinguished from a human by its movement. The determination of the presence or absence of periodicity is performed, for example, as follows.

The periodicity determination unit 22 Fourier-transforms the history of the horizontal or vertical optical flow input from the measurement unit 21 for the measurement points. Then, regarding the frequency of the maximum output excluding the DC component, if the output is equal to or greater than a predetermined threshold value, the measuring point is determined to be the period measuring point, and that frequency is defined as the reference frequency of the period measuring point. . The periodicity determination unit 22 outputs the period measurement points and their reference frequencies to the phase determination unit 23 . It should be noted that the threshold value of the output for judging that it is a periodic measurement point can be set based on the expected movement of the observation target.

In step S23, the phase determination unit 23 selects one of the periodicity measurement points determined to have periodicity by the periodicity determination unit 22 as the target measurement point Ps. Furthermore, the phase determination unit 23 sets one of the phases in the operation period, which is the period of movement of the target measurement point Ps, as the reference phase Ts. Then, in step S24, the phase determination unit 23 determines whether or not each of the captured frames forming the image data is captured at a time corresponding to the reference phase Ts. In the following description, the frames captured at the reference phase Ts are referred to as "reference phase frames", and the set of reference phase frames is referred to as "reference phase frame group Fs". That is, the reference phase frame group Fs is a set of photographic frames photographed at times corresponding to the reference phase Ts in each cycle when the photographic frames forming the image data are divided into a plurality of cycles in units of operation cycles. be.

Since the motion of the observed object with the period measuring point as a part is repeated, at the same phase of the working cycle, the period measuring point is at the same position and the observed objects are all the same shape. In other words, the reference phase frame group Fs is a set of captured frames having the same shape of the observation target.

For example, the phase determination unit 23 selects the measurement point closest to the center of the image as the target measurement point Ps from among the period measurement points having the frequency corresponding to the operation period as the reference frequency. When the object of observation is a walking human, the reference frequency is about 2 to 3 Hz. Then, the phase determination unit 23 sets the reference phase Ts in the first operation cycle, which is the latest operation cycle in the history of the optical flow, for example, based on the temporal change of the optical flow at the target measurement point Ps. As the reference phase Ts, for example, a phase corresponding to the time when the magnitude of the horizontal velocity of the target measurement point Ps is maximum is set. Then, the phase determination unit 23 determines whether or not the phase corresponding to the time when each captured frame was captured is the same as the reference phase Ts in the second and subsequent operation cycles in the optical flow history. The phase determination unit 23 outputs to the image storage unit 24 the reference phase frame determined to have the same phase as the reference phase Ts corresponding to the time when the image was taken.

In step S25, the image storage unit 24 cuts out an image area around the target measurement point Ps from the reference phase frame so as to include an observation target having other measurement points that move in the operation cycle, and stores an image of this image area. Stored in the image recording device 262 . For example, the image recording device 262 stores an image of an image area that includes all period measurement points that are within a predetermined distance from the target measurement point Ps and have the same reference frequency as the target measurement point Ps. The size of the image area corresponds to the distance from the imaging device 10 to the target measurement point Ps and the size of the assumed observation target.

In step S26, the image synthesizing unit 25 adds the image regions of the plurality of reference phase frames stored in the image recording device 262 to generate a synthesized image. A high-quality composite image can be obtained by performing addition processing on images having the same shape of the observation target. By using the high-quality composite image generated by the image processing device, for example, it is possible to improve the accuracy of object recognition for the observation target. For object recognition, the synthetic image may be magnified by a predetermined factor. It should be noted that addition averaging processing may be performed on the image as addition processing.

Note that the image area stored in the image recording device 262 may also include periodic measurement points with a reference frequency that is twice the reference frequency of the target measurement point Ps. For example, the period of movement of the human head during walking is half the period of movement of the hands and feet. Therefore, when the measurement point of interest Ps is a human hand or foot, the head can be included in the image area by including the periodic measurement points with twice the reference frequency.

Also, in order to ensure that the entire observation target is included in the image area, a margin may be given to the range of the image area as follows. That is, the image recording device 262 may store an image of an image area that is expanded by a certain length in the vertical and horizontal directions outside the minimum area that includes all the target periodic measurement points.

It should be noted that an image for which the image quality of image synthesis is to be lowered may be excluded from the target of image synthesis. For example, an image of an image region cropped from a previously captured reference phase frame is correlated with an image region of an image region cropped from a later captured reference phase frame. That is, the correlation between the image of the reference phase frame previously stored in the image recording device 262 and the image of the reference phase frame to be stored later is obtained. This image correlation is performed on overlapping portions by aligning the positions of the target measurement points Ps in the two images. If the correlation value of the image is equal to or less than a predetermined threshold value, the image of the reference phase frame to be stored is not stored in the image recording device 262, and the image is not used for image synthesis. For example, when the image of the reference phase frame of the first operation cycle is stored in the image recording device 262, the correlation between the image stored in the image recording device 262 and the image of the reference phase frame of the second and subsequent operation cycles is obtained.

The correlation value is less than or equal to the threshold when the shape of the observation target is greatly deformed, and when the shape of the observation target is greatly changed between images, the image quality of the synthesized image is degraded. By not storing in the image recording device 262 an image whose correlation value is equal to or less than a predetermined threshold value, it is possible to prevent deterioration of the image quality of an image obtained by synthesizing observation target images. Therefore, the image stored in the image recording device 262 is initialized.

Note that if an image area that is not used for image synthesis continues two or more times, the target measurement point Ps and the reference phase Ts corresponding to the image area may be excluded from the targets used for image synthesis. This can prevent generation of a composite image with poor image quality.

By the way, if the image is simply added and enlarged, it is difficult to reproduce the sub-pixel unit information buried in the pixel in the generated synthesized image. Therefore, in order to add the second and subsequent images to the first enlarged image of the reference phase Ts, the amount of change in the position of the observation target is calculated from the history of the movement of the target measurement point Ps, that is, the history of the optical flow. calculate. Then, the positions of the images of the second and subsequent reference phases Ts are sequentially moved by the amount of positional change, and the images are added. In this way, the image stored in the image recording device 262 is enlarged and a high-quality composite image in which information in sub-pixel units is reproduced is generated for the observation target by addition processing of the images corrected for the positional change amount of the observation target. be done. The image combining unit 25 outputs the generated combined image to the image display device 30 and the object recognition device 40 .

In steps S30 and S40, object recognition is performed using the processing results of the image processing method described above.

In step S<b>30 , the image display device 30 displays the high-quality synthesized image output from the image synthesizing section 25 . The image display device 30 is, for example, a high resolution monitor. For example, the image display device 30 may display a synthesized image enlarged by a predetermined multiple.

In step S40, the object recognition device 40 receives the high-quality synthesized image output from the image synthesizing unit 25, and recognizes an object to be observed included in the synthesized image. The object recognition device 40 uses artificial intelligence learned by machine learning, for example, and refers to a learning database in which various shape data about the observation target is stored to perform object recognition of the observation target.

As described above, in the image processing method according to the first embodiment, periodic measurement points are extracted with respect to the optical flow indicating the movement of the measurement points between the captured frames. Then, one time in the operation cycle of the target measurement point Ps selected from the periodic measurement points is set as the reference phase Ts, and the captured frames captured at the time corresponding to the reference phase Ts are extracted. As a result, a reference phase frame group Fs in which the observation target has the same shape is obtained. As shown in FIG. 12, a high-quality composite image is generated by performing addition processing on the image regions of the reference phase frame group Fs.

Therefore, according to the image processing method according to the first embodiment, a high-quality image can be obtained even for a moving observation target. By using this high-quality composite image, it is easier to recognize the object to be observed, and the accuracy is improved, compared to using an image that has not undergone image processing. The outline of the shape of the observed object is clear when the synthesized image is enlarged.

Observation targets can be selected arbitrarily. For example, when a walking human being is to be observed, a periodic measurement point whose movement in the vertical or horizontal direction has periodicity and whose reference frequency is about 2 to 3 Hz is selected as the target measurement point Ps. Note that the reference frequency can be changed as appropriate according to the object of observation. A point is selected as the target measurement point Ps. At this time, a periodic measurement point that moves at a cycle that is a multiple of the operation period of the target measurement point Ps may also be included in the observation target. As a result, many periodic measurement points can be included in the image area for an observation target in which all the periodic measurement points do not have the same reference frequency. For example, in an observation target assumed to be a human being, when the hand or foot is set as the target measurement point Ps, the image area can include the head that moves twice as much as the movement period of the hand or foot.

The time when the speed of movement of the measurement point is maximum or minimum may be used as the reference phase Ts of the target measurement point Ps at which the periodic movement of a person appears during walking. Alternatively, the time at which the movement of the target measurement point Ps changes from zero, that is, the time at which the target measurement point Ps starts moving from a stationary state may be used as the reference phase Ts. As a result, the time that can be easily extracted from the measured optical flow becomes the reference phase Ts, and it is possible to easily select the phase in which the observation target has the same shape. Further, when the knee of a walking human being is set as the target measurement point Ps, the time when the speed reaches the second maximum value in the operation cycle may be set as the reference phase Ts.

By the way, for a person far away from the imaging device 10, the movement of the head is more likely to be observed than the movement of the hands and feet. Therefore, there is a high possibility that the head will be selected as the target measurement point Ps. The period of a portion of the human head or body that moves in the same manner with respect to the left and right movements of the limbs is half the period of the movements of the hands and feet. Therefore, when the head is selected as the target measurement point Ps, by selecting one shooting frame during a cycle twice as long as the operation cycle of the target measurement point Ps, the observation target can be photographed in the same shape. Frames can be extracted. Therefore, a high-quality composite image can be generated by performing addition processing on every other image area of the reference phase frame in order of shooting time. For example, addition processing is performed on the image area of only the reference phase frame for the even-numbered or odd-numbered reference phase Ts. This is equivalent to resetting the reference phase Ts with a period that is twice the operating period as one period. As a result, when a periodic measurement point having a period half the actual period of the movement of the observation target is selected as the target measurement point Ps, a high-quality synthetic image can be generated.

For example, at a long distance from the imaging device 10, the image data has almost no features of a human shape. Therefore, when a small number of periodic measurement points are detected, there is a high possibility that the target measurement point Ps is the head or body. Therefore, when the number of measurement points in the motion cycle included in the image area is less than a predetermined set value, one reference phase is set for a cycle twice the motion cycle, assuming that the target measurement point Ps is the head or body. is set again, and it is determined whether or not each photographed frame is a reference phase frame. This set value can be arbitrarily set, but if there are about 1 or 2 periodic measurement points, it is estimated that the head or body is highly likely. Therefore, for example, the set value may be 2-3.

The image processing method and the object recognition method according to the first embodiment can be used, for example, to capture an image captured by an imaging device 10 mounted on a mobile object such as an in-vehicle camera, or to capture an image captured by a fixed imaging device 10 such as a security camera. It can be used for image processing such as

(Second embodiment)
In the image processing apparatus 20 according to the second embodiment shown in FIG. 13, the phase determination section 23 sets a plurality of reference phases Ts for the operation period of one target measurement point Ps. Then, the phase determination unit 23 determines whether or not each captured frame was captured at a time corresponding to the reference phase Ts for each of the plurality of reference phases Ts. That is, the fact that a plurality of reference phases Ts are set for one target measurement point Ps is different from the first embodiment in which one reference phase Ts is set for one target measurement point Ps.

For example, the phase determination unit 23 sets a plurality of reference phases Ts from the optical flow in the first operation cycle. A case where three reference phases Ts are set will be exemplified below. For example, the phase at which the speed of horizontal movement of the target measurement point Ps is minimum is set as the reference phase Ts1. Further, the phase at which the motion speed is maximum is defined as a reference phase Ts2, and the phase at which the motion speed changes from zero is defined as a reference phase Ts3.

The phase determination unit 23 determines whether or not each captured frame was captured at a time corresponding to one of the set reference phases Ts1, Ts2, and Ts3. Then, the phase determination unit 23 determines the first reference phase frame group Ft1 set up at the time corresponding to the reference phase Ts1, the second reference phase frame group Ft2 captured at the time corresponding to the reference phase Ts2, and the reference phase A third reference phase frame group Ft3 captured at a time corresponding to Ts3 is output to the image storage unit 24 .

At this time, the phase determination unit 23 converts the frame information about the first reference phase frame group Ft1, the second reference phase frame group Ft2, and the third reference phase frame group Ft3 to the reference phase Ts from which each reference phase frame group was judged. , and output to the image storage unit 24 . The frame information is information indicating at what time in the image data each photographed frame was photographed, and is, for example, a tag in which a frame number assigned to the photographed frame is recorded.

The image storage unit 24 stores the images of the first reference phase frame group Ft1, the second reference phase frame group Ft2, and the third reference phase frame group Ft3 in the first phase image recording device 262_T1 of the image processing device 20 shown in FIG. , the second phase image recording device 262_T2, and the third phase image recording device 262_T3, respectively. The method of selecting images to be stored in the image recording apparatus and the determination using correlation are the same as in the first embodiment. For example, when the correlation value of the image of the reference phase frame before and after is equal to or less than the threshold value, the image stored in the image recording device is initialized.

The image synthesizing unit 25 synthesizes the images stored in the image recording device to generate a synthesized image, as in the first embodiment. At this time, the image regions of the reference phase frame group are added for each reference phase Ts to generate a plurality of synthesized images. That is, the images stored in each of the first phase image recording device 262_T1, the second phase image recording device 262_T2, and the third phase image recording device 262_T3 are synthesized in the same manner as in the first embodiment, and generates a composite image of As a result, three synthesized images with different shapes of observation targets are generated.

In the image processing method according to the second embodiment, a high-quality composite image is generated for each of a plurality of reference phases Ts set for one target measurement point Ps. Therefore, compared to the case where one reference phase Ts is set for one target measurement point Ps, the interval at which the synthesized image is generated becomes shorter.

Therefore, the image processing method according to the second embodiment is particularly effective when the movement period of the observation target is long, or when object recognition is performed at shorter intervals. For example, when the walking cycle of a person is slow compared to the imaging rate of the imaging device 10, if image synthesis is performed based on only one reference phase Ts, the object recognition device 40 will receive the images at intervals shorter than the imaging rate. Only synthetic images are input. As a result, the interval at which object recognition is performed becomes longer. For this reason, there is a possibility that there is not enough time to reflect the result of object recognition in the control of an automobile, for example. In contrast, in the second embodiment, by setting a plurality of reference phases Ts for the target measurement point Ps, the time interval for object recognition can be shortened.

As described above, according to the image processing method according to the second embodiment, similarly to the first embodiment, it is possible to generate a high-quality composite image of a moving observation target. Furthermore, according to the second embodiment, since a high-quality composite image is generated for each of a plurality of reference phases Ts, composite images can be generated at time intervals shorter than the operation cycle of the target measurement point Ps. By performing object recognition using synthesized images generated at short time intervals, it is also possible to acquire information on distant objects with high frequency. Others are substantially the same as those of the first embodiment, and redundant description is omitted.

(Third embodiment)
In the image processing apparatus 20 according to the third embodiment shown in FIG. 14, the phase determination section 23 selects a plurality of measurement points of interest Ps having the same operation cycle. As a result, a plurality of periodic portions are selected for the observation target. Then, the phase determination unit 23 sets different phases as the reference phases Ts for the plurality of target measurement points Ps. Since the reference phases Ts are different from each other, the shape of the observation target at the time corresponding to each reference phase Ts is different. A plurality of measurement points of interest Ps are selected, and a reference phase Ts is set for each of the measurement points of interest Ps, which is different from the first embodiment in which one reference phase Ts is set for one measurement point of interest Ps. It is a point.

For example, when the assumed observation target is a human, the phase determination unit 23 selects a plurality of target measurement points Ps from the period measurement points of the reference frequency of about 2 to 3 Hz, which is the period of the movement of the hands and feet when the human walks. select. A case in which three measurement points of interest Ps are selected will be exemplified below. For example, the phase determination unit 23 selects the periodic measurement point closest to the center of the image as the first target measurement point Ps1. Further, the phase determination section 23 selects a second measuring point of interest Ps2 and a third measuring point of interest Ps3 that are separated from the first measuring point of interest Ps1 by a certain amount. The mutual distances of the first measurement point of interest Ps1, the second measurement point of interest Ps2, and the third measurement point of interest Ps3 are set according to the range of the image including the observation target. That is, the mutual distance range of the selected plurality of target measurement points Ps is set according to the distance from the imaging device 10 to the observation target and the size of the assumed observation target.

The phase determination unit 23 sets reference phases Tp1, Tp2, and Tp3 of the first measurement point of interest Ps1, the second measurement point of interest Ps2, and the third measurement point of interest Ps3, respectively, from the optical flow in the first operation cycle. For example, the phases at which the speed of horizontal motion is minimum are set as reference phases Tp1, Tp2, and Tp3. Furthermore, the phase determination unit 23 determines whether or not each captured frame was captured at a time corresponding to one of the reference phases Tp1, Tp2, and Tp3. Then, the phase determination unit 23 selects the first reference phase frame group Fp1, the second reference phase frame group Fp2, and the third reference phase frame group Fp3 captured at times corresponding to the reference phases Tp1, Tp2, and Tp3, respectively. It is output to the image storage unit 24 in association with the frame information and the information of the target measurement point Ps.

The image storage unit 24 stores the image of the first reference phase frame group Fp1, the image of the second reference phase frame group Fp2, and the image of the third reference phase frame group Fp3 in the first measurement of the image processing device 20 shown in FIG. They are stored in the point image recording device 262_P1, the second measurement point image recording device 262_P2, and the third measurement point image recording device 262_P3, respectively. The method of selecting images to be stored in the image recording apparatus and the determination using correlation are the same as in the first embodiment.

The image synthesizing unit 25 adds image regions of the reference phase frame group Fs for each target measurement point Ps to generate a plurality of synthesized images. That is, the image synthesizing unit 25 synthesizes the stored images for each of the first measurement point image recording device 262_P1, the second measurement point image recording device 262_P2, and the third measurement point image recording device 262_P3 in the same manner as in the first embodiment. to generate a composite image. Since a different reference phase Ts is set for each target measurement point Ps, a plurality of synthesized images with different shapes of observation targets are generated.

According to the image processing method according to the third embodiment, similarly to the first embodiment, it is possible to generate a high-quality composite image of a moving observation target. Furthermore, in the image processing method according to the third embodiment, a high-quality composite image is generated for each of the plurality of target measurement points Ps. Therefore, compared to the case where one reference phase Ts is set for one target measurement point Ps, the interval at which the synthesized image is generated becomes shorter. That is, by performing object recognition using high-quality composite images generated at short time intervals, it is possible to acquire information on distant objects with high frequency.

Therefore, the image processing method according to the third embodiment is particularly effective when the movement period of the observation target is long or when object recognition is performed at shorter intervals. Others are substantially the same as those of the first and second embodiments, and redundant description is omitted.

(Fourth embodiment)
According to the image processing method described above, a high-resolution composite image can be generated by performing addition processing on the image regions of the reference phase frame group Fs. Further, by subjecting the image areas of the reference phase frame group Fs to addition processing, it is possible to generate a composite image with high gradation.

That is, the image synthesizing unit 25 can increase the gradation of image data obtained by digitizing a captured image by analog-to-digital (AD) conversion by the imaging device 10 . For example, the image synthesizing unit 25 uses the image stored in the image recording device 262 to align the position of the target measurement point Ps in the image of the first operation cycle with the position of the target measurement point Ps of the second and subsequent operation cycles. Addition processing of the image area is performed while shifting the position of the image as shown. For the image position shift amount, for example, the observation target position change amount calculated from the history of the optical flow of the target measurement point Ps is used. A high gradation image can be obtained by correcting the displacement of the images stored in the image recording device 262 and performing addition processing. For example, when the imaging device 10 outputs 8-bit data for each pixel by AD conversion, a synthesized image restored to 16 bits is generated. This increases the contrast of the composite image and reduces noise for observation objects that are moving repetitively.

The image processing apparatus according to the fourth embodiment is particularly effective when the contrast of the image is low and the luminance difference between pixels is hidden by the AD conversion of the imaging apparatus 10 . For example, in the evening when the surroundings are dark, the contrast between the observation target and its surroundings is small, and the observation target cannot be seen clearly, which may make object recognition difficult. In this regard, according to the image processing apparatus according to the fourth embodiment, by generating a high-gradation image, the difference in contrast between the object to be observed and its surroundings is clarified, and the object to be observed can be seen clearly. be able to.

(Other embodiments)
While embodiments of the present invention have been described above, the discussion and drawings forming part of this disclosure should not be construed as limiting the invention. Various alternative embodiments, implementations and operational techniques will become apparent to those skilled in the art from this disclosure.

For example, in the above description, a case where a walking human being is assumed as an observation target is explained, but the observation target is not limited to this, and the observation target can be a human riding a bicycle or an animal other than a human. It could be.

In addition, the period of a portion of the observation target having periodic motion may not only be twice the period of the target measurement point Ps, but may be three times or more. For this reason, the image area may be set so as to include periodic measurement points that move not only with a double cycle but also with a multiplication cycle.

DESCRIPTION OF SYMBOLS 10... Imaging device 20... Image processing apparatus 21... Measurement part 22... Periodicity determination part 23... Phase determination part 24... Image storage part 25... Image synthesizing part 26... Recording device 30... Image display device 40... Object recognition device 261... Measurement result recording device 262... Image recording device

Claims (16)

In an image processing method for processing image data composed of a plurality of captured frames captured continuously over time,
measuring the movement of a plurality of measurement points in the image of the image data;
Determining whether the movement measured for each of the measurement points has periodicity,
One of the measurement points determined to have periodicity in the measured motion is selected as a target measurement point, and one phase of an operation cycle, which is the motion cycle of the target measurement point, is set as a reference phase. Set,
determining whether each of the captured frames is a reference phase frame captured at a time corresponding to the reference phase;
cutting out an image area around the measurement point of interest from the reference phase frame so as to include an observation target having the other measurement points that move in the motion cycle;
An image processing method comprising adding the image regions of a plurality of the reference phase frames to generate a synthesized image. setting a plurality of the reference phases for the operation period;
determining whether each of the captured frames is the reference phase frame captured at a time corresponding to one of the plurality of reference phases;
2. The image processing method according to claim 1, wherein the image regions of the reference phase frame are added for each of the reference phases to generate a plurality of the composite images. selecting a plurality of the target measurement points having the same operation cycle;
setting different phases as the reference phases for the plurality of measurement points of interest;
determining whether each of the captured frames is the reference phase frame captured at a time corresponding to one of the reference phases of the plurality of measurement points of interest;
2. The image processing method according to claim 1, wherein the image regions of the reference phase frame are added for each of the target measurement points to generate a plurality of the composite images. 2. The addition process of the image area is performed while the image of the image area is moved according to the position change amount of the observation target calculated based on the movement history of the target measurement point. 4. The image processing method according to any one of items 1 to 3. 5. The image processing method according to any one of claims 1 to 4, wherein the composite image is generated by increasing the gradation of the image data in the addition processing of the image areas. 6. The method according to any one of claims 1 to 5, wherein it is determined whether or not there is periodicity in the movement of said measurement points by using the movement of said measurement points in the vertical direction or the movement in the horizontal direction. The described image processing method. 7. The image processing method according to any one of claims 1 to 6, wherein the image area is set so as to include the measurement point that moves in a cycle that is a multiple of the operation cycle of the target measurement point. 8. The image processing method according to any one of claims 1 to 7, wherein the reference phase is set to a time when a speed of movement of the target measurement point is maximum or minimum. 8. The image processing method according to any one of claims 1 to 7, wherein the reference phase is a time when the speed of movement of the target measurement point changes from zero. 8. The image processing method according to any one of claims 1 to 7, wherein the reference phase is a time when the speed of movement of the target measurement point reaches a second maximum value in the operation cycle. 11. The image processing method according to any one of claims 1 to 10, wherein the image regions of the reference phase frames are added alternately in order of photographing time to generate the composite image. When the number of measurement points in the operation period included in the image area is smaller than a set value, one reference phase is set again for a period twice as large as the operation period, and for each of the photographing frames. 12. The image processing method according to any one of claims 1 to 11, wherein it is determined whether or not it is the reference phase frame. A correlation is obtained between an image of the image region cropped from the reference phase frame captured earlier and an image of the image region cropped from the reference phase frame captured later, and a correlation value is equal to or less than a predetermined threshold. if the image region cropped from the reference phase frame taken later is not used to generate the composite image,
13. The method according to any one of claims 1 to 12, wherein when the image area not used for generating the composite image continues twice or more, the target measurement point and the reference phase are excluded from the targets used for image synthesis. The image processing method according to any one of the items. Displaying the processing result of the image processing method according to any one of claims 1 to 13,
An object recognition method, wherein object recognition of the observation target is executed from the processing result using a learning database storing various shapes of the observation target. 15. The object recognition method according to claim 14, wherein the observation target is a walking human being. An image processing device for processing image data composed of a plurality of frames captured continuously over time, the image processing device comprising:
measuring the movement of a plurality of measurement points in the image of the image data;
Determining whether the movement measured for each of the measurement points has periodicity,
One of the measurement points determined to have periodicity in the measured motion is selected as a target measurement point, and one phase of an operation cycle, which is the motion cycle of the target measurement point, is set as a reference phase. Set,
determining whether each of the captured frames is a reference phase frame captured at a time corresponding to the reference phase;
cutting out an image area around the measurement point of interest from the reference phase frame so as to include an observation target having the other measurement points that move in the motion cycle;
An image processing apparatus that generates a composite image by adding the image regions of the plurality of reference phase frames.

Images