JP7200002B2 - Image processing device, imaging device, image processing method, program, and storage medium - Google Patents

Description

The present invention relates to an image processing device that obtains the velocity of a subject, an imaging device that includes the image processing device, an image processing method, a program, and a storage medium.

An electronic camera has been proposed that detects the distance to a photographed subject and measures the actual moving speed of the subject corresponding to the subject image in the photographed image (Patent Document 1). Further, an imaging apparatus has been proposed that obtains the moving speed of a subject based on the distance to the subject detected by a distance sensor and the panning angle detected by a gyro sensor (Patent Document 2).

JP 2005-159674 A JP 2007-225550 A

Various objects can be subjects, and the environment in which the subjects are imaged is also diverse. Therefore, it may be difficult to accurately acquire the speed of the subject depending on the motion of the subject. For example, when the subject is moving toward the camera at a low speed, the accuracy of the obtained subject speed may be low because the subject moves a short distance between frames.

In view of the above circumstances, an object of the present invention is to provide an image processing apparatus capable of obtaining with high accuracy the velocity of a subject corresponding to a subject image in a captured image, an imaging apparatus provided with the image processing apparatus, an image processing method, a program, and a memory. It is to provide a medium.

To achieve the above object, the image processing apparatus of the present invention provides a tracking method that selects a plurality of corresponding images including a subject image corresponding to a subject image in a reference image included in a plurality of captured images from a plurality of the captured images. a distance acquisition unit configured to acquire, for each of the reference image and the corresponding image, distance information relating to a distance between an acquisition device that acquired the reference image or the corresponding image and a subject corresponding to the subject image; A position acquisition unit that acquires position information regarding the position of the subject for each of the image and the corresponding image, and acquires three-dimensional position information of the subject with reference to the reference image using the distance information and the position information. and a speed acquisition unit that acquires the speed of the subject corresponding to the subject image using the three-dimensional position information in the plurality of corresponding images selected based on the reliability of each of the corresponding images.

According to the present invention, the velocity of the subject corresponding to the subject image in the captured image can be obtained with high accuracy.

1 is a block diagram of an imaging device including an image processing device according to an embodiment of the present invention; FIG. 1 is an explanatory diagram of an imaging device according to an embodiment of the present invention; FIG. 4 is a flowchart showing speed acquisition processing according to the embodiment of the present invention; FIG. 4 is an explanatory diagram relating to an image displayed on the display unit according to the embodiment of the present invention; FIG. 4 is an explanatory diagram of three-dimensional position reconstruction according to the embodiment of the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments described below are merely examples of configurations that can implement the present invention. The following embodiments can be appropriately modified or changed according to the configuration of the device to which the present invention is applied and various conditions. Therefore, the scope of the invention is not limited by the configurations described in the following embodiments.

The imaging device 1 of this embodiment includes an image processing device according to the present invention. The imaging device 1 of this embodiment can be applied to various electronic camera devices such as digital still cameras, digital video cameras, surveillance cameras, industrial cameras, and medical cameras. Also, the imaging device 1 of the present embodiment can be applied to various information processing devices having an imaging function, such as smartphones and tablet terminals.

<Structure of Imaging Device>
FIG. 1 is a block diagram schematically showing the configuration of an imaging device 1 according to an embodiment of the invention. The imaging device 1 has an imaging optical system 10 , an imaging device 11 , a control section 12 , an image processing device 13 , a storage section 14 , an input section 15 , a display section 16 and a communication section 17 .

The imaging optical system 10 is a photographing lens having a plurality of lens groups (not shown), and forms an image of a subject on the imaging device 11 . The imaging optical system 10 may be detachable from the imaging device 1 or may be integrated with the imaging device 1 . If it is detachable, it can be understood that the imaging optical system 10 is not a component of the imaging device 1 . An exit pupil 101 of the imaging optical system 10 exists at a position separated from the imaging device 11 by a predetermined distance. In this specification, the z-axis is parallel to the optical axis 102 of the imaging optical system 10 . Also, the x-axis and the y-axis are orthogonal to each other and orthogonal to the z-axis and the optical axis, respectively.

The imaging device 11 is an image sensor including a CMOS (complementary metal oxide semiconductor), a CCD (charge-coupled device), or the like, and has a range-finding function by an imaging plane phase difference ranging method (image plane phase difference method). . The imaging device 11 photoelectrically converts a subject image formed on the imaging device 11 via the imaging optical system 10 to generate and output an image signal corresponding to the subject image.

The control unit 12 is a control device including elements such as a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and controls each unit of the imaging device 1 . The control unit 12 controls, for example, automatic focusing by autofocus (AF), change of focus position, change of F-number (aperture), and image capture, as well as a storage unit 14, an input unit 15, a display unit 16, and controls the communication unit 17 .

The image processing device 13 is a logic circuit that executes various processes related to image processing. The image processing device 13 has an image generation unit 130 , a memory 131 , a tracking unit 132 , a distance acquisition unit 133 , a position acquisition unit 134 and a speed acquisition unit 135 . The image generation unit 130 performs signal processing such as noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction on the image signal output from the image sensor 11 to generate a captured image (video data). ). The memory 131 accumulates captured images output by the image generator 130 . The tracking unit 132 selects a plurality of captured images including the subject image for which velocity is to be obtained, and tracks the position of the subject image on the image. The distance acquisition unit 133 acquires distance information about the distance between the imaging device 1 and the subject corresponding to the subject image captured in the captured image obtained by the imaging device 1 having a distance measuring function, based on the parallax amount described later. (calculate. The position acquisition unit 134 acquires (calculates) position information about the position of the subject image in the captured image and motion information indicating the position and orientation of the imaging device 1 when the captured image was acquired. The motion information described above is preferably supplied to the position acquisition unit 134 from a gyro sensor and an acceleration sensor (not shown) included in the imaging device 1 . The above distance information, position information, and motion information are preferably stored in the memory 131 in association with the corresponding captured image. Also, the position acquisition unit 134 acquires (calculates) three-dimensional position information of the subject image at each time based on the distance information, the position information, and the motion information. The velocity acquisition unit 135 acquires (calculates) the velocity of the subject corresponding to the subject image using the three-dimensional position information of the captured image selected based on the reliability to be described later. Note that the image processing device 13 may be composed of a memory storing a program for realizing the above processing and a CPU executing the above program.

The storage unit 14 is a non-volatile storage medium such as a flash memory, and stores captured images (video data) acquired by the imaging device 1, imaging information, sensor data, intermediate data, and parameters used by the imaging device 1. etc. is stored. The imaging information is information associated with a captured image and indicating the imaging conditions of the captured image. The storage unit 14 is preferably a high-speed readable/writable storage medium with a large capacity.

The input unit 15 is an operation interface composed of, for example, dials, buttons, switches, etc., and is used by the user to input information and change settings. The display unit 16 is display means such as a liquid crystal display or an organic EL display, and displays information for the user, such as captured images stored in the memory 131, composition at the time of shooting, various setting screens, and message information. indicate. A touch panel that serves as both the input unit 15 and the display unit 16 may be employed.

The communication unit 17 is a communication interface that connects the imaging device 1 and other devices by wire or wirelessly, and is capable of transmitting and receiving information such as a captured image, subject position, and subject speed to and from other devices.

<Structure of image sensor>
FIG. 2 is an explanatory diagram of the imaging device 11 compatible with the imaging surface phase difference ranging method according to the embodiment of the present invention. The imaging element 11 has a plurality of pixels arranged over the x-axis direction and the y-axis direction (that is, on the xy plane). As shown in the pixel cross-sectional view of FIG. 2A, each pixel has a microlens 201, a color filter 202, and a pair of photoelectric conversion units 203A and 203B. Each pixel is provided with RGB (Red, Green, Blue) spectral characteristics according to the wavelength band of the color filter 202 . The plurality of color filters 202 are arranged, for example, according to a coloration pattern (not shown) such as a Bayer arrangement. The photoelectric conversion units 203A and 203B formed in the substrate 204 have sensitivity with respect to the wavelength band to be detected. A wiring (not shown) is connected to each pixel. A pixel signal output from each pixel is supplied to a pixel signal processing unit (not shown), processed, and output from the image sensor 11 as an image signal. Note that the plurality of pixels described above need not be arranged on the xy plane.

FIG. 2B is a diagram showing the exit pupil 101 of the imaging optical system 10 as viewed from the intersection point (center image height) between the optical axis 102 and the image sensor 11 . The exit pupil 101 is divided into a first pupil region 210 and a second pupil region 220, which are different pupil regions. The exit pupil 101 and each photoelectric conversion unit 203 are in a conjugate relationship due to each microlens 201 . The first light flux that has mainly passed through the first pupil region 210 enters the photoelectric conversion unit 203A, and the second light flux that has mainly passed through the second pupil region 220 enters the photoelectric conversion unit 203B. The photoelectric conversion units 203A and 203B obtain an A image (A image signal) and a B image (B image signal) by photoelectrically converting the incident light beams, and output them to the image processing device 13 . The image processing device 13 performs various types of image processing (image generation, distance measurement calculation, etc.) on the input A image and B image, and stores the execution results in the storage unit 14 as necessary. The image processing device 13 can add the A image and the B image to acquire image information (image pickup signal) after the addition. Further, the distance acquisition unit 133 of the image processing device 13 can acquire (calculate) the defocus amount and, in turn, the distance to the subject by executing the correlation calculation between the A image and the B image as follows.

FIG. 2B shows the center-of-gravity position of the first pupil region 210 (first center-of-gravity position 211) and the center-of-gravity position of the second pupil region 220 (second center-of-gravity position 221). In this embodiment, the first center-of-gravity position 211 is decentered (moved) from the center of the exit pupil 101 along the first axis 200 (x-axis). On the other hand, the second center-of-gravity position 221 is decentered (moved) along the first axis 200 from the center of the exit pupil 101 in the direction opposite to the first center-of-gravity position 211 . In FIG. 2B, the direction connecting the first barycentric position 211 and the second barycentric position 221 corresponds to the pupil splitting direction, and the distance between the first barycentric position 211 and the second barycentric position 221 (the distance between the barycenters) is It corresponds to the baseline length 222 .

The positions of the A image and the B image respectively change with respect to the pupil division direction (x-axis direction in this example) due to defocus. The amount of relative positional change between these images, that is, the amount of parallax (phase difference) between the A image and the B image indicates a value corresponding to the amount of defocus (the amount of defocus). The amount of parallax described above is calculated according to the difference between the positions with the highest degree of similarity by executing the correlation calculation between the local regions of the A image and the B image within a predetermined region. As a method for calculating the correlation value, for example, a generally used SAD (Sum of Absolute Difference) method or SSD (Sum of Squared Difference) method can be adopted. The amount of parallax calculated as described above can be converted into the amount of defocus based on a geometric relationship (for example, the principle of triangulation) using the baseline length. The defocus amount can be converted into an object distance based on the imaging relationship of the imaging optical system 10 . A configuration may be adopted in which the defocus amount or the subject distance is obtained by multiplying the parallax amount described above by a predetermined conversion coefficient. As described above, it is possible to obtain the distance from the imaging device 1 (pixel of interest) to the subject using the imaging device 11 having the imaging surface phase difference ranging function.

<Speed acquisition processing flow>
FIG. 3 is a flow chart of speed acquisition processing in this embodiment. The image processing device 13 of the present embodiment acquires (calculates) the velocity of the subject in the video captured and recorded by the imaging device 1 as follows.

In step S301, the user of the imaging device 1 operates the input unit 15 to select the image 401 whose velocity is to be obtained. As shown in FIG. 4A, a selectable image 401 stored in the storage unit 14 is displayed in the display area of the display unit 16 over 3 rows and 3 columns. The video 401 is, for example, a moving image file (video data) acquired using the imaging device 11 of the imaging device 1 having an imaging plane ranging function, and is a plurality of images associated with imaging information used for ranging. Contains an image (multiple frames). Note that the video 401 is not limited to a moving image, and may be, for example, a plurality of still images obtained in chronological order and whose relative acquisition times are clear.

In step S302, a reference frame (reference image) Fb including a subject image that is a target (tracking target) for velocity calculation is selected from the video 401 selected in step S301. More specifically, the user operates the input unit 15 to select the reference frame Fb while referring to the image 401 displayed on the display unit 16 as shown in FIG. 4B. The user can display a frame F corresponding to an arbitrary time on the display unit 16 by operating the scroll bar 402 . When the subject is moving toward the imaging device 1, it is preferable to select the frame F in which the subject is positioned closer to the imaging device 1 as the reference frame Fb. This is because the subject is displayed relatively large in the captured image, and the distance measurement accuracy is affected by the distance to the subject. Although the distance measurement accuracy varies depending on the distance measurement method, the measured distance generally includes an error of about several percent with respect to the object distance.

Note that in step S302, the tracking unit 132 may select the reference frame Fb based on the ranging accuracy determined by the imaging conditions indicated by the imaging information associated with the captured image, regardless of the user's selection. The focal length of the imaging optical system 10, the aperture value (F number), and the distance to the subject are exemplified as imaging conditions that affect the distance measurement accuracy. According to the above configuration, an appropriate reference frame Fb for velocity acquisition can be selected based on the imaging conditions, which are objective parameters.

Next, the user operates the input unit 15 to specify the subject image within the reference frame Fb (within the reference image) that is the target of velocity acquisition. The user can, for example, specify a subject image (subject area) by directly specifying a rectangular area 403 surrounding the subject image, or specify a rectangular area 403 automatically set by specifying a point on or near the subject image. You can specify the subject image (subject area) with . Further, it is also possible to employ a configuration in which the tracking unit 132 designates a subject image (subject area) through general object recognition processing, regardless of user selection.

In step S303, the tracking unit 132 tracks the object image specified in step S302 in a plurality of frames F over time before and after the reference frame Fb. Specifically, as shown in FIG. 4(c), the tracking unit 132 performs template matching using a subject image (subject region) in the reference frame Fb as a template so that the degree of similarity with the template is higher than a predetermined threshold. A frame (captured image) F having a region is selected. That is, the tracking unit 132 selects the frame F based on the similarity of the subject images. According to the above configuration, a frame F having an image area similar to the designated subject image is appropriately selected. Since the frame F selected as described above includes the subject image corresponding to the designated subject image in the reference frame Fb, it may be hereinafter referred to as "corresponding frame (corresponding image) F". Further, hereinafter, a corresponding frame F at a time earlier than the time corresponding to the reference frame Fb will be referred to as a "corresponding frame Fs", and a corresponding frame F at a time later than the time corresponding to the reference frame Fb will be referred to as a "corresponding frame Fe ” may be called. The tracking unit 132 terminates the tracking process when the corresponding frame F is no longer detected before or after the reference frame Fb.

In step S304, the position acquisition unit 134 reconstructs the three-dimensional position of the subject from the corresponding frame Fs to the corresponding frame Fe based on the imaging position (reference imaging position) of the imaging device 1 in the reference frame Fb. More specifically, it is as follows.

The distance acquisition unit 133 acquires (calculates) distance information regarding the distance (subject distance) between the subject corresponding to the designated subject image and the imaging device 1 for each of the reference frame Fb and the corresponding frames Fs to Fe. The distance acquisition unit 133 calculates the distance for each pixel corresponding to the subject area (rectangular area 403 in FIG. 4B) based on the method described above with reference to FIG. It is preferable to obtain a statistical value (median value, average value, etc.) as the representative distance of the subject.

The position acquisition unit 134 acquires (calculates) the center of the rectangular area 403, which is the subject area, as a representative position, which is position information regarding the position of the designated subject, for each of the reference frame Fb and the corresponding frames Fs to Fe. Then, the representative three-dimensional position (Xi, Yi, Zi) (i=Fs to Fe) of the subject is obtained for each of the frames Fs to Fe.

Next, the position acquisition unit 134 acquires motion information of each frame Fs to Fe with respect to the imaging position (reference imaging position) of the imaging device 1 when the reference frame Fb was imaged. The motion information is information indicating changes in the position and posture of the imaging position of the imaging device 1 with respect to the reference imaging position. Based on the output information from the gyro sensor and the acceleration sensor associated with each frame Fs to Fe stored in the memory 131, the position acquisition unit 134 obtains rotation R (for example, rotation matrix) and translation R indicating motion information. Determine T (eg, translation vector). Then, the position acquisition unit 134 obtains the representative three-dimensional positions of the subject in each of the frames Fs to Fe by calculating R ⁻¹ and T ⁻¹ , which are inverse calculations of changes in the position and orientation of the imaging device 1 from the reference frame Fb. applied to transform into a three-dimensional coordinate system with reference to the reference frame Fb.

FIG. 5 is an explanatory diagram of the three-dimensional position reconstruction in step S304. The left column of FIG. 5(a) shows how the imaging device 1 captures images of the subject at times t1 to t4, and the right column shows captured images captured at times t1 to t4. In this example, the frame F at the time t3 when the distance between the imaging device 1 and the subject is the shortest is selected as the reference frame Fb. Corresponding frames Fs to Fe (corresponding to times t1, t2, and t4) are frames F in which the tracking unit 132 has been able to track the object. FIG. 5B shows a three-dimensional image of the object reconstructed by the position acquisition unit 134 with reference to the reference frame Fb based on the representative distances acquired for the corresponding frames Fs to Fe and the representative three-dimensional positions after conversion. indicate position. FIG. 5B is an overhead view of the reconstructed three-dimensional position of the subject. FIG. 5B shows the three-dimensional positions of the subject in the reference frame Fb and the corresponding frames Fs to Fe (that is, times t1 to t4) and the three-dimensional positions of the subject in the corresponding frames F at other times. It shows the trajectory of movement.

In steps S305 and S306, the velocity acquisition unit 135 selects the corresponding frame F to be used for acquiring the velocity of the subject based on the reliability C, and calculates the velocity of the subject based on the three-dimensional position information of the subject in the selected corresponding frame F. is obtained (calculated). An example of reliability C, which is a parameter related to subject distance and imaging conditions, will be described below.

A first example of the reliability C is the subject distance (that is, the distance between the imaging device 1 and the subject) Da in the corresponding frame F (hereinafter referred to as the target frame Fa) to be judged and the subject distance Db in the reference frame Fb. is a difference value Df (Df=|Da-Db|). The speed acquisition unit 135 selects the target frame Fa showing the difference value Df (Df>δ1) exceeding the threshold δ1 (first threshold) as the corresponding frame F to be used for acquiring the speed of the subject. The threshold δ1 is a distance resolution, and is a value determined by imaging conditions and subject distance. According to the above configuration, it is possible to suppress the acquisition of an inaccurate subject velocity based on the corresponding frame F including the subject image at a position below the distance resolution with respect to the reference frame Fb. Note that the difference value Df in this example may be a difference value of the subject distance D in a specific direction (for example, one of the X direction, Y direction, and Z direction), or may be a three-dimensionally calculated subject distance D. A difference value of the distance D may be used.

A second example of the reliability C is the amount of change V indicated by the motion information of the target frame Fa. As the change in the position and orientation of the imaging device 1 increases, the amount of change V also increases. The change amount V may be indicated by the rotation R, the translation T, or a total change value obtained based on the rotation R and the translation T. The speed acquisition unit 135 selects the target frame Fa exhibiting a change amount V (V<δ2) below the threshold δ2 (second threshold) as the corresponding frame F to be used for acquiring the speed of the subject. According to the above configuration, an object speed with low accuracy can be acquired based on the corresponding frame F at the time when the imaging device 1 moves discontinuously or rapidly (that is, when the object moves relatively fast). is suppressed.

The speed acquisition unit 135 repeats the loop of steps S305 and S306 to execute selection processing and speed acquisition processing for each of the corresponding frames Fs to Fe. That is, if unprocessed corresponding frames Fs to Fe exist (step S306: No), the process of step S305 is repeatedly executed to obtain the velocity for each corresponding frame Fs to Fe. When the process of step S305 is completed for all corresponding frames Fs to Fe (step S306: Yes), step S307 is executed.

In step S307, the velocity acquisition unit 135 acquires the subject velocity in the corresponding frame F that was not selected as the subject of velocity acquisition in steps S305 and S306 by complementing processing based on the corresponding frame F in which the velocity of the subject was acquired.

It should be noted that the subject speed in each corresponding frame F obtained in steps S305 to S307 is preferably recorded in the video data by the speed obtaining unit 135 as metadata. Also, when the display unit 16 displays the image 401, it is preferable that the subject velocity in each corresponding frame F is displayed near the subject image in the corresponding frame F. FIG.

As described above, the object velocity can be accurately calculated based on the object distance and the three-dimensional position of the object obtained from the video data (plurality of frames F) including the object image obtained by the imaging device 1 having the distance measuring function. It is possible to obtain (calculate) well. In addition, in the above embodiment, frames (captured images) with low reliability C are not used to acquire the subject velocity, so the acquired subject velocity is more accurate.

Furthermore, in the above configuration, the three-dimensional position of the subject is obtained (calculated) based on motion information indicating the position and orientation of the imaging device 1 . Therefore, it is possible to accurately acquire (calculate) the subject velocity even from video data acquired while moving the imaging device 1 (for example, viewing video data in which the subject image is continuously positioned substantially at the center of the screen). It is possible.

<Modification>
Each of the above embodiments can be modified in various ways. Specific modification modes are exemplified below. Two or more aspects arbitrarily selected from the above embodiments and the following examples can be combined as appropriate as long as they do not contradict each other.

The above-described embodiment solves the problem of accurately obtaining the subject velocity for viewing video data obtained while moving the image capturing apparatus 1, based on the motion information indicating the position and orientation of the image capturing apparatus 1. is solved by the configuration of acquiring the three-dimensional position of However, since the above problem does not exist in the imaging device 1 (for example, a monitoring camera) whose position and posture change little, even if the three-dimensional information of the subject (and thus the subject velocity) is acquired without using the motion information, good. According to the configuration of this modification, the position acquisition unit 134 does not need to acquire motion information, so the configuration of the image processing device 13 can be simplified.

In the above-described embodiment, the image processing device 13 included in the imaging device 1 processes the video (captured image) acquired by the imaging device 1 to acquire the subject velocity. However, the image processing device 13 may be a separate device independent from the imaging device 1 . That is, the independent image processing device 13 may process the video (captured image) acquired by another acquisition device as described above to acquire the subject velocity.

The image processing device 13 can execute the speed acquisition process described above on any video (captured image). For example, the above processing may be performed on the image stored in the image capturing device 1, or the image before being stored in the image capturing device 1 may be subjected to the above processing.

Each functional block of the image processing device 13 may calculate each value (distance, position, motion information, three-dimensional position, speed, etc.) by arithmetic operation, or may store input and output in association with each other in advance. Each value may be obtained by referring to a data set such as a table.

In the above-described embodiment, the velocity acquisition section 135 acquires the subject velocity based on the three-dimensional position of the subject acquired for each corresponding frame F by the position acquisition section 134 . As shown in FIG. 5C, the velocity acquisition unit 135 of this modification smoothes the movement trajectory of the subject indicated by the three-dimensional position of the subject, and acquires the velocity of the subject using the smoothed movement trajectory. can do. The above smoothing processing is realized by, for example, moving average processing or spline processing. According to the configuration of this modified example, errors and fluctuations in the three-dimensional position of the subject can be reduced, so it is possible to obtain the subject velocity with higher accuracy.

In the above-described embodiment, the object speed is calculated based on the three-dimensional position of the object. However, for an image in which the subject moves two-dimensionally with respect to the imaging device 1 as shown in FIG. The speed may be calculated using a coordinate system that omits the height from the ground. In other words, the velocity acquisition unit 135 may map the three-dimensional position indicated by the three-dimensional position information of the subject on a two-dimensional plane, and acquire the velocity of the subject based on the two-dimensional position after mapping. The above mapping may be realized by not using the dimensional data to be omitted, or by obtaining a two-dimensional plane on which the three-dimensional position of the subject is to be mapped and projecting the three-dimensional position of the subject onto the two-dimensional plane. may be implemented. According to the above configuration, the object velocity is acquired in a coordinate system in which dimensions other than the main movement direction are omitted, so that it is possible to suppress errors in the object position and improve the accuracy of velocity acquisition. In addition, since the processing amount of the three-dimensional calculation is reduced, the processing load on the image processing device 13 can also be reduced.

The mapping onto the two-dimensional plane described above may be performed based on the user's designation, or may be performed based on the determination by the speed acquisition unit 135 . For example, the velocity acquisition unit 135 determines whether the variation amount Fl in one dimension (for example, the vertical dimension) of the three-dimensional positions in the corresponding frames Fs to Fe is below the threshold δ3 (third threshold) (Fl<δ3). determine whether or not Then, when the variation amount Fl is less than the threshold value δ3, it is preferable that the velocity acquisition unit 135 performs mapping on a two-dimensional plane (for example, front, back, left, and right planes) corresponding to two dimensions other than the one dimension. According to the above configuration, it is automatically determined whether or not mapping onto a two-dimensional plane is necessary, thereby improving convenience for the user.

Alternatively, the velocity acquisition unit 135 may perform pattern recognition on the subject image within the reference frame Fb. If the object type (category) into which the subject image is classified by pattern recognition corresponds to a specific category indicating an object that is likely to move on a plane (for example, “human” or “automobile”), the speed acquisition unit 135 , preferably performs the mapping described above. Various techniques can be applied to the above pattern recognition. For example, the velocity acquisition unit 135 uses a trained model trained by inputting learning data belonging to the above-described specific category (learning data labeled with the specific category) into a neural network to perform pattern recognition on the subject image. may be executed. According to the above configuration, it is possible to determine whether or not mapping is necessary based on the property of the subject itself, which is different from the three-dimensional position.

The subject image may vary in how it is captured in a plurality of frames F. FIG. Therefore, in step S303, when the similarity is relatively lowered during template matching, the tracking unit 132 uses the matched subject image as a new template (that is, updates the template) to update the template. Processing may continue. Alternatively, the tracking unit 132 may perform the tracking process based on the similarity between the detected object image in the frame F detected by the general object recognition process or the pattern recognition process by deep learning and the template of the reference frame Fb. .

In step S304 of the above-described embodiment, the distance acquisition unit 133 extracts a part of the rectangular area 403, for example, a partial area including the center point, a plurality of continuous areas at predetermined intervals, and an edge area located at the end. A representative distance may be obtained from pixels corresponding to the selected area.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist thereof. For example, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors of a computer of the system or device executes the program. It is also possible to implement the process of reading and executing the The above storage medium is a computer-readable storage medium. The invention can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

1 Imaging device (acquisition device)
REFERENCE SIGNS LIST 10 imaging optical system 11 imaging element 12 control unit 13 image processing device 130 image generation unit 132 tracking unit 133 distance acquisition unit 134 position acquisition unit 135 velocity acquisition unit

Claims (14)

a tracking unit that selects, from the plurality of captured images, a plurality of corresponding images including a subject image corresponding to a subject image in a reference image included in the plurality of captured images;
a distance acquisition unit that acquires, for each of the reference image and the corresponding image, distance information regarding a distance between an acquisition device that acquired the reference image or the corresponding image and a subject corresponding to the subject image;
position information relating to the position of the subject is obtained for each of the reference image and the corresponding image, and three-dimensional position information of the subject relative to the reference image is obtained using the distance information and the position information; an acquisition unit;
a speed acquisition unit that acquires the speed of the subject corresponding to the subject image using the three-dimensional position information in the plurality of corresponding images selected based on the reliability of each of the corresponding images. image processing device. The reliability is a difference value between the distance between the acquisition device and the subject in each of the plurality of corresponding images and the distance between the acquisition device and the subject in the reference image,
2. The image processing apparatus according to claim 1, wherein the speed acquisition unit selects the corresponding images in which the difference value exceeds a first threshold. The position acquisition unit acquires motion information indicating the position and orientation of the acquisition device for each of the plurality of corresponding images, and uses the distance information, the position information, and the motion information to set the reference image as a reference. 3. The image processing apparatus according to claim 1, wherein the three-dimensional positional information of the subject is acquired. The reliability is the amount of change indicated by the exercise information,
4. The image processing apparatus according to claim 3, wherein the speed acquisition unit selects the corresponding images in which the amount of change is less than a second threshold. 3. The speed obtaining unit smoothes a moving trajectory of the subject indicated by the three-dimensional position information of the subject, and obtains the speed of the subject using the smoothed moving trajectory. The image processing apparatus according to any one of claims 1 to 4. The speed acquisition unit maps the three-dimensional position indicated by the three-dimensional position information of the subject on a two-dimensional plane, and acquires the speed of the subject based on the two-dimensional position after mapping. The image processing apparatus according to any one of claims 1 to 5. When the amount of variation in one dimension of the three-dimensional positions in the plurality of corresponding images is below a third threshold, the velocity acquisition unit obtains 7. The image processing apparatus according to claim 6, wherein said mapping is performed. The velocity acquisition unit performs the mapping on the two-dimensional plane when an object type classified by pattern recognition of the subject image in the reference image corresponds to a specific category. Item 7. The image processing device according to item 6. 2. The tracking unit selects the reference image based on at least one of a focal length, an aperture value, and a distance to the subject indicated by imaging information attached to the captured image. 9. The image processing apparatus according to any one of claims 8 to 8. 10. The tracking unit according to any one of claims 1 to 9, wherein the tracking unit selects the corresponding image from the plurality of captured images based on similarity of the subject images included in the plurality of captured images. 2. The image processing device according to item 1. An imaging device comprising the image processing device according to any one of claims 1 to 10,
An imaging device that outputs an image signal corresponding to the subject image,
An imaging apparatus, wherein the image processing apparatus acquires the captured image by performing signal processing on the image signal. a step of selecting a plurality of corresponding images from the plurality of captured images, including a subject image corresponding to a subject image in a reference image included in the plurality of captured images;
obtaining, for each of the reference image and the corresponding image, distance information relating to the distance between an acquisition device that acquired the reference image or the corresponding image and a subject corresponding to the subject image;
a step of acquiring position information regarding the position of the subject for each of the reference image and the corresponding image, and acquiring three-dimensional position information of the subject relative to the reference image using the distance information and the position information; When,
and obtaining the velocity of the subject corresponding to the subject image using the three-dimensional position information in the plurality of corresponding images selected based on the reliability of each of the corresponding images. Image processing method. a step of selecting a plurality of corresponding images from the plurality of captured images, including a subject image corresponding to a subject image in a reference image included in the plurality of captured images;
obtaining, for each of the reference image and the corresponding image, distance information relating to the distance between an acquisition device that acquired the reference image or the corresponding image and a subject corresponding to the subject image;
a step of acquiring position information regarding the position of the subject for each of the reference image and the corresponding image, and acquiring three-dimensional position information of the subject relative to the reference image using the distance information and the position information; When,
obtaining the velocity of the subject corresponding to the subject image using the three-dimensional position information in the plurality of corresponding images selected based on the reliability of each of the corresponding images. A program characterized by causing a a step of selecting a plurality of corresponding images from the plurality of captured images, including a subject image corresponding to a subject image in a reference image included in the plurality of captured images;
obtaining, for each of the reference image and the corresponding image, distance information relating to the distance between an acquisition device that acquired the reference image or the corresponding image and a subject corresponding to the subject image;
a step of acquiring position information regarding the position of the subject for each of the reference image and the corresponding image, and acquiring three-dimensional position information of the subject relative to the reference image using the distance information and the position information; When,
obtaining the velocity of the subject corresponding to the subject image using the three-dimensional position information in the plurality of corresponding images selected based on the reliability of each of the corresponding images. A computer-readable storage medium that stores a program to be executed by

Images