JP2015087904A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for creating an image representing the trajectory of an object from a plurality of images obtained by photographing the object in time intervals.SOLUTION: An image processor includes an extraction part for receiving a first image and a second image obtained by sequentially photographing an object, extracting a blur of the object included in the first image as a first trajectory of the object, and extracting a blur of the object included in the second image as a second trajectory of the object, an estimation part for estimating features of motions of the object at times when the first image and the second image are respectively photographed on the basis of respective image features of the first trajectory and the second trajectory extracted by the extraction part, and an interpolation part for acquiring a trajectory of the object between photographing of the first image and photographing of the second image on the basis of the features of the motions of the object estimated by the estimation part and interpolating a trajectory between the first trajectory and the second trajectory by using the acquired trajectory.

Description

  The present invention relates to an image processing apparatus and an image processing method.

  For example, there is a technology that generates an image that expresses the motion of an object by cutting out an image of the object from each of a plurality of images obtained by capturing a moving object at a predetermined time interval and combining the cut out images into one image. It has been proposed (see, for example, Patent Document 1).

JP 2004-164563 A

  When a moving object is photographed at a predetermined time interval, the time when each image is photographed is discretely distributed on the time axis. For this reason, the image of the object cut out from each of the plurality of captured images represents the object at each moment distributed discretely on the time axis. Therefore, even if the images of the objects cut out from the respective images are combined with each other, it is difficult to generate an image that continuously represents the trajectory drawn in the process of movement of the object.

  An object of the image processing apparatus and the image processing method disclosed herein is to provide a technique for generating an image representing a trajectory of an object from a plurality of images obtained by photographing the object at time intervals.

  According to one aspect, the image processing apparatus receives a first image and a second image obtained by sequentially photographing an object, and uses a blur of the object included in the first image as a first trajectory of the object. An extraction unit that extracts and extracts the blurring of the object included in the second image as the second trajectory of the object, and the first trajectory and the first trajectory extracted by the extraction unit, based on the characteristics of each image. Based on the motion characteristics of the object estimated by the estimation section, the estimation section estimating the motion characteristics of the object at the time when each of the image and the second image was captured, and after the first image is captured An interpolating unit that obtains the trajectory of the object until two images are captured and interpolates the trajectory between the first trajectory and the second trajectory using the obtained trajectory.

  According to another aspect, the image processing method receives an input of a first image and a second image obtained by sequentially photographing an object, and blurs the object included in the first image. And the blur of the object included in the second image is extracted as the second trajectory of the object, and the first image and the second image are extracted based on the characteristics of the extracted images of the first trajectory and the second trajectory. The object motion characteristics at the time when each of the images is photographed are estimated, and the object between the time when the first image is photographed and the time when the second image is photographed based on the estimated object motion characteristics And the trajectory is interpolated between the first trajectory and the second trajectory using the obtained trajectory.

  The image processing apparatus and the image processing method according to the present disclosure can generate an image representing a locus of an object from a plurality of images obtained by photographing the object at time intervals.

It is a figure which shows one Embodiment of an image processing apparatus. It is a figure which shows the example of the image image | photographed with the imaging device shown in FIG. It is a figure which shows the example of interpolation of the 1st locus | trajectory by the interpolation part shown in FIG. 1, and a 2nd locus | trajectory. It is a figure which shows operation | movement of the image processing apparatus shown in FIG. It is a figure which shows another embodiment of an image processing apparatus. It is a figure which shows the example of the change of the pixel value in the locus | trajectory of the object shown in FIG. It is a figure which shows the example of the area | region estimated by the prediction part shown in FIG. It is a figure which shows the example of the interpolation locus | trajectory produced | generated by the production | generation part shown in FIG. It is a figure which shows the hardware structural example of the digital camera carrying an image processing apparatus. It is a figure which shows operation | movement of the image processing apparatus shown in FIG. It is a figure which shows the example of the process which detects the group of the locus | trajectory which shows the motion of the same object. It is a figure which shows an example of the hardware constitutions of an image processing apparatus. It is a figure which shows another example of the hardware constitutions of an image processing apparatus.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 shows an embodiment of an image processing apparatus.

  The image processing apparatus 10 illustrated in FIG. 1 includes an extraction unit 11 that receives images IMG_1,..., IMG_n from the imaging device GRD, and an estimation unit 12 and an interpolation unit 13 connected to the extraction unit 11. The outputs of the extraction unit 11 and the interpolation unit 13 are connected to a synthesizing device SYN that generates a synthesized image GSN. The functions and operations of the extraction unit 11, the estimation unit 12, and the interpolation unit 13 will be described with reference to FIG.

  The photographing device GRD is a digital camera or the like, and obtains n (n is an integer of 2 or more) images IMG_1,..., IMG_n by photographing a moving object at a predetermined time interval, for example, and obtains the obtained image IMG_1. ,..., IMG_n are passed to the image processing apparatus 10. Note that the photographing device GRD is not limited to a digital camera, and may be a camera mounted on a portable terminal device such as a mobile phone or a smartphone as long as it has a continuous shooting function and a function for adjusting an exposure time. Further, in the reference numerals indicating the respective images in FIG. 1, the number after the symbol “_” indicates the photographing order by the photographing device GRD. In the following description, when the images IMG_1,..., IMG_n are not distinguished from each other, they are collectively referred to as an image IMG.

  The exposure time used when photographing an object by the photographing apparatus GRD is set to a time at which the position of the object in the photographed image IMG causes a blur of about several pixels due to the movement of the object. Note that the exposure time used by the imaging device GRD is preferably set to a value that does not saturate the imaging device due to the amount of light incident on the imaging device mounted on the imaging device GRD during the exposure time.

  Each of the images IMG_1,..., IMG_n photographed by the photographing apparatus GRD described above includes a trajectory in which the object moves during the exposure time used for photographing as a blurring of an image indicating the object.

  FIG. 2 shows an example of an image IMG photographed by the photographing apparatus GRD shown in FIG. An image IMG_k shown in FIG. 2A and an image IMG_k + 1 shown in FIG. 2B are k (k is n−1 or less) in n images obtained by photographing by the photographing device GRD. This is an example of images taken in the (positive integer) th and (k + 1) th. That is, the image IMG_k is an example of a first image obtained by photographing a moving object, and the image IMG_k + 1 is an example of a second image taken after the first image. For example, the image IMG_k + 1 is an image photographed after the image IMG_k by the photographing device GRD.

  The image IMG_k shown in FIG. 2A is the image IMG_k taken in the process in which the ball BL, which is an example of an object, rolls on the lawn LN indicated by the shaded area toward the pin PIN. The ellipsoidal trajectory Tr_k along which the ball BL has moved during the exposure time is included. In the example of FIG. 2A, the trajectory Tr_k indicates that the ball that was at the position indicated by the dashed circle Ps_k at the start of the exposure time has moved to the position of the dashed circle Pe_k at the end of the exposure time. Yes. That is, the trajectory Tr_k shown in FIG. 2A is a part of the trajectory followed by the ball while n images are captured, and the first trajectory extracted from the image IMG_k corresponding to the first image. It is an example. Note that the circle indicating the ball BL and the circles Ps_k and Pe_k indicating the position of the ball BL at the start and end of the exposure time are elements shown for explaining the trajectory of the ball BL, and are included in the image IMG_k. Absent.

  Similarly, the image IMG_k + 1 shown in FIG. 2B is an oval shape in which the ball moves during the exposure time when the image IMG_k + 1 is photographed in the process of rolling the ball over the lawn LN toward the pin PIN. Trajectory Tr_k + 1. In the example of FIG. 2B, the trajectory Tr_k indicates that the ball that was at the position indicated by the broken-line circle Ps_k + 1 at the start of the exposure time has moved to the position of the broken-line circle Pe_k + 1 at the end of the exposure time. Yes. That is, the trajectory Tr_k + 1 shown in FIG. 2B is a part of the trajectory followed by the ball while n images are captured, and the trajectory of the second trajectory extracted from the image IMG_k + 1 corresponding to the second image. It is an example. Note that the circle indicating the ball BL and the circles Ps_k + 1 and Pe_k + 1 indicating the position of the ball BL at the start and end of the exposure time are elements shown for explaining the trajectory of the ball BL, and are included in the image IMG_k + 1. Absent.

  The extraction unit 11 of the image processing apparatus 10 illustrated in FIG. 1 performs, for example, blurring of the image of the ball BL due to the movement of the ball BL illustrated in FIG. 2 from each of the two images IMG photographed sequentially. Extracted as trajectories Tr_k and Tr_k + 1 which are part of the BL trajectory. For example, the extraction unit 11 extracts a trajectory Tr_k indicating that the ball BL has moved between the position of the circle Ps_k and the position of the circle Pe_k from the image IMG_k illustrated in FIG. Similarly, the extraction unit 11 extracts a trajectory Tr_k + 1 indicating that the ball BL has moved between the position of the circle Ps_k + 1 and the position of the circle Pe_k + 1 from the image IMG_k + 1 illustrated in FIG.

  The extraction unit 11 extracts, for example, a technique for detecting an image of a moving object from a plurality of images photographed in time series, and extracts trajectories Tr_k and Tr_k + 1 included in two sequentially captured images IMG_k and IMG_k + 1. You may use for.

  For example, the extraction unit 11 generates a difference image indicating a difference between the images IMG_k and IMG_k + 1 illustrated in FIGS. 2A and 2B, and pixels having a difference greater than or equal to a predetermined value are connected in the generated difference image. Detect the area that is. Here, the region detected from the difference image has a feature different from the feature of the image indicating the background (for example, lawn LN) that does not move between the two images IMG_k and IMG_k + 1 in either of the images IMG_k and IMG_k + 1. This is an area showing the moving ball BL. In the example of the difference image generated from the images IMG_k and IMG_k + 1, the extraction unit 11 detects two regions corresponding to the trajectory Tr_k and the trajectory Tr_k + 1 shown in FIGS. 2A and 2B from the difference image. Then, the extraction unit 11 compares the pixel features between the inside and outside of the boundary of the region of the image IMG_k corresponding to each of the two regions detected from the difference image, and the features are different between the inside and outside of the boundary. The region is extracted as a trajectory Tr_k included in the image IMG_k. Similarly, the extraction unit 11 extracts, as a trajectory Tr_k + 1 included in the image IMG_k + 1, a region having different characteristics between the inside and the outside of the boundary of the region of the image IMG_k + 1 corresponding to each of the two regions detected from the difference image. . For example, in the image IMG_k, the region corresponding to the trajectory Tr_k has different features on the inner side and the outer side of the boundary, and thus is extracted as the trajectory Tr_k by the extraction unit 11. On the other hand, in the image IMG_k, the region corresponding to the trajectory Tr_k + 1 is not extracted as a trajectory because the inside and the outside of the boundary are both lawn LN and the features are common.

  Note that the technique used by the extraction unit 11 to extract the trajectories Tr_k and Tr_k + 1 from two images IMG photographed sequentially distinguishes a background that does not move in each image IMG and an object whose position changes in each image IMG. Any technique can be used. For example, the extraction unit 11 detects a region in which a motion vector obtained for each block including a predetermined number of pixels is similar in a plurality of images IMG taken sequentially, and extracts the detected region as a moving object. Technology may be used.

  The estimation unit 12 illustrated in FIG. 1 receives the image IMG and information indicating the trajectories Tr_k and Tr_k + 1 from the extraction unit 11. The estimation unit 12 estimates the feature of the motion indicated by the ball BL included in the image IMG at the time when the images IMG_k and IMG_k + 1 are captured based on the features of the images of the regions extracted as the trajectories Tr_k and Tr_k + 1. .

  Based on the correlation between the direction of change in the pixel value indicating the color or luminance of the pixel included in the region of the image IMG extracted as the trajectory Tr_k or the trajectory Tr_k + 1 and the direction of motion of the object, for example, The direction of motion of the object is estimated from the direction of change of the pixel value. Further, the estimation unit 12 estimates the moving speed of the object in the image IMG from the estimated length of the motion direction of the region of the image IMG extracted as the trajectory Tr_k or the trajectory Tr_k + 1. A method of estimating the motion characteristics of the object by the estimation unit 12 will be described later with reference to FIGS. 5 and 6.

  For example, when the estimation unit 12 estimates the direction Dr_k indicated by the arrow as the direction of motion from the trajectory Tr_k extracted from the image IMG_k shown in FIG. 2A, the points Qs_k and Qe_k are both sides of the trajectory Tr_k. Specify as the end point of. Similarly, when the estimation unit 12 estimates the direction indicated by the arrow as the motion direction Dr_k + 1 from the trajectory Tr_k + 1 extracted from the image IMG_k + 1 illustrated in FIG. 2B, the estimation unit 12 sets the points Qs_k + 1 and Qe_k + 1 to both sides of the Tr_k + 1. Specify as the end point of. Further, the estimation unit 12 divides the distance between the end points specified for each of the trajectory Tr_k and the trajectory Tr_k + 1 by the exposure time, so that the estimated direction of motion is obtained at the time when each of the images IMG_k and IMG_k + 1 is captured. Find the speed of the moving ball BL. Then, the estimating unit 12 starts from each end point Qs_k, Qe_k of the trajectory Tr_k from the obtained direction Dr_k and speed, has a direction away from the trajectory Tr_k along the estimated direction of motion, and the estimated speed Two velocity vectors V1_k and V2_k indicating Similarly, the estimation unit 12 obtains two velocity vectors V1_k + 1 and V2_k + 1 having directions from the end points Qs_k + 1 and Qe_k + 1 of the trajectory Tr_k + 1 as starting points and away from the trajectory Tr_k + 1 from the obtained direction Dr_k + 1 and speed. Then, the estimation unit 12 uses the obtained two sets of velocity vectors (V1_k, V2_k) and (V1_k + 1, V2_k + 1) as information indicating the characteristics of the motion of the ball BL at the time when each of the images IMG_k and IMG_k + 1 is captured. The data is passed to the interpolation unit 13 shown in FIG.

  Furthermore, the estimation unit 12 may select one of the two velocity vectors V1_k and V2_k based on the relative position between the trajectory Tr_k and the trajectory Tr_k + 1, and pass the selected velocity vector to the interpolation unit 13. For example, in the example of FIGS. 2A and 2B, the two end points Qs_k + 1 and Qe_k + 1 of the trajectory Tr_k + 1 are on the side indicated by the velocity vector V2_k relative to the two end points Qs_k and Qe_k of the trajectory Tr_k. In this case, the estimation unit 12 specifies the velocity vector V2_k illustrated in FIG. 2A as the direction of the movement of the object at the time when the image IMG_k was captured. Here, since there is little change in the characteristics of the motion of the ball BL during the period from when the image IMG_k is captured to when the image IMG_k + 1 is captured, the motion of the ball BL at each time when the images IMG_k and IMG_k + 1 are captured. The directions are almost the same. Therefore, the estimation unit 12 selects a velocity vector V2_k + 1 having a direction similar to the velocity vector V2_k selected based on the relative position between the locus Tr_k and the locus Tr_k + 1 as a feature of the motion of the ball BL at the shooting time of the image IMG_k + 1. May be. In this case, the estimation unit 12 uses the velocity vector V2_k illustrated in FIG. 2A and the velocity vector V2_k illustrated in FIG. 2B as information indicating the characteristics of the motion of the ball BL when each of the images IMG_k and IMG_k + 1 is captured. The indicated velocity vector V2_k + 1 is passed to the interpolation unit 13.

  The interpolation unit 13 illustrated in FIG. 1 receives the image IMG from the extraction unit 11 and receives information indicating the characteristics of the motion of the ball BL from the estimation unit 12. Based on the motion characteristics of the ball BL estimated by the estimation unit 12, the interpolation unit 13 follows the trajectory of the ball BL moved at the shooting interval between the image IMG_k corresponding to the first image and the image IMG_k + 1 corresponding to the second image. Is generated. The interpolation unit 13 then interpolates between the trajectory Tr_k extracted from the image IMG_k as the first trajectory and the trajectory Tr_k + 1 extracted from the image IMG_k + 1 as the second trajectory using the generated trajectory. For example, the interpolating unit 13 obtains the average of the components indicating the directions of the two velocity vectors V2_k and V2_k + 1 estimated with respect to the trajectories Tr_k and Tr_k + 1 and the average of the velocities, so Find the speed. The interpolation unit 13 generates a trajectory of the ball BL at the shooting interval of the images IMG_k and IMG_k + 1 from the direction and speed indicated by the obtained speed. Then, the interpolation unit 13 performs interpolation by connecting the trajectories Tr_k and Tr_k + 1 corresponding to the first trajectory and the second trajectory by the generated trajectory.

  FIG. 3 shows an example of interpolation between the first locus and the second locus by the interpolation unit 13 shown in FIG. 3 that are equivalent to the elements shown in FIG. 2 are denoted by the same reference numerals as those in FIG. 2 and description of the elements may be omitted.

  In FIG. 3A, interpolation is performed between the trajectories Tr_k and Tr_k + 1 extracted as the first trajectory and the second trajectory from the images IMG_k and IMG_k + 1 shown in FIG. 2, respectively, by the interpolating unit 13 shown in FIG. An example of the interpolated trajectory Trc_k obtained in (1) is shown. In FIG. 3A, coordinate axes X and Y are used in the interpolation unit 13 to associate the positions of the trajectories Tr_k and Tr_k + 1 with each other.

  A rectangle Tri_k surrounded by a broken line in FIG. 3A is an example of the trajectory of the ball BL at the shooting interval of the images IMG_k and IMG_k + 1 interpolated by the interpolation unit 13 based on the velocity vectors V2_k and V2_k + 1. The trajectory Tri_k shown in FIG. 3A has a pair of sides parallel to the direction indicated by the velocity vector Vi_k obtained from the velocity vectors V2_k and V2_k + 1 with an interval indicated by the average width w of the trajectory Tr_k. Parallelograms facing each other.

  Further, the interpolation unit 13 illustrated in FIG. 1 performs the color and color of each pixel included in the image indicating the trajectory Tri_k based on the pixel value indicating the color and luminance of each pixel included in the image indicating the trajectory Tr_k or Tr_k + 1. It is desirable to set a pixel value indicating luminance. For example, the interpolation unit 13 obtains an average value of the pixel values of a plurality of pixels located closer to the center than the contour of the trajectory Tr_k, and uses the obtained average value for each pixel included in the image indicating the trajectory Tri_k. The pixel value may be set.

  The interpolating unit 13 uses the trajectory Tri_k shown in FIG. 3A to connect the trajectories Tr_k and Tr_k + 1, thereby generating a continuous trajectory Trc_k from the end point Qs_k of the trajectory Tr_k to the end point Qe_k + 1 of the trajectory Tr_k + 1. be able to.

  Then, the image processing apparatus 10 illustrated in FIG. 1 sequentially performs the above-described processing using each image IMG captured by the imaging apparatus GRD as a first image, thereby performing n images IMG_1,..., IMG_n. It is possible to generate a trajectory that continuously represents the movement of the object during the period in which the image is captured.

  For example, the interpolating unit 13 sets each of the two images IMG taken consecutively as the first image and the second image, and calculates the first locus and the second locus extracted from the first image and the second image. The interpolated trajectory may be passed to the synthesizing device SYN every time a trajectory for interpolating the gap is generated. For example, the extraction unit 11 passes the image IMG_1 received first from the imaging device GRD to the synthesizing device SYN as information indicating the background of the ball BL.

  FIG. 3B shows a trajectory Trc_k obtained by arranging the trajectory Tri_k generated by the interpolation unit 13 between the trajectories Tr_k and Tr_k + 1 shown in FIGS. 2A and 2B and the image IMG_1. An example of a synthesized image GSN synthesized by the synthesizing device SYN shown in FIG.

  The synthesizing device SYN shown in FIG. 1 is, for example, an image IMG_k that is each of n images IMG_1,..., IMG_n, and is interpolated each time interpolation is performed using the trajectory Tri_k generated by the interpolation unit 13. The trajectory Trc_k is received. Then, the synthesizing device SYN sequentially adds the received trajectory Trc_k to the image IMG_1. As a result, when the interpolation process of the trajectory Tri_k for all of the n images IMG is completed, the ball BL moves in the background including the lawn LN and the pin PIN during the period when the n images IMG were taken. A composite image GSN that continuously represents the trajectory can be generated.

  The synthesizing device SYN may be provided outside the image processing device 10 as illustrated in FIG. 1 or may be included in the image processing device 10. The image processing apparatus 10 including components having the function of the synthesizing apparatus SYN will be described later with reference to FIGS. 5 and 9 to 10.

  FIG. 4 shows the operation of the image processing apparatus 10 shown in FIG. The processing in steps S301 to S303 shown in FIG. 4 shows the operation of the image processing apparatus 10 shown in FIG. 1 and an example of an image processing method. The processing illustrated in FIG. 4 may be executed by, for example, hardware mounted on the image processing apparatus 10 or described in a format that can be read by the processor mounted on the image processing apparatus 10 by the processor. It may be realized by executing a program.

  Each process shown in FIG. 4 is included in the image processing apparatus 10 by sequentially setting each image IMG from the first image IMG_1 taken to the image IMG_n-1 taken first to the first image as a first image, for example. The extraction unit 11, the estimation unit 12, and the interpolation unit 13 are repeatedly executed. 4 may be executed by the estimation unit 12 and the interpolation unit 13 each time the second and subsequent images IMG_2,..., IMG_n are captured by the imaging device GRD. When the second and subsequent images are captured, for example, in each process shown in FIG. 4, the newly captured image is taken as the second image, and is captured one image before the newly captured image. This image is the first image.

  In step S <b> 301, as described with reference to FIG. 2, the extraction unit 11 detects an object that appears when the ball BL, which is an example of an object, moves during the exposure time from each of the first image and the second image. Image blur is extracted as a first trajectory and a second trajectory.

  In step S302, the estimation unit 12 estimates the feature of the motion of the object at the time when the first image and the second image were taken based on the shapes of the first trajectory and the second trajectory extracted in the process of step S301. To do.

  In step S303, the interpolation unit 13 generates a trajectory of the object moved at the shooting interval between the first image and the second image based on the motion feature estimated in the process of step S302, and uses the generated trajectory. To interpolate between the first trajectory and the second trajectory.

  By executing the processes described above, the image processing apparatus 10 can detect the movement of the object during the exposure time included in each of the two images captured at different times with the predetermined exposure time. A trajectory for interpolating is generated. Thereby, the image processing apparatus 10 can generate a continuous trajectory including the trajectory to be interpolated. Then, when each image IMG from the image IMG_1 to the image IMG_n−1 is sequentially set as the first image, the trajectory generated by the image processing device 10 is, for example, the image IMG_1 that is first captured by the synthesizing device SYN. Are synthesized sequentially. As a result, the image processing apparatus 10 can continuously indicate the entire trajectory of the movement of the object in the process in which n images IMG are captured by the imaging device GRD.

  That is, the image processing apparatus 10 illustrated in FIG. 1 can generate an image that continuously represents the locus of an object from images obtained by sequentially capturing moving objects at time intervals. Thereby, for example, the image processing apparatus 10 expresses the trajectory of the ball rolling on the lawn of the golf course in the daytime as a continuous trajectory similar to the case where the taillight of the car running on the night highway is photographed with long exposure. An image to be generated can be generated.

  FIG. 5 shows another embodiment of the image processing apparatus. Note that among the components shown in FIG. 5, components equivalent to those shown in FIG. 1 are denoted by the same reference numerals and description of the components may be omitted.

  An image processing apparatus 10 illustrated in FIG. 5 includes an extraction unit 11 illustrated in FIG. 1, an estimation unit 12a corresponding to the estimation unit 12 illustrated in FIG. 1, and an interpolation unit corresponding to the interpolation unit 13 illustrated in FIG. 13a and a synthesizing unit 14 corresponding to the synthesizing device SYN shown in FIG. Furthermore, the image processing apparatus 10 includes a trajectory database DB, and the extraction unit 11, the estimation unit 12a, and the interpolation unit 13a can exchange information with each other via the trajectory database DB.

  The extraction unit 11 illustrated in FIG. 5 sequentially receives, as a first image and a second image, two images IMG having a continuous shooting order from n images IMG taken by the photographing device GRD. Then, the extraction unit 11 extracts the blurring of the object image included in the image IMG received as the first image or the second image as the first locus or the second locus. Then, the extraction unit 11 accumulates, in the trajectory database DB, information indicating blurring of the object image extracted from each image as the first trajectory or the second trajectory as information indicating the trajectory of the object included in each image.

  Here, when the extraction unit 11 receives each image IMG taken after the third image, the extraction unit 11 newly uses the information indicating the locus extracted from the image IMG taken before the received image IMG. You may extract the locus | trajectory from the received image IMG. For example, when extraction of the trajectory from each of the images IMG_1 and IMG_2 is completed and the third image IMG_3 is received, the extraction unit 11 extracts the trajectory from the image IMG_3 from the image IMG_2. You may use the information which shows the made | formed locus | trajectory.

  The estimation unit 12a illustrated in FIG. 5 includes a calculation unit 121 and a specifying unit 122, and calculates the color or luminance of each pixel included in the trajectory of the object extracted as the first trajectory or the second trajectory by the extraction unit 11. The direction of the movement of the object is estimated based on the feature of change depending on the position of the pixel value indicated.

  The interpolation unit 13 a includes a detection unit 131 and a generation unit 132, and the detection unit 131 further includes a prediction unit 134 and a collation unit 133. Then, the interpolation unit 13a interpolates between each of the first trajectories extracted by the extraction unit 11 and the second trajectory indicating the motion in the second image of the object indicated by the first trajectory.

  Further, the synthesizing unit 14 has a function corresponding to the synthesizing device SYN shown in FIG. 1, and receives the first image IMG_1 taken from the extracting unit 11 and receives the interpolated trajectory from the interpolating unit 13a. A composite image GSN is generated by adding the interpolated trajectory to the image IMG_1.

  FIG. 6 shows an example of a change in pixel value in the trajectory Tr_k of the object shown in FIG. The coordinate axes X and Y shown in FIG. 6A are used in the estimation unit 12a to indicate the positional relationship of each pixel. Also, the coordinate axis B shown in FIGS. 6B and 6C indicates the magnitude of the luminance value.

  FIG. 6A shows an oval closed curve C1, C2, C3, C4 in a trajectory Tr_k of the object, each showing a range in which pixels whose luminance values are greater than or equal to the numerical values B1, B2, B3, B4 are distributed. Contour lines. In FIG. 6, the values increase in the order of numerical values B1, B2, B3, and B4. 6B shows a change in the luminance value B of each pixel arranged in the direction Dr1 shown by the broken line in FIG. 6A, and FIG. 6B shows the alternate long and short dash line in FIG. The change of the luminance value B of each pixel arranged in the direction Dr2 shown in FIG.

  The example of FIG. 6A shows a trajectory Tr_k extracted from the image IMG_k when the ball BL shown in FIG. 2 moves in the direction Dr1. As can be seen from FIG. 6A, the gradient of the luminance value in the direction Dr1 is smaller than the gradient of the luminance value in other directions including the direction Dr2 intersecting the direction Dr1. In other words, there is a correlation between the gradient of the luminance value in the trajectory of the object extracted as the first trajectory or the second trajectory from the image IMG and the direction in which the object depicting the trajectory has moved. Similarly, between the gradient of the chromaticity value indicating the color of each pixel included in the trajectory of the object extracted from the image IMG as the first trajectory or the second trajectory, and the direction in which the object depicting the trajectory moves There is a correlation. Therefore, in the region indicating the extracted locus, the direction in which the gradient of the pixel value indicating the color or the luminance is smaller than the other directions indicates the direction in which the object depicting the extracted locus has moved.

  The calculation unit 121 illustrated in FIG. 5 has a gradient indicating a change due to the position of the pixel value indicating the color or luminance of each pixel included in each of the object trajectories extracted as the first trajectory or the second trajectory by the extraction unit 11. Is calculated.

  The identification unit 122 illustrated in FIG. 5 identifies a direction in which a gradient smaller than the other directions is calculated based on the gradient of the pixel value obtained by the calculation unit 121 for the trajectory of the object extracted as the first trajectory. To do. Then, the specifying unit 122 sets the specified direction as the direction of motion of the object at the time when the image IMG including the first trajectory is captured. Similarly, the specifying unit 122 specifies a direction in which a gradient smaller than the other directions is calculated based on the gradient of the pixel value obtained by the calculating unit 121 for the trajectory of the object extracted as the second trajectory. Then, the specifying unit 122 sets the specified direction as the direction of movement of the object at the time when the second image is captured.

  Further, the specifying unit 122 obtains a distance between two end points of the trajectory that is the point farthest in the direction of motion specified for each of the first trajectory and the second trajectory for each of the first trajectory and the second trajectory. . Then, the specifying unit 122 obtains the speed of the object in the image IMG including each of the first locus and the second locus based on the obtained distance and the exposure time.

  For example, the specifying unit 122 obtains the length L_k in the direction Dr1 of the trajectory Tr_k illustrated in FIG. For example, the specifying unit 122 obtains the length of the direction Dr1 in the closed curve C1 having the smallest luminance value as the length L_k among the closed curves shown in FIG. Then, the specifying unit 122 can determine the speed at which the ball BL illustrated in FIG. 2 moves in the direction Dr1 in the image IMG_k including the trajectory Tr_k by dividing the obtained length L_k by the exposure time.

  Then, the specifying unit 122 corresponds to each of the trajectories of the object extracted as the first trajectory or the second trajectory, and stores a velocity vector indicating the direction and speed of the motion obtained as described above. Accumulate in DB. In addition, the identification unit 122 detects the end points on both sides detected for each of the trajectories in the process of obtaining the speed of the object as part of the information corresponding to each of the trajectories of the object extracted as the first trajectory or the second trajectory. It is desirable to store information indicating the position in the trajectory database DB.

  Note that, as in the example illustrated in FIG. 6A, when the shape of the region indicating the trajectory of the object can be analyzed to identify the major axis and the minor axis of the region shape, the calculation unit 121 Thus, the direction of obtaining the gradient of the luminance value or chromaticity value may be narrowed down to the major axis direction and the minor axis direction. In this case, the specifying unit 122 compares the gradient calculated in the major axis direction with the gradient calculated in the minor axis direction by the calculating unit 121, and specifies the direction in which the gradient is small as the direction of motion of the object. Then, the speed of movement of the object is obtained from the length in the specified direction.

  By the way, each image IMG photographed by the photographing device GRD shown in FIG. 5 may include a plurality of moving objects. In this case, for example, the extraction unit 11 illustrated in FIG. 5 extracts the trajectory of the object included in each image IMG corresponding to each of the plurality of objects as the first trajectory or the second trajectory. Each object trajectory is distinguished by attaching a unique number to each image. Then, when accumulating the extracted object trajectory in the trajectory database DB, the extraction unit 11 determines the imaging order indicating the image IMG including the extracted object trajectory and the number for distinguishing the object trajectory and the object trajectory. Correlate with the information shown. Further, in this case, the estimation unit 12a illustrated in FIG. 5 performs the object indicated by the object trajectory for each of the object trajectories extracted from each image IMG as the first trajectory or the second trajectory by the extraction unit 11. A velocity vector indicating the characteristics of the motion is obtained.

  When a plurality of moving objects are included, the detection unit 131 illustrated in FIG. 5 includes, for each first trajectory extracted from the first image by the extraction unit 11, a plurality of second trajectories extracted from the second image. A second trajectory in which a motion feature similar to the motion feature estimated for the first trajectory is estimated is detected. For example, the detection unit 131 calculates the velocity vectors estimated for the trajectories of a plurality of objects extracted from the kth image IMG_k and the trajectories of the plurality of objects extracted from the k + 1th image IMG_k + 1. The estimated velocity vectors are checked against each other. Note that the numerical value k indicating the photographing order is a positive integer of n−1 or less. Then, the detection unit 131 sets a set of object trajectories for which velocity vectors similar to each other are estimated as a first trajectory and a second trajectory indicating the motion of the same object in each of the images IMG_k and IMG_k + 1.

  For example, prior to collating the motion features indicated by the trajectories of a plurality of objects extracted from each of the images IMG_k and IMG_k + 1 with each other, the detection unit 131 detects the motion estimated for the trajectory extracted from the image IMG_k. You may narrow down based on the feature.

  Based on the velocity vector estimated for each of the first trajectories by the estimating unit 12a, the predicting unit 133 of the detecting unit 131 determines the position where the object depicting the first trajectory is predicted to exist in the second image. Find the area to include. In addition, the collation unit 134 of the detection unit 131 has the first trajectory and the motion feature out of the second trajectory extracted from the region predicted by the prediction unit 133 based on the velocity vector estimated for the first trajectory. A similar second trajectory is detected.

  FIG. 7 shows an example of a region predicted by the prediction unit 133 shown in FIG. A region Tr1_k indicated by shading in FIG. 7 indicates one of the trajectories of the object extracted from the image IMG_k. Each of the regions Tr1_k + 1 and Tr2_k + 1 indicated by shading in FIG. 7 represents one of the trajectories of the object extracted from the image IMG_k + 1. In FIG. 7, the difference in the type of shading indicates a difference in color or luminance indicated by the shaded area. That is, the object trajectory Tr1_k and the object trajectory Tr1_k + 1 shown in FIG. 7 have similar colors or brightness, and the object trajectory Tr1_k + 1 and the object trajectory Tr2_k + 1 show different colors or brightness.

  The regions RNG1 and RNG2 illustrated in FIG. 7 are predicted to have an object at the shooting time of the image IMG_k + 1 by the prediction unit 133 illustrated in FIG. 5 based on the motion direction Dr1_k estimated for the trajectory Tr1_k. Indicates the area.

  In the example of FIG. 7, the regions RNG1 and RNG2 are centered on a point inside the trajectory Tr1_k, and are twice the angle θ (for example, 20 degrees) indicating a threshold for determining that the directions indicated by the two velocity vectors are similar. It is shown as a sector with a central angle. The distance Ex_k from the intersection of the sector arc indicating the region RNG1 and the straight line indicating the direction of motion Dr1_k estimated based on the trajectory Tr1_k to the end point Q1s_k of the trajectory Tr1_k is the speed estimated based on the trajectory Tr1_k. Is set based on For example, the distance Ex_k is a numerical value of about 1.1 to 1.5, which is a distance that the object moves from the time when the image IMG_k is captured at the speed estimated based on the trajectory Tr1_k to the time when the image IMG_k + 1 is captured. It is desirable to set a value obtained by multiplying a predetermined coefficient having a value. In FIG. 7, although not shown, the distance from the intersection of the fan-shaped central angle indicating the region RNG2 and the fan-shaped arc to the straight line indicating the direction of motion Dr1_k to the end point Qe1_k of the trajectory Tr1_k is Set in the same way as the corresponding element.

  By using the regions RNG1 and RNG2 shown in FIG. 7, even when the images IMG_k and IMG_k + 1 each include a lot of object trajectories, the trajectories to be collated by the collation unit 134 shown in FIG. 5 are narrowed down. be able to. Note that the shape of the region obtained by the predicting unit 133 is not limited to the sector shown in FIG. 7, and the direction of motion estimated for the first trajectory selected from the trajectories of a plurality of objects included in the previously captured image IMG_k. And any shape that reflects at least one of speed. For example, the prediction unit 133 is a circular region having a radius Ex_k centered on each of the end points Qs1_k and Qe1_k of the trajectory Tr1_k as a region including a predicted position in the image IMG_k + 1 of the object depicting the trajectory Tr1_k illustrated in FIG. An area may be obtained.

  For example, in the example of FIG. 7, the velocity vector collated with the velocity vector estimated for the object trajectory Tr1_k by the matching unit 134 is narrowed down to the velocity vector estimated for the object trajectories Tr1_k + 1 and Tr2_k + 1 included in the region RNG2. It is.

  Note that when the motion direction of the object indicated by each of the first trajectories extracted from the first image is already known, the prediction unit 133 illustrated in FIG. 5 performs the object in the direction opposite to the motion direction. The generation of the region including the position predicted when the is moved may be omitted. For example, when the motion of the object indicated by the trajectory Tr1_k illustrated in FIG. 7 is already determined to be the direction from the end point Q1s_k to the end point Q1e_k, the prediction unit 133 illustrated in FIG. Omits the generation of the region RNG1 shown in FIG. In addition, regarding the motion of the object indicated by the trajectory Tr1_k illustrated in FIG. 7, when it is already known that the direction of motion is the direction from the end point Q1e_k to the end point Q1s_k, the prediction unit 133 illustrated in FIG. Omits the generation of the region RNG2 shown in FIG.

  The collation unit 134 illustrated in FIG. 5 refers to the trajectory database DB, and stores the velocity vector corresponding to the trajectory Tr1_k illustrated in FIG. 7 and the trajectories Tr1_k + 1 and Tr2_k + 1. Get velocity vector. Next, the collation unit 134 determines the angles α1 and α2 formed by the motion direction Dr1_k indicated by the trajectory Tr1_k and the motion directions Dr1_k + 1 and Dr2_k + 1 indicated by the trajectories Tr1_k + 1 and Tr2_k + 1 based on the acquired velocity vector. Ask for each. For example, the collation unit 134 sets the angle α1 to the acute angle among the angles formed by the straight line indicated by the dotted line indicating the direction Dr1_k and the straight line indicated by the broken line indicating the direction Dr1_k + 1, and the obtained angle α1 is the threshold θ It is determined that the direction of motion is similar when: Similarly, the collating unit 134 sets the angle α2 as the acute angle among the angles formed by the straight line indicated by the dotted line indicating the direction Dr1_k and the straight line indicated by the alternate long and short dash line indicating the direction Dr2_k + 1, and the obtained angle α2 is When it is less than or equal to the threshold θ, it is determined that the direction of motion is similar.

  For example, when the angle α1 illustrated in FIG. 7 is equal to or smaller than the threshold θ, the collation unit 134 determines that the trajectory Tr1_k and the trajectory Tr1_k + 1 indicate the same object motion. On the other hand, when the angle α2 illustrated in FIG. 7 is larger than the threshold θ, the matching unit 134 determines that the trajectory Tr1_k and the trajectory Tr2_k + 1 indicate different object motions.

  Furthermore, the collation unit 134 collates the speed of the movement indicated by the trajectory Tr1_k with the speed of the movement indicated by each of the trajectories Tr1_k + 1 and Tr2_k + 1 based on the acquired velocity vector, and the two results are the same. It may be used to determine whether or not the object moves.

  For example, the matching unit 134 obtains a ratio between the speed of the motion indicated by the speed vector of the trajectory Tr1_k and the speed of the motion indicated by each speed vector of the trajectories Tr1_k + 1 and Tr2_k + 1, and the obtained ratio is within a predetermined range. In some cases, it is determined that the same object moves. Note that the predetermined range used for the determination based on the speed ratio is desirably set to a range of about 0.9 to 1.1, for example.

  As described above, the detection unit 131 including the prediction unit 133 and the collation unit 134 shows the individual trajectories in the collation between the trajectory Tr1_k extracted from the image IMG_k and the trajectories Tr1_k + 1 and Tr2_k + 1 extracted from the image IMG_k + 1. Use motion features. Thereby, for example, the detection unit 131 determines whether or not the image of the object photographed at a certain time and the image of the object photographed at another time indicate the same object based on the mutual positional relationship. Compared to the above, it is possible to detect a trajectory indicating the movement of the same object with high accuracy.

  Furthermore, the collation unit 134 collates the color of the image region indicating the trajectory Tr1_k with the color of the image region indicating each of the trajectories Tr1_k + 1 and Tr2_k + 1, and when the colors of the two collated trajectories are similar, It may be determined that the two verified trajectories indicate the movement of the same object.

  By referring to the trajectory database DB, the collation unit 134 can acquire the image extracted from the image IMG_k as the trajectory Tr1_k and the image extracted from the image IMG_k + 1 as the trajectories Tr1_k + 1 and Tr2_k + 1 illustrated in FIG. And the collation part 134 compares the color shown with the acquired image mutually, for example, acquires the collation result that the locus | trajectory Tr1_k and locus | trajectory Tr1_k + 1 shown in FIG. 7 are mutually similar colors. On the other hand, the collation unit 134 obtains a collation result that the color indicated by the trajectory Tr1_k is different from the color indicated by the trajectory Tr2_k + 1 by comparing the colors indicated in the acquired image with each other.

  In addition to the similarity of the direction and the speed of the motion, the similarity of the feature of the image showing the trajectory is considered in addition to the similarity of the direction and speed of the motion by collating the color of the image showing the trajectory together with the collation of the velocity vector. It is possible to detect a set of trajectories indicating the motion of the same object. For example, the detection unit 131 having the collation unit 134 that collates the colors of the images indicating the trajectory is different from the trajectory Tr1_k even when another trajectory having a similar velocity vector exists in the region RNG2 illustrated in FIG. The trajectory Tr1_k + 1 can be detected based on the similarity of the colors. Thereby, compared with the case where the similarity of the feature of the image which shows a locus | trajectory is not considered, the detection precision of the locus | trajectory which shows the motion of the same object by the detection part 131 can be improved.

  The collating unit 134 may execute the speed vector collation and the locus color collation described above in any order, or may execute both in parallel.

  A set of trajectories detected as the first trajectory and the second trajectory indicating the motion of the same object in the first image and the second image, which are the images IMG in which the imaging order is continuous, is shown by the detection unit 131 described above. The information is passed to the generation unit 132 shown in FIG.

  The generation unit 132 generates an interpolation trajectory that is a trajectory for interpolating between each of the plurality of first trajectories extracted from the first image and the second trajectory detected by the detection unit 131 for the first trajectory. .

  For example, the generation unit 132 refers to the trajectory database DB based on the information passed from the detection unit 131, and determines each end point of the trajectory detected as the first trajectory and the second trajectory indicating the motion of the same object. Get the indicated coordinates. And the production | generation part 132 produces | generates the interpolation locus | trajectory which interpolates between a 1st locus | trajectory and a 2nd locus | trajectory by calculating | requiring the curve which connects the acquired endpoint sequentially using a spline function etc., for example.

  FIG. 8 shows an example of an interpolation locus generated by the generation unit 132 shown in FIG. Of the elements shown in FIG. 8, elements equivalent to those shown in FIG. 7 are denoted by the same reference numerals, and description of the elements may be omitted.

  8 is generated by the generation unit 132 when the trajectory Tr1_k and the trajectory Tr1_k + 1 illustrated in FIG. 7 are detected as the first trajectory and the second trajectory of the same object by the detection unit 131 illustrated in FIG. An example of the interpolation trajectory Tri_k is shown.

  In FIG. 8, a curved line Ci_k indicated by a broken line indicates a curve obtained by performing spline interpolation on the end points Q1s_k, Q1e_k of the locus Tr1_k and the end points Q1s_k + 1, Q1e_k + 1 of the locus Tr1_k + 1.

  For example, the generation unit 132 expands the line width of the curve Ci_k illustrated in FIG. 8 and connects it to the contours of both the trajectory Tr1_k corresponding to the first trajectory and the trajectory Tr1_k + 1 corresponding to the second trajectory. In step 8, an interpolation trajectory Tri_k surrounded by a dotted line is generated.

  Then, the generation unit 132 interpolates the trajectory Tr1_k and the trajectory Tr1_k + 1 illustrated in FIG. 8 using the generated interpolation trajectory, thereby generating a trajectory Tr1c_k continuous from the end point Q1s_k of the trajectory Tr1_k to the end point Q1e_k + 1 of the trajectory Tr1_k + 1. Generate.

  The interpolating unit 13a having the detecting unit 131 and the generating unit 132 described above performs the motion of the same object in an image in which the imaging order is continuous even when each image IMG includes trajectories of a plurality of objects. It is possible to detect the set of trajectories shown and interpolate between the two detected trajectories.

  That is, the image processing apparatus 10 shown in FIG. 5 uses the trajectory of an object from images that are sequentially photographed over time regardless of the number of moving objects included in each image IMG photographed by the photographing apparatus GRD. Can be generated continuously.

  For example, the synthesis unit 14 sequentially adds the trajectory of the object extracted from the second and subsequent images IMG_2 to IMG_n to the first image IMG_1 passed from the extraction unit 11 and the interpolation trajectory generated by the interpolation unit 13a. Synthesize.

  Thereby, the composition unit 14 presents a trajectory that continuously indicates the motion of each object during the period in which n images IMG were captured by the image capturing device GRD, together with the background indicating the scenery around each object, and the like. A GSN can be generated. Further, the synthesis unit 14 may emphasize the trajectory of the object by expressing the contour of the continuous trajectory of the object in the composite image GSN with a color that makes the object stand out from the image around the trajectory.

  The image processing apparatus 10 disclosed herein can be realized using hardware such as a digital camera, a mobile terminal device having a camera function, a portable game machine, and the like, and a hardware such as a processor. .

  FIG. 9 shows a hardware configuration example of a digital camera to which the image processing apparatus 10 is applied. 9 that are equivalent to the components shown in FIG. 1 or FIG. 5 are denoted by the same reference numerals and description of the components may be omitted.

  A digital camera 20 shown in FIG. 9 includes a processor 21, a memory 22, a signal processing unit 23, an imaging control unit 24, an input device 25 such as an operation panel, a display device 26 such as a liquid crystal display, and a memory card. Slot 27. The processor 21, the memory 22, the signal processing unit 23, the imaging control unit 24, the input device 25, the display device 26, and the memory card slot 27 illustrated in FIG. 9 are connected to each other via a bus. Yes.

  The digital camera 20 further includes an imaging element SNC and an imaging optical system OPT. The imaging optical system OPT detects an object including a moving object MV included in the imaging range VW and a background around the object. Is formed on the light receiving surface of the image sensor SNC. The output signal of the image sensor SNC is input to the signal processor 23, and the signal processor 23 generates image data indicating the image of the subject from the output signal of the image sensor SNC. In addition, the imaging control unit 24 controls the focusing operation for focusing the lens included in the imaging optical system OPT on the subject and the exposure time during which light from the subject is incident on the imaging element SNC based on an instruction from the processor 21. To do.

  The processor 21 and the memory 22 illustrated in FIG. 9 are included in the image processing apparatus 10, and the imaging element SNC, the signal processing unit 23, the imaging control unit 24, and the imaging optical system OPT are illustrated in FIG. 1 or FIG. 5. It is included in the photographing device GRD.

  The memory card slot 27 can be loaded with a memory card 28, and reads information recorded on the memory card 28 and records information on the memory card 28.

  The memory 22 shown in FIG. 9 is an application program for the processor 21 to execute the process of generating an image showing the continuous trajectory of the object described with reference to FIGS. 5 to 8 together with the operating system of the digital camera 20. Is stored. Note that an application program for executing processing for generating an image showing a continuous trajectory of an object can be recorded and distributed in the memory card 28, for example. Then, by loading the memory card 28 in the memory card slot 27 and performing a reading process, an application program for executing a process for generating an image showing a continuous trajectory of the object may be stored in the memory 22. Good. In addition, an application program for processing for generating an image showing a continuous trajectory of an object is downloaded via a communication device (not shown) connected to a network such as the Internet, and the downloaded program is read into the memory 22. It can also be made.

  The processor 21 may fulfill the functions of the extraction unit 11, the estimation unit 12a, and the interpolation unit 13a illustrated in FIG. 5 by executing an application program stored in the memory 22. The trajectory database DB shown in FIG. 5 can be realized by using a part of the storage area of the memory 22.

  For example, when an instruction to start photographing the n images IMG_1 to IMG_n illustrated in FIG. 5 is input via the input device 25, the processor 21 indicates a continuous trajectory of an object from the images IMG_1 to IMG_n. A process of generating an image is executed. Further, prior to the execution of the process of generating an image showing a continuous trajectory of the object, the processor 21 sets the exposure time used for photographing the images IMG_1 to IMG_n with respect to the imaging control unit 24 shown in FIG. I do. As the exposure time, for example, the processor 21 sets a time during which a blur of several pixels or more is generated in an image of the moving object MV included in each image IMG and the output signal of the image sensor SNC is not saturated.

  FIG. 10 shows the operation of the image processing apparatus 10 shown in FIG. Each process of step S302 and step S311 to step S319 illustrated in FIG. 10 is an example of a process included in an application program for generating an image indicating a continuous trajectory of an object. Further, each processing of step S302 and steps S311 to S319 is executed by the processor 21 shown in FIG.

  In step S311, the processor 21 acquires the first image IMG_1 taken from the signal processing unit 23, and stores the acquired image IMG_1 in the memory 22 as a composite image in an initial state.

  In step S312, the processor 21 acquires a newly captured image IMG_k + 1 (k is a positive integer equal to or less than n−1) from the signal processing unit 23, and indicates a difference image indicating a difference between the acquired image IMG_k + 1 and the image IMG_k. To extract the trajectory of the object MV shown in FIG. In the process of step S312, the processor 21 includes a plurality of moving objects when each of the image IMG_k corresponding to the first image and the image IMG_k + 1 corresponding to the second image includes a plurality of moving objects. The trajectory corresponding to is extracted from the difference image. Further, the processor 21 stores image data representing each trajectory in a trajectory database DB provided in the memory 22 or the memory card 28 corresponding to information indicating each extracted trajectory.

  In step S302, the processor 21 estimates the feature of the motion of the object indicated by each trajectory based on the feature of the image indicating each trajectory extracted in the process of step S312. The processor 21, for example, as described with reference to FIG. 6, based on the correlation between the gradient of the luminance value in the image area indicating each extracted locus and the direction of the motion of the object depicting the locus, Estimate the direction of motion of the object and the speed of motion. Further, the processor 21 stores information indicating a velocity vector indicating the direction and speed of the motion estimated for each trajectory in the trajectory database DB in correspondence with information indicating each trajectory.

  In step S313, the processor 21 sequentially selects one of the trajectories extracted from the previous image IMG_k by the processing of step S312 and sets the selected trajectory as the first trajectory.

  In step S314, the processor 21 detects a trajectory indicating the same object motion as the first trajectory selected in step S312 from the trajectories extracted from the new image IMG_k + 1. The processor 21 detects each set of trajectories indicating the movement of the same object in two images IMG_k and IMG_k + 1 in which the imaging order is continuous by executing each process shown in FIG.

  FIG. 11 shows an example of processing for detecting a set of trajectories indicating the movement of the same object. Each process of step S321 to step S329 illustrated in FIG. 11 is an example of a process of detecting a set of trajectories indicating the movement of the same object illustrated in step S314 in FIG. In addition, each processing of step S321 to step S329 is executed by the processor 21 shown in FIG.

  In step S321, the processor 21 obtains a region including a position where an object moving along the first locus selected in the process of step S313 illustrated in FIG. 10 exists in the new image IMG_k + 1. The processor 21 refers to the trajectory database DB for the first trajectory, thereby acquiring the motion feature indicated by the first trajectory. Then, based on the direction and speed of the movement indicated by the acquired movement characteristics, the processor 21 moves the object moving along the first trajectory in the new image IMG_k + 1 as described with reference to FIG. The area including the existing position is obtained.

  In step S322, the processor 21 registers the correspondence between the trajectory extracted from the new image IMG_k + 1 in the process of step S312 shown in FIG. 10 and the trajectory in the previous image IMG_k in the process of step S327. Select a trajectory that is not. For example, the processor 21 sequentially selects and selects a trajectory in which information indicating a correspondence relationship with the trajectory included in the image IMG_k is not registered among trajectories registered in the trajectory database DB corresponding to the image IMG_k + 1. Information indicating the trajectory is obtained. The registration of the correspondence relationship between the trajectory included in image IMG_k and the trajectory included in image IMG_k + 1 will be described later in step S327.

  In step S323, the processor 21 determines whether or not the trajectory selected in the process in step S322 is included in the region obtained in the process in step S321.

  For example, when both of the end points registered in the trajectory database DB corresponding to the trajectory selected in step S322 are included in the region obtained in step S321 (Yes in step S323 (YES)), The processor 21 proceeds to the process of step S324.

  In step S324, the processor 21 determines whether or not the directions of motion indicated by the trajectory selected in the process of step S322 and the first trajectory selected in the process of step S313 illustrated in FIG. 10 are similar to each other. Determine whether.

  For example, the processor 21 is indicated by a velocity vector registered in the trajectory database DB corresponding to each of the trajectory selected in the process of step S322 and the first trajectory selected in the process of step S313 shown in FIG. Find the difference in direction of motion. Then, the processor 21 performs the determination process in step S324 by comparing the obtained difference with a threshold value (for example, ± 20 degrees) for determining similarity in direction, and the difference in the direction of motion is equal to or less than the threshold value. If (Yes in step S324 (YES)), the process proceeds to step S325.

  In step S325, the processor 21 determines whether or not the speeds of movement indicated by the trajectory selected in step S322 and the first trajectory selected in step S313 shown in FIG. 10 are equal to each other. Determine whether.

  For example, the processor 21 is indicated by a velocity vector registered in the trajectory database DB corresponding to each of the trajectory selected in the process of step S322 and the first trajectory selected in the process of step S313 shown in FIG. Find the ratio of the speed of movement. The processor 21 determines whether or not the obtained ratio is included in a range (e.g., 0.9 to 1.1) set for determining the similarity of the speed, whereby the determination process in step S325 is performed. I do. If the ratio of the speeds of movement is within the set range (Yes in step S325 (YES)), the processor 21 determines that both speeds are equal, and the process of step S326 is performed. move on.

  In step S326, the processor 21 determines the color of the area of the image IMG_k indicating the locus selected in the process of step S322 and the area of the image IMG_k indicating the first locus selected in the process of step S313 illustrated in FIG. It is determined whether the colors are similar to each other.

  The processor 21 represents, for example, an image area registered in the trajectory database DB corresponding to each of the trajectory selected in the process of step S322 and the first trajectory selected in the process of step S313 shown in FIG. Each representative color indicating a typical color is obtained. The processor 21 further includes each component of chromaticity coordinates indicating the representative color obtained for the locus selected in the process of step S322, and chromaticity coordinates indicating the representative color obtained for the first locus selected in the process of step S313. The ratio with each component of is obtained. Then, when all of the ratios obtained for the respective components are within a predetermined range (for example, 0.9 to 1.1) (affirmative determination in step S326 (YES)), the process of step S327 is performed. Proceed to

  The processor 21 executes the processes in steps S323 to S325 described above, thereby detecting a trajectory in which the direction and speed of movement are similar to each other in the two images IMG_k and IMG_k + 1 in which the shooting order is continuous. Can do. Further, by executing the process of step S326, the processor 21 can further detect a trajectory showing a similar color among trajectories similar in motion characteristics to the trajectory selected as the first trajectory. . Thereby, even when the trajectories of a large number of moving objects are included in each of the images IMG_k and IMG_k + 1, it is possible to detect a set of trajectories indicating the movement of the same object in the images IMG_k and IMG_k + 1.

  The processing order of the processes in steps S323 to S326 described above is not limited to the order shown in FIG. 11, and the processes in steps S323 to S326 may be executed in any order. If the determination is affirmative in all the processes in steps S323 to S326, the processor 21 proceeds to the process in step S327.

  In step S327, the processor 21 detects the trajectory selected in the process of step S322 as a second trajectory indicating the same object motion as the first trajectory selected in the process of step S313 shown in FIG. The correspondence relation of the trajectory is registered in the trajectory database DB. For example, the processor 21 uses the information registered in the trajectory database DB corresponding to the trajectory included in the image IMG_k + 1 detected as the first trajectory as information indicating the correspondence relationship between the two trajectories in the image IMG_k. Add identification information to indicate

  On the other hand, if the determination in step S323 to step S326 described above is negative (NO), the processor 21 proceeds to the process of step S328.

  In step S328, the processor 21 has not yet registered the correspondence relationship with the trajectory extracted from the image IMG_k among the trajectories extracted from the image IMG_k + 1 in the process of step S312 shown in FIG. Determine if there is no trajectory.

  For example, the processor 21 refers to the trajectory database DB, and information indicating the correspondence relationship with the first trajectory is not registered from the trajectories registered corresponding to the image IMG_k + 1, and is still selected in the process of step S322. Detects a trajectory that is not. When the information indicating the correspondence relationship with the first trajectory is unregistered and a trajectory that has not been selected in the process of step S322 is detected (affirmative determination in step S328 (YES)), the processor 21 proceeds to step S322. Return to the process.

  In this case, the processor 21 selects a trajectory that has not yet been selected from the trajectories registered in the trajectory database DB in association with the image IMG_k + 1 and whose correspondence with the first trajectory is not registered. Then, the processes after step S323 are executed for the selected locus.

  On the other hand, when there is no trajectory that is not yet selected in the process of step S322 in the trajectory that has not been registered with the first trajectory (No determination in step S328 (NO)), the processor 21 The process proceeds to step S329.

  In step S329, the processor 21 outputs a detection result indicating that the trajectory corresponding to the trajectory selected as the first trajectory in the process of step S313 illustrated in FIG. 10 is not included in the new image IMG_k + 1.

  After the process of step S327 or step S329 described above is completed, the processor 21 ends the process of detecting a set of trajectories indicating the movement of the same object, and proceeds to the process of step S315 shown in FIG.

  In step S315, as described with reference to FIG. 8, the processor 21 interpolates between the locus selected as the first locus in the process of step S313 and the locus detected as the second locus in the process of step S314. Generate an interpolation trajectory. The trajectory Tr1_k shown in FIG. 8 is an example of the first trajectory selected in the process of step S313, and the trajectory Tr1_k + 1 shown in FIG. 8 is an example of the second trajectory detected in the process of step S315. . Similarly, the interpolation trajectory Tri_k illustrated in FIG. 8 is an example of the interpolation trajectory generated by the process of step S316. Further, in the process of step S314 described with reference to FIG. 11, when a detection result indicating that the second trajectory corresponding to the trajectory selected as the first trajectory is not included in the image IMG_k + 1 is obtained, the processor 21 omits the processes of steps S315 and S316.

  In step S316, the processor 21 adds an image indicating the interpolation trajectory generated in step S315 and the trajectory detected as the second trajectory in step S314 to the composite image stored in the memory 22. As a result, the movement of the object MV includes the trajectories included in each of the two images in the order of photographing, such as the interpolated trajectory Trc_k shown in FIG. 3 and the interpolated trajectory Trc1_j shown in FIG. It is possible to generate a composite image showing a trajectory that continuously represents.

  In step S317, the processor 21 determines whether or not all the trajectories extracted from the image IMG_k in step S312 have been selected in step S313.

  If there is still a locus that has not been selected in the process of step S313 (No determination in step S317 (NO)), the processor 21 returns to the process of step S313, and steps S314 to S3 are performed for the newly selected locus. The process of S317 is executed.

  When the processes in steps S313 to S317 are completed for all the trajectories extracted from the image IMG_k in the process of step S312 (Yes in step S317 (YES)), the processor 21 proceeds to the process of step S318.

  In step S318, for example, the processor 21 determines whether or not shooting is being continued based on a signal indicating an instruction from the input device 25 illustrated in FIG.

  For example, when the signal from the input device 25 indicates that the release button provided on the input device 25 is being pressed, the processor 21 performs shooting by the user of the digital camera 20. Judge that continuation is instructed. In this case, the processor 21 proceeds to step S312 according to the affirmative determination (YES) route of step S318, and starts processing for a newly captured image.

  On the other hand, when the signal from the input device 25 indicates that the release button has been released, the processor 21 determines that the user of the digital camera 20 has instructed to stop shooting. In this case, the processor 21 proceeds to step S319 according to a negative determination (NO) route of step S318.

  In step S319, the processor 21 passes the composite image stored in the memory 22 to the display device 24 and displays it on the display screen included in the display device 24, thereby allowing the user of the digital camera 20 to continuously display the objects MV. A composite image including a simple trajectory. The composite image presented in the process of step S319 continuously represents the trajectory of the movement of the object MV from the start of shooting of the object MV to the end of shooting.

  That is, when the processor 21 executes the processing shown in FIGS. 10 and 11, the object MV is captured during the period from the start to the end of shooting in the process of discretely shooting the moving object MV with the digital camera 20. It is possible to generate a composite image that continuously represents the trajectory of the movement.

  10 and FIG. 11 is not limited to a processor mounted on a device including a shooting function such as the digital camera shown in FIG. 9, but is mounted on a computer device that does not include a shooting function. May be executed by a designated processor.

  FIG. 12 shows an example of the hardware configuration of the image processing apparatus 10. Note that, among the components shown in FIG. 12, components equivalent to those shown in FIG. 1 or FIG. 5 are denoted by the same reference numerals and description of the components may be omitted.

  The computer device 30 shown in FIG. 12 includes a processor 31, a memory 32, a hard disk device 33, a general-purpose interface 34, an input device 35, a display device 36, a network interface 37, and an optical drive 38. Yes. The processor 31, the memory 32, the hard disk device 33, the general-purpose interface 34, the input device 35, the display device 36, the network interface 37, and the optical drive 38 illustrated in FIG. 12 are connected to each other via a bus. Has been.

  The computer device 30 is connected to the photographing device GRD via the general-purpose interface 34, and is connected to a network NW such as the Internet via the network interface 37. The processor 31, the memory 32, the hard disk device 33, and the general-purpose interface 34 illustrated in FIG. 12 are included in the image processing apparatus 10.

  The input device 35 is, for example, a keyboard or a mouse, and is used for inputting an instruction from a user of the computer device 30. The display device 36 is a liquid crystal display device, for example, and is used for presenting images to the user.

  The optical drive 38 can be mounted with a removable disk RMD such as an optical disk, and reads information recorded on the mounted removable disk RMD and records information on the removable disk RMD.

  The memory 32 illustrated in FIG. 12 stores an application program for the processor 31 to execute processing for generating an image indicating the continuous trajectory of the object illustrated in FIGS. 10 and 11 together with the operating system of the computer device 30. doing. Note that an application program for executing processing for generating an image showing a continuous trajectory of an object can be recorded and distributed, for example, on a removable disk RMD. Then, by loading the removable disk RMD into the optical drive device 38 and performing reading processing, an application program for executing processing for generating an image showing a continuous trajectory of the object may be stored in the memory 32. Good. In addition, an application program for processing for generating an image showing a continuous trajectory of an object can be downloaded via the network interface 37 from the server device SV connected to the network NW. Then, the downloaded program may be read into the memory 32 or the hard disk device 33.

  Then, the processor 31 performs the functions of the extraction unit 11, the estimation unit 12a, and the interpolation unit 13a illustrated in FIG. 5 by executing the application program stored in the memory 32. Further, the trajectory database DB shown in FIG. 5 can be realized by using a part of the storage area of the memory 32 or the hard disk device 33.

  For example, the processor 31 captures image data indicating the images IMG_1 to IMG_n from the photographing device GRD via the general-purpose interface 34 after the photographing of the n images IMG_1 to IMG_n illustrated in FIG. get. The processor 31 stores the acquired image data in the hard disk device 33 or the like. Further, the processor 31 sequentially reads out the accumulated image data, and executes the processing in steps S312 to S317 shown in FIG. 10 based on the read-out image, thereby obtaining an image showing a continuous trajectory of the object. Generate.

  The image processing apparatus 10 illustrated in FIG. 12 captures a process of generating an image indicating a continuous locus of an object from a plurality of images obtained by discretely capturing a moving object by a processor 31 mounted on the computer apparatus 30. By executing the processing independently of the processing, the processing speed can be increased.

  Further, the process of generating an image showing a continuous trajectory of an object can be provided as a cloud service via a network such as the Internet.

  FIG. 13 shows another example of the hardware configuration of the image processing apparatus 10 shown in FIG. Note that, among the components shown in FIG. 13, components equivalent to those shown in FIG. 1 or FIG. 5 are denoted by the same reference numerals and description of the components may be omitted.

  The server device 40 illustrated in FIG. 13 includes a processor 41, a memory 42, a hard disk device 43, a network interface 44, and an optical drive 45. The processor 41, the memory 42, the hard disk device 43, the network interface 44, and the optical drive 45 illustrated in FIG. 13 are connected to each other via a bus.

  The server device 40 is connected to a network NW such as the Internet via a network interface 44. The processor 41, the memory 42, the hard disk device 43, and the network interface 44 illustrated in FIG. 13 are included in the image processing apparatus 10.

  The optical drive 45 can be mounted with a removable disk RMD such as an optical disk, and reads information recorded on the mounted removable disk RMD and records information on the removable disk RMD.

  The server device 40 is connected to a mobile terminal UE possessed by a service user using a process of generating an image showing a continuous trajectory of an object via the network NW.

  The mobile terminal UE is, for example, a smartphone, and has a camera function and a wireless network connection function. Then, a plurality of images taken by the user using the camera function of the mobile terminal UE are passed to another server device SVa connected to the network NW via the network NW, and further, image storage for storing images Accumulated in the device Gac. The images stored in the image storage device Gac shown in FIG. 13 correspond to the n images IMG_1 to IMG_n shown in FIG.

  The memory 42 illustrated in FIG. 13 stores an application program for the processor 41 to execute processing for generating an image indicating the continuous trajectory of the object illustrated in FIGS. 10 and 11 together with the operating system of the server device 40. doing. Note that an application program for executing processing for generating an image showing a continuous trajectory of an object can be recorded and distributed, for example, on a removable disk RMD. Then, by loading the removable disk RMD into the optical drive device 45 and performing reading processing, an application program for executing processing for generating an image showing a continuous trajectory of the object may be stored in the memory 42. Good. In addition, an application program for processing for generating an image indicating a continuous trajectory of an object can be downloaded from the server device SVa connected to the network NW via the network interface 44. Then, the downloaded program may be read into the memory 42 or the hard disk device 43.

  The processor 41 performs the functions of the extraction unit 11, the estimation unit 12a, and the interpolation unit 13a illustrated in FIG. 5 by executing the application program stored in the memory 42. Further, the trajectory database DB shown in FIG. 5 can be realized by using a part of the storage area of the memory 42 or the hard disk device 43.

  For example, when a service is requested from the mobile terminal UE, the processor 41 displays a plurality of images specified in the service request from the images stored in advance in the image storage device Gac via the server device SVa. get. The processor 41 stores the acquired image data in the hard disk device 33 or the like as the images IMG_1 to IMG_n illustrated in FIG. Further, the processor 41 sequentially reads the stored image data, and executes the processing of steps S312 to S317 shown in FIG. 10 based on the read image, thereby obtaining an image showing a continuous trajectory of the object. Generate.

  The image processing apparatus 10 illustrated in FIG. 13 can provide a service for generating an image indicating a continuous locus of an object from a plurality of images obtained by discretely capturing moving objects via the network NW.

  From the above detailed description, features and advantages of the embodiment will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and changes. Therefore, there is no intention to limit the scope of the inventive embodiments to those described above, and appropriate modifications and equivalents included in the scope disclosed in the embodiments can be used.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
A first image and a second image obtained by sequentially photographing an object are input, and blurring of the object included in the first image is extracted as a first trajectory of the object and included in the second image. An extraction unit that extracts a blur of the object as a second locus of the object;
Based on the characteristics of the images of the first trajectory and the second trajectory extracted by the extraction unit, characteristics of the motion of the object at the time when each of the first image and the second image is captured An estimation unit for estimating
Based on the motion characteristics of the object estimated by the estimation unit, a trajectory of the object is obtained between the time when the first image is captured and the time when the second image is captured, and the determined trajectory is used. An interpolation unit for interpolating a trajectory between the first trajectory and the second trajectory;
An image processing apparatus comprising:
(Appendix 2)
In the image processing apparatus according to attachment 1,
The estimation unit includes
A direction in which the gradient of the pixel value indicating the color or luminance of each pixel included in each of the first trajectory and the second trajectory is smaller than the other directions is obtained, and the obtained directions are represented by the first image and the second trajectory. An image processing apparatus, characterized in that the direction of movement of the object, which is one of the characteristics of the movement of the object at the time when each of the images was taken.
(Appendix 3)
In the image processing apparatus according to appendix 1 or appendix 2,
A plurality of objects are included in each of the first image and the second image, and the first locus and the second locus of each of the plurality of objects from the first image and the second image by the extraction unit. Is extracted,
The interpolation unit
For each first trajectory extracted from the first image by the extraction unit, the first trajectory extracted from the first image from among the plurality of second trajectories extracted from the second image. A detection unit that detects one second trajectory in which a motion feature similar to the motion feature estimated by the estimation unit is estimated as a second trajectory corresponding to the first trajectory;
A generation unit configured to generate an interpolation trajectory for interpolating between each of the plurality of first trajectories extracted from the first image and each of the second trajectories detected corresponding to the first trajectory; An image processing apparatus.
(Appendix 4)
In the image processing device according to attachment 3,
The detector is
One of the plurality of second trajectories extracted from the second image by the extraction unit corresponds to one second trajectory having characteristics similar to those of the image of the first trajectory. An image processing apparatus, wherein the second locus is detected.
(Appendix 5)
In the image processing device according to attachment 3,
The detector is
A prediction unit that obtains a region including a position where the object is predicted to be present in the second image based on a motion direction and speed included in a motion characteristic estimated for the first trajectory; ,
From the second trajectory included in the region obtained by the prediction unit, the second trajectory in which a motion feature similar to the motion feature estimated for the first trajectory is estimated is obtained as the second trajectory. An image processing apparatus that detects a second locus corresponding to one locus.
(Appendix 6)
In response to the input of the first image and the second image obtained by sequentially photographing the object, the blur of the object included in the first image is extracted as the first trajectory of the object, and the second image is extracted. Extracting the blur of the contained object as a second trajectory of the object;
Based on the extracted image characteristics of the first trajectory and the second trajectory, the motion characteristic of the object at the time when the first image and the second image are captured is estimated. ,
Based on the estimated motion characteristics of the object, a trajectory of the object from the first image is captured until the second image is captured, and the first trajectory is used to determine the trajectory of the object. Interpolating a trajectory between one trajectory and the second trajectory;
An image processing method.

DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus; 11 ... Extraction part; 12, 12a ... Estimation part; 13, 13a ... Interpolation part; 14 ... Synthesis part; 121 ... Calculation part; 122 ... Identification part; 133 ... Prediction unit; 134 ... Collation unit; 20 ... Digital camera; 21, 31, 41 ... Processor; 22, 32, 42 ... Memory; 23 ... Signal processing unit; 24 ... Imaging control unit; 26, 36 ... display device; 27 ... memory card slot; 28 ... memory card; 33, 43 ... hard disk device; 34 ... general-purpose interface; 37, 44 ... network interface; 38 ... optical drive device; GRD ... photographing device; Synthesizer; DB ... locus database; RMD ... removable disk; NW ... network; SNC ... imaging device; OPT ... imaging optical system; , SVa ... server; Gac ... image storage apparatus; UE ... portable terminal

Claims (5)

A first image and a second image obtained by sequentially photographing an object are input, and blurring of the object included in the first image is extracted as a first trajectory of the object and included in the second image. An extraction unit that extracts a blur of the object as a second locus of the object;
Based on the characteristics of the images of the first trajectory and the second trajectory extracted by the extraction unit, characteristics of the motion of the object at the time when each of the first image and the second image is captured An estimation unit for estimating
Based on the motion characteristics of the object estimated by the estimation unit, a trajectory of the object is obtained between the time when the first image is captured and the time when the second image is captured, and the determined trajectory is used. An interpolation unit for interpolating a trajectory between the first trajectory and the second trajectory;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The estimation unit includes
A direction in which the gradient of the pixel value indicating the color or luminance of each pixel included in each of the first trajectory and the second trajectory is smaller than the other directions is obtained, and the obtained directions are represented by the first image and the second trajectory. An image processing apparatus, characterized in that the direction of movement of the object, which is one of the characteristics of the movement of the object at the time when each of the images was taken. The image processing apparatus according to claim 1 or 2,
A plurality of objects are included in each of the first image and the second image, and the first locus and the second locus of each of the plurality of objects from the first image and the second image by the extraction unit. Is extracted,
The interpolation unit
For each first trajectory extracted from the first image by the extraction unit, the first trajectory extracted from the first image from among the plurality of second trajectories extracted from the second image. A detection unit that detects one second trajectory in which a motion feature similar to the motion feature estimated by the estimation unit is estimated as a second trajectory corresponding to the first trajectory;
A generation unit configured to generate an interpolation trajectory for interpolating between each of the plurality of first trajectories extracted from the first image and each of the second trajectories detected corresponding to the first trajectory; An image processing apparatus. The image processing apparatus according to claim 3.
The detector is
One of the plurality of second trajectories extracted from the second image by the extraction unit corresponds to one second trajectory having characteristics similar to those of the image of the first trajectory. An image processing apparatus, wherein the second locus is detected. In response to the input of the first image and the second image obtained by sequentially photographing the object, the blur of the object included in the first image is extracted as the first trajectory of the object, and the second image is extracted. Extracting the blur of the contained object as a second trajectory of the object;
Based on the extracted image characteristics of the first trajectory and the second trajectory, the motion characteristic of the object at the time when the first image and the second image are captured is estimated. ,
Based on the estimated motion characteristics of the object, a trajectory of the object from the first image is captured until the second image is captured, and the first trajectory is used to determine the trajectory of the object. Interpolating a trajectory between one trajectory and the second trajectory;
An image processing method.

Images