JP6476414B2 - Still image extraction method and image processing apparatus for realizing the method - Google Patents

Description

  The present technology relates to a still image extraction method for extracting a plurality of frames as a still image from a moving image stream, and an image processing apparatus that implements the method.

  In order to obtain data around the entire object, the object is photographed as a subject from each position around the object. Here, it is desirable that the positions of the cameras (hereinafter referred to as shooting positions) where shooting is performed in order to interactively display and browse still images around the entire object are equally spaced.

  Conventionally, in order to obtain a still image at each shooting position described above, a still image is shot at each shooting position in order (see Patent Document 1). That is, the processing operation in which the camera is fixed at the shooting position and the camera shoots at the shooting position is repeatedly performed at each shooting position while moving the camera.

JP 2007-72537 A

  The present technology provides a still image extraction method and an image processing apparatus that can extract a still image at an appropriate shooting position at an equal distance from each other in a short time.

The still image extraction method of the present technology moves the camera relatively in one direction on the circumference or straight line centered on the subject, and captures the shooting time at equal intervals from a moving image stream that is continuously captured in time. An operation for extracting a plurality of first frames, an operation for specifying a shooting position at each shooting time of each of the plurality of first frames, and a shooting for each of the plurality of first frames. Based on the time and the shooting position, an operation for estimating the shooting time of each of a plurality of frames obtained by shooting at a plurality of shooting positions at equal distances, and a plurality of first times corresponding to each of the estimated plurality of shooting times A still image extraction method comprising: extracting each of the two frames as a still image from a moving image stream.

  According to the present technology, it is possible to extract a still image at an appropriate shooting position at an equal interval in a short time.

It is a lineblock diagram showing an image processing system in an embodiment of this art. FIG. 5 is an explanatory diagram for describing a method for photographing a subject in a photographing unit according to an embodiment. It is a flowchart which shows operation | movement of the image processing system 1 in embodiment. It is a flowchart which shows operation | movement of the still image extraction method in embodiment. It is a flowchart which shows the identification method of the imaging position of the camera 21 in embodiment. It is explanatory drawing which shows the initial sampling point in embodiment. It is a flowchart which shows operation | movement of the method of estimating the time stamp of the resampling point in embodiment. It is explanatory drawing which shows the relationship between the time stamp of the initial sampling point and the angle from a reference point in embodiment. It is explanatory drawing which shows the resampling point in embodiment. It is explanatory drawing which shows the method of estimating each time stamp of the some flame | frame obtained by imaging | photography at a resampling point. It is a flowchart which shows operation | movement of the still image extraction method by other embodiment by this technique. It is explanatory drawing which shows the imaging | photography method of the to-be-photographed object by other embodiment of this technique.

  Hereinafter, an image processing apparatus according to an embodiment of the present technology will be described with reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

  The inventor provides the accompanying drawings and the following description in order for those skilled in the art to fully understand the present technology, and is not intended to limit the subject matter described in the claims. Absent.

[1-1. Constitution]
FIG. 1 is a configuration diagram illustrating an image processing system according to an embodiment of the present technology.

  The image processing system 1 according to the present embodiment acquires a moving image stream by continuously shooting a subject while moving the camera relative to the subject, and each of a plurality of frames from the moving image stream. It is a system that extracts as a still image. The image processing system 1 includes an image processing device 10, a photographing unit 20, a control unit 30, a photographed image storage unit 40, and a multi-viewpoint image database (DB) 50.

  In the image processing system 1 according to the present embodiment, the moving image stream is generated by shooting at a frame rate of, for example, 60 fps (frame per second), and is shot at regular intervals, that is, every 1/60 seconds. It includes a plurality of frames obtained by shooting. Each frame has a time stamp to which “1” is added every time it is generated by photographing. The time stamp indicates the shooting time of the frame to which the time stamp is attached. For example, when the frame rate of the moving image stream is 60 fps, the time stamp one second after the start of moving image shooting indicates a value obtained by adding “60” to the time stamp at the start of moving image shooting. When the time stamp at the start of moving image shooting is “10”, the time stamp one second after the start of moving image shooting is “70”.

  In the present embodiment, the moving image stream is a moving image generated from frames shot at equal time intervals, but is not limited thereto. For example, it may be a collection of still images taken continuously at equal time intervals. For example, an image ID is associated with still images continuously shot at equal time intervals, and the image ID can be handled in the same manner as a time stamp. Therefore, in this embodiment, the time stamp is used in the still image extraction method, but in the case of a collection of still images in which the moving image stream is continuously shot at equal time intervals, the image ID is used in the still image extraction method. .

  The image processing apparatus 10 performs an operation of extracting a plurality of frames as still images from the moving image stream stored in the captured image storage unit 40. The still images of a plurality of frames are images shot at a plurality of shooting positions that are separated at equal intervals. The moving image stream is obtained by shooting the subject continuously in time while moving the camera relative to the subject. For example, the image processing apparatus 10 captures 36 images that form angles at equal intervals of 10 degrees from a moving image stream obtained by shooting with a camera moving on a circle around a subject. Extract still images taken at the location.

  The image processing apparatus 10 includes an image extraction unit 11 and a shooting position calculation unit 12.

  The image extraction unit 11 extracts a plurality of first frames included in the moving image stream, and acquires shooting times of the extracted plurality of first frames. In addition, the image extraction unit 11 performs an initial sampling operation and a resampling operation.

  The shooting position calculation unit 12 specifies information regarding the shooting position of the camera at the shooting time acquired in each of the plurality of first frames from the plurality of first frames. The shooting position calculation unit 12 is a camera at a plurality of shooting positions at equal intervals based on the shooting time acquired in each of the first frames and information on the shooting position specified in each of the first frames. The operation of estimating the photographing time of each of a plurality of frames obtained by photographing with the above is performed.

  In the initial sampling, the image extraction unit 11 extracts a plurality of first frames included in the moving image stream, and obtains time stamps indicating the photographing times of the extracted plurality of first frames. The first frame extracted by the initial sampling is called an initial sampling frame. At this time, the image extraction unit 11 extracts a plurality of initial sampling frames so that the acquired time stamps are equally spaced.

  In the re-sampling, the image extraction unit 11 extracts a second frame corresponding to each shooting time (time stamp) estimated by the shooting position calculation unit 12 from the moving image stream. The second frame extracted from the moving image stream is extracted as a plurality of frames obtained by shooting at a plurality of shooting positions that are spaced at equal intervals. Hereinafter, the second frame extracted by resampling by the image extraction unit 11 is referred to as a resampling frame. For example, the image extraction unit 11 extracts 36 resampling frames as frames obtained by shooting with a camera at shooting positions at equal intervals of 10 degrees.

  The shooting position calculation unit 12 specifies the shooting position of the camera from the time stamp of each initial sampling frame. That is, the shooting position calculation unit 12 can specify the position of the camera relative to the subject when the subject is shot. Hereinafter, the shooting position specified by each initial sampling frame is referred to as an initial sampling point.

  The shooting position calculation unit 12 further estimates time stamps of a plurality of frames obtained by shooting at a plurality of shooting positions based on the initial sampling points and the time stamps acquired by the initial sampling. Hereinafter, the imaging position specified by each resampling frame is referred to as a resampling point.

  FIG. 2 is an explanatory diagram for explaining a method of photographing a subject in the photographing unit according to the embodiment.

  As shown in FIG. 2, the photographing unit 20 includes a camera 21, a robot arm 22, and a turntable 23. The turntable 23 is a table on which the subject 24 is placed and rotates, and rotates about the rotation axis of the turntable 23 (Z axis in FIG. 2) as the rotation center. Note that the rotation speed of the turntable 23 is not necessarily a constant speed due to the influence of the position of the center of gravity of the subject 24 placed on the turntable 23 or the accuracy of the mechanical structure of the turntable 23. The camera 21 generates a moving image stream of the rotating subject 24 by photographing the subject 24 placed on the turntable 23. That is, the camera 21 generates a moving image stream by shooting the subject 24 while moving relative to the subject 24 on the circumference around the subject 24. The robot arm 22 is attached with a camera 21 and enables the camera 21 to move in the circumferential direction on the XZ plane of FIG.

  The subject 24 is an object that is placed on the rotation center of the turntable 23 and photographed by the camera 21. The subject 24 is extracted by the image processing apparatus 10 and is a still image of the subject 24 obtained by photographing with the camera 21 at a plurality of photographing positions whose distances are equally spaced (that is, at equally spaced angles). For example, it is expressed by three-dimensional computer graphics (CG).

  In this embodiment, the turntable 23 rotates to cause the camera 21 to photograph the subject 24 while moving relative to the subject 24. However, the present invention is not limited to this. For example, the subject 24 may be photographed while the camera 21 actually moves on the circumference around the subject 24.

  In the present embodiment, the camera 21 captures the subject 24 while moving relative to the subject 24 on the circumference around the subject 24. However, the present invention is not limited to this. For example, the subject 24 may be photographed while the camera 21 moves along a straight line. Thereby, each of the re-sampling frames may be extracted as a plurality of frames obtained by photographing with the camera 21 at a plurality of photographing positions at equal intervals.

  The control unit 30 controls photographing by the photographing unit 20. Specifically, the control unit 30 controls the start and end of moving image shooting by the camera 21, the rotation of the turntable 23, and the movement of the camera 21 by the robot arm 22. As a result, the control unit 30 rotates the turntable 23 to cause the camera 21 to photograph the subject 24 from a position on the circumference centered on the subject 24 that is parallel to the table surface of the turntable 23. Further, the control unit 30 controls the robot arm 22 to move the camera 21 in the circumferential direction on the XZ plane of FIG. That is, the control unit 30 repeatedly controls the rotation of the turntable 23 and the movement of the camera 21 by the robot arm 22 to cause the camera 21 to photograph the subject 24 from a position on the spherical surface centering on the subject 24. .

  The captured image storage unit 40 is, for example, a nonvolatile memory that stores the moving image stream generated by the imaging unit 20. In addition, the captured image storage unit 40 stores a time stamp indicating the shooting time of each frame included in the moving image stream.

  The multi-viewpoint image DB 50 is configured by, for example, a non-volatile memory, and is extracted by the image processing apparatus 10 and obtained by shooting with the camera 21 at a plurality of shooting positions whose distances are estimated at equal intervals. Are stored as still images.

[1-2. Operation]
An operation of the image processing system 1 in the present embodiment configured as described above will be described.

  FIG. 3 is a flowchart showing the operation of the image processing system 1 in the embodiment.

  First, as illustrated in FIG. 3, the photographing unit 20 photographs the subject 24 under the control of the control unit 30 (step S201). Specifically, the control unit 30 controls the rotation of the turntable 23, so that the camera 21 captures the subject 24 placed on the turntable 23 from around 360 degrees of the subject 24. Further, when the control unit 30 controls the movement of the camera 21 by the robot arm 22, the camera 21 moves in a circumferential direction around the subject 24 on a plane perpendicular to the table surface of the turntable 23. The camera 21 again captures the subject 24 from around 360 degrees around the subject 24 from the moved position. The control unit 30 repeatedly controls the rotation of the turntable 23 and the movement of the camera 21 by the robot arm 22, so that the camera 21 generates a moving image stream in which the subject 24 is captured from a position on the spherical surface centered on the subject 24. To do. The generated moving image stream is stored in the captured image storage unit 40. At this time, a time stamp indicating the shooting time of each frame included in the moving image stream is also stored. That is, a plurality of frames included in the moving image stream and time stamps corresponding to the plurality of frames are stored in the captured image storage unit 40.

  Next, the image processing apparatus 10 stores a plurality of frames (resampling frames) obtained by photographing with the camera 21 at a plurality of photographing positions (resampling points) whose distances are estimated to be equal intervals. Extracted from the moving image stream stored in the unit 40 (step S202).

  Finally, the image processing apparatus 10 stores the extracted plurality of resampled frames as still images in the multi-viewpoint image DB 50 (step S203).

  FIG. 4 is a flowchart illustrating the operation of the still image extraction method according to the embodiment. FIG. 4 is a flowchart showing in detail the process of step S202 of FIG.

  First, the image extraction unit 11 extracts a plurality of initial sampling frames from the moving image stream stored in the captured image storage unit 40, and obtains a time stamp corresponding to each of the extracted plurality of initial sampling frames. Is performed (step S301). For example, the image extraction unit 11 extracts eight initial sampling frames, and acquires eight time stamps corresponding to the extracted eight initial sampling frames. Here, the extracted eight initial sampling frames are frames in which eight time stamps corresponding to the initial sampling frames are equally spaced. That is, the image extraction unit 11 extracts initial sampling frames at regular time intervals. Note that the number of initial sampling frames is not limited to eight, but may be two to seven, or nine or more. By increasing the number of initial sampling frames, the accuracy of estimation of time stamps of a plurality of frames obtained by photographing by the camera 21 at a re-sampling point described later is increased.

  Next, the shooting position calculation unit 12 specifies the shooting position (initial sampling point) of the camera 21 at the time stamp of each of the eight initial sampling frames by the image extraction unit 11 (step S302).

  Further, the photographing position calculation unit 12 uses the camera 21 at the re-sampling point based on the identified eight initial sampling points and the time stamps of the eight initial sampling frames acquired by the image extraction unit 11. Time stamps of a plurality of frames obtained by photographing are estimated (step S303).

  Finally, the image extracting unit 11 extracts a resampled frame from the moving image stream stored in the photographed image storage unit 40 as each of a plurality of frames obtained by photographing by the camera 21 at the resampling point. Is performed (step S304).

  FIG. 5 is a flowchart showing a method for specifying the shooting position of the camera 21 in the embodiment. FIG. 5 is a flowchart showing in detail the process in step S302 of FIG.

  First, the shooting position calculation unit 12 extracts feature points (feature amounts) of the eight initial sampling frames (step S401). The feature points are extracted by SIFT or Harris corner detection or the like. Note that the feature point extraction method is not limited to these methods.

  Next, the shooting position calculation unit 12 calculates corresponding points between the frames by matching the feature points of the extracted eight initial sampling frames (step S402).

  Finally, the shooting position calculation unit 12 specifies a shooting position from each of the corresponding points between the calculated frames by using an SFM (Structure From Motion) algorithm (step S403).

In this way, the initial sampling point that is the shooting position of the camera 21 at the time stamp of each of the eight initial sampling frames is specified. The initial sampling point is represented by coordinates (X _i , Y _i , Z _i ) where the origin is arbitrary, for example. In the present embodiment, since eight initial sampling frames are extracted by the initial sampling, i = 0 to 7.

FIG. 6 is an explanatory diagram showing initial sampling points in the embodiment. S ₀ (X ₀ , Y ₀ , Z ₀ ) to S ₇ (X ₇ , Y ₇ , Z ₇ ) shown in FIG. 6 are each initial sampling point and its coordinates. In the present embodiment, the image extraction unit 11 extracts eight initial sampling frames at regular time intervals. As shown in FIG. 6, the initial sampling points do not exist at positions at equal distances. That is, the image extraction unit 11 extracts eight initial sampling frames so that the time stamps acquired by the initial sampling are equally spaced, but the initial sampling points of the extracted eight initial sampling frames have the same distance. There will be no gaps. This is because the turntable 23 does not rotate at a constant speed. Therefore, even when the image extraction unit 11 extracts a plurality of initial sampling frames so that the time stamps acquired by the initial sampling are equally spaced, the initial sampling point is a photographing position with incomplete positioning.

  FIG. 7 is a flowchart showing the operation of the method for estimating the time stamp of the re-sampling point in the embodiment. FIG. 7 is a flowchart showing in detail the processing in step S303 in FIG.

  First, the photographing position calculation unit 12 calculates the position of the rotation center of the turntable 23 on which the subject 24 is placed (step S501). Specifically, the position of the rotation center of the turntable 23 is calculated by the least square method using the following equations 1-7.

  The following expression 1 is an expression showing the relationship between the center point of the sphere and the radius.

X _i , Y _i , and Z _i shown in Equation 1 are the x component, y component, and z component of the coordinates of the initial sampling point, respectively. X, Y, and Z are the x component, y component, and z component of the coordinates of the rotation center of the turntable 23, respectively. R is the distance between the rotation center of the turntable 23 and the initial sampling point. That is, r is a radius of a sphere with the rotation center of the turntable 23 as the center.

  The following formulas 2 and 3 are formulas showing the accumulated square error of the difference between the left side and the right side of formula 1.

Here, assuming that A = −2X, B = −2Y, C = −2Z, and D = X ² + Y ² + Z ² −r ² , Expression 2 can be summarized as the following Expression 3.

The following equations 4 to 7 are equations when the value obtained by partial differentiation of equation 3 with variables A to D becomes zero. When the value obtained by partial differentiation of Equation 3 with variables A to D is 0, the center point and radius of the sphere that minimizes the cumulative square error shown in Equation 3 are calculated. Specifically, the coordinates (X ₀ , Y ₀ , Z ₀ ) to (X ₇ , Y ₇ , Z ₇ ) of the initial sampling point are substituted into (X _i , Y _i , Z _i ) in the following expressions 4 to 7: Thus, the coordinates of the rotation center of the turntable 23 are calculated.

  In this way, the position of the rotation center of the turntable 23 is calculated.

  In addition, although the coordinate of the rotation center of the turntable 23 was calculated using the least square method as shown in Formulas 1-7, it is not restricted to this. For example, a robust estimation method adapted to a RANSAC (RANdom Sample Consensus) algorithm may be used.

Next, the imaging position calculation unit 12 uses one of the initial sampling points as a reference point, a vector from the rotation center of the turntable 23 to the reference point, and initial sampling that excludes the reference point from the rotation center of the turntable 23. The angle between the point and the vector is calculated (step S502). For example, the point S _b a S ₀ points indicated in Figure 6 as a reference point. Further, when the rotation center of the turntable 23 is the origin O, the coordinates of the point S _b are (x _b , y _b , z _b ), and the coordinates of the other seven initial sampling points are (x _i , y _i , z _i ) (i = 1 to 7). It calculates a vector from the rotation center O of the turn table 23 to the reference point S _b, the angle theta _i between the vector from the rotation center O of the turntable 23 to the other seven initial sampling points S _i, respectively The equations are shown in the following equations 8 and 9.

  In this way, the position of the initial sampling point is calculated as an angle from the reference point. In addition, the image extraction unit 11 acquires time stamps for each of the eight initial sampling frames (step S301). Therefore, the angle of the eight initial sampling points from the reference point corresponds to the time stamp of each of the eight initial sampling frames.

  Next, the shooting position calculation unit 12 estimates time stamps of a plurality of frames obtained by shooting with the camera 21 at the re-sampling point. (Step S503).

FIG. 8 is an explanatory diagram showing the relationship between the time stamp of the initial sampling point and the angle from the reference point in the embodiment. The time stamp s _i of each initial sampling point corresponds to the angle θ _i from the reference point.

In FIG. 8, the angle θ _i from the reference point is an example of the angle calculated by the imaging position calculation unit 12 in step S502. In the present embodiment, the image extraction unit 11 extracts initial sampling frames at intervals of 1 second, and the frame rate of the moving image stream is set to 60 fps. Therefore, the time stamp s _i is set every 60 (1 second). The time stamps are equally spaced. On the other hand, the angles θ _i from the reference point are not equally spaced. This is because the turntable 23 does not rotate at a constant speed as described above. Therefore, even when the image extraction unit 11 extracts a plurality of initial sampling frames so that the time stamps are equally spaced, the initial sampling point is a photographing position with incomplete positioning.

  Although the frame rate of the moving image stream is 60 fps, the present invention is not limited to this. The frame rate of the moving image stream may be smaller than 60 fps, for example, 30 fps, or larger than 60 fps, for example, 120 fps.

FIG. 9 is an explanatory diagram showing re-sampling points in the embodiment. As shown in FIG. 9, the shooting position calculation unit 12 obtains time stamps of 16 frames obtained by shooting with the camera 21 at re-sampling points at equal intervals of 22.5 degrees, for example. presume. 9, the point _{T x} is (x = 0 to 15), a re-sampling point.

  In the present embodiment, there are 16 re-sampling points, but the present invention is not limited to this. There may be fewer than 16 re-sampling points, for example, there will be 8 re-sampling points at equally spaced angles every 45 degrees. Further, the number of re-sampling points may be larger than 16, and for example, there are 36 re-sampling points at equal intervals of every 10 degrees.

The shooting position calculation unit 12 estimates a time stamp t _{x of a} frame obtained by shooting with the camera 21 at the re-sampling point T _x . Specifically, the angle theta _m and theta _m-1 from the initial sampling point reference point _{S m} and _{S m-1} in the vicinity of the re-sampling point _{T x,} associated with the time stamp _{s m} and _{s m-1} from the time stamp t _x of frames obtained by photographing by the camera 21 at the resampling point T _x it is estimated. That is, the linear sampling based on the angles θ _m and θ _m−1 from the reference points of the initial sampling points S _m and S _m−1 and the time stamps s _m and s _m−1 can be performed at the re-sampling point T _x . The time stamp t _{x of the} frame obtained by photographing with the camera 21 is estimated. In the present embodiment, since there are 16 re-sampling points, x = 0 to 15. Since the initial sampling points are 8, m = 1 to 7. The following expression 10 is used to calculate the frame time stamp t _x obtained by photographing with the camera 21 at the re-sampling point T _x .

  In Equation 10, θ represents the angle between re-sampling points that are equally spaced apart. That is, as shown in FIG. 9, in this embodiment, the time stamps of a plurality of frames obtained by photographing with the camera 21 at re-sampling points at equal intervals of 22.5 degrees are estimated. Therefore, θ = 22.5.

FIG. 10 is an explanatory diagram showing a method of estimating time stamps of a plurality of frames obtained by photographing at a re-sampling point. As shown in FIG. 10, the angles θ ₂ and θ ₁ from the reference points of the initial sampling points S ₂ and S ₁ (m = 2) in the vicinity of the re-sampling point T ₂ (x = 2) and the time stamp s ₂ and by linear interpolation based on the s _1, timestamp t ₂ of the frame obtained by the photographing by the camera 21 at the resampling point T ₂ it is estimated. The time stamp t ₂ can be estimated by substituting the time stamp corresponding to the initial sampling points S ₂ and S ₁ and the angle from the reference point shown in FIG. In the present embodiment, the time stamp t ₂ is estimated to be 140 from θ ₂ = 75, θ ₁ = 30, s ₂ = 180, and s ₁ = 120.

As shown in FIG. 8, the eight initial sampling frames obtained by photographing by the camera 21 at the eight initial sampling points are frames of time stamps at equal intervals such as every 60 (1 second). By doing so, as shown in FIG. 6, the initial sampling points can be widely dispersed on the circumference. Therefore, the accuracy of estimation of each time stamp t _x of a plurality of frames obtained by photographing by the camera 21 at the re-sampling point T _x is increased. On the other hand, for example, when the initial sampling point is biased to a part on the circumference, the accuracy of the estimation of the time stamp of the frame obtained by photographing by the camera 21 at the resampling point is biased to a part on the circumference. The lower the re-sampling point from the initial sampling point group, the lower the value.

  In the present embodiment, the plurality of initial sampling frames are time-stamp frames that are equally spaced in time. However, the present invention is not limited to this. For example, the plurality of initial sampling frames do not have to be time-stamped frames at equal intervals.

Next, as with the time stamp t ₂ estimated to be 140, the time stamp of the frame obtained by imaging by the camera 21 at another re-sampling point is estimated. Thereby, time stamps of a plurality of frames obtained by photographing by the camera 21 at a plurality of re-sampling points are estimated.

  Finally, the image extraction unit 11 uses the camera 21 at the re-sampling point to resample the time-stamp resampling frames estimated by the shooting position calculation unit 12 from the moving image stream stored in the shot image storage unit 40. Each of a plurality of frames obtained by photographing is extracted by resampling. Then, the image extraction unit 11 stores the extracted resampling frame in the multi-viewpoint image DB 50 as a still image.

[1-3. Effect]
As described above, the present technology is a still image extraction method in which a plurality of frames are extracted as still images from a moving image stream captured continuously in time while moving the camera relative to a subject. An operation of extracting a plurality of first frames included in the moving image stream, obtaining an imaging time of each of the extracted plurality of first frames, and a camera at the imaging time of each of the plurality of first frames Based on the operation of specifying the shooting position from the plurality of first frames, the shooting position of the camera specified by each of the plurality of first frames, and the acquired shooting time, a plurality of distances are equally spaced. The operation of estimating the shooting time of each of the plurality of frames obtained by shooting at the shooting position and the second frame at each of the estimated shooting times are shot at a plurality of shooting positions at equal intervals. The operation of extracting from the moving image stream as each of a plurality of frames obtained by this is performed.

  According to this, it is possible to extract a still image at an appropriate shooting position whose distance is equally spaced by shooting in a short time.

  That is, in the present embodiment, a plurality of initial sampling frames included in the moving image stream are extracted by initial sampling, and time stamps corresponding to the plurality of initial sampling frames are acquired. Then, the initial sampling points of each of the plurality of extracted initial sampling frames are identified, and the image is taken by the camera at the re-sampling points based on the identified initial sampling points and the time stamps corresponding to the plurality of initial sampling frames. The time stamp of each of the plurality of frames obtained by is estimated. Finally, a re-sampling frame corresponding to each estimated time stamp is extracted from the moving image stream as each of a plurality of frames obtained by photographing by the camera 21 at the re-sampling point.

  As a result, when the subject is photographed while moving the camera relative to the subject, the distance is evenly spaced from the image group obtained by photographing at the photographing position where the distance is not evenly spaced even if the distance is regular. It is possible to extract still images obtained by shooting at a plurality of shooting positions. Therefore, it is possible to extract a still image at an appropriate shooting position that is spaced at equal intervals by shooting in a short time.

  In addition, the shooting position of the second frame extracted from the moving image stream is specified based on the second frame, and the shooting position of the second frame and a plurality of shootings at the same distance are separated. It is determined whether or not the positional deviation between one of the positions and the equally spaced position is within a predetermined range. If the positional deviation is not within a predetermined range, using the extracted shooting position and the shooting time of the second frame, the shooting time of the frame obtained by shooting with the camera at the equally spaced position is again set. The second frame at the shooting time estimated and re-estimated is extracted from the moving image stream as frames obtained by shooting with the camera at a plurality of shooting positions that are spaced at equal intervals. .

  According to this, it is possible to extract with high accuracy still images at shooting positions that are spaced at equal intervals.

  In addition, when estimating the shooting time, by performing linear interpolation based on the specified shooting position of each of the plurality of first frames and the acquired shooting time, by shooting at a plurality of shooting positions at equal intervals. The imaging time of each of a plurality of obtained frames is estimated.

  According to this, it is possible to estimate the respective shooting times for a plurality of frames obtained by shooting with a camera at a plurality of shooting positions that are spaced at equal intervals.

  Further, when specifying the shooting positions of each of the plurality of first frames, the corresponding points between the frames are calculated by matching a plurality of feature points extracted from each of the plurality of first frames. Then, the photographing position is specified from each of the corresponding points between the calculated frames by using an SFM (Structure From Motion) algorithm.

  According to this, it is possible to specify the shooting position of the camera at the acquired shooting time of each of the plurality of frames included in the moving image stream.

  In addition, when extracting the second frames at the respective estimated shooting times, angles of equal intervals are obtained from a moving image stream obtained by shooting while moving the camera on the circumference around the subject. Each of the second frames is extracted as a plurality of frames obtained by photographing with a camera at a plurality of photographing positions.

  According to this, it is possible to extract all-around images from shooting positions arranged at equal intervals from a moving image stream obtained by shooting at a position on the circumference around the subject.

  Further, when extracting the estimated second frames at the respective shooting times, the parallax of the parallax is obtained from the moving image stream obtained by moving the camera in the tangential direction of the circumference around the subject. A plurality of second frames for generating a certain three-dimensional image are extracted.

  According to this, it is possible to generate a three-dimensional image with parallax from a moving image stream obtained by photographing from the depth direction.

  Further, when extracting the second frames at the respective estimated shooting times, a plurality of shooting positions that are equidistant from a moving image stream obtained by shooting while moving the camera on a straight line Each of the second frames is extracted as a plurality of frames obtained by shooting with the camera.

  In addition, the image processing apparatus of the present technology extracts a plurality of frames as still images from a moving image stream continuously captured in time while moving the camera relative to the subject. The image processing apparatus includes an image extraction unit and a shooting position calculation unit. The image extraction unit performs an operation of extracting a plurality of first frames included in the moving image stream and acquiring a shooting time of each of the extracted first frames. The shooting position calculation unit identifies information on the shooting position of the camera at the shooting time acquired in each of the plurality of first frames from the plurality of first frames, and the shooting time acquired in each of the first frames; Then, based on the information regarding the shooting position specified in each of the first frames, an operation of estimating the shooting time of each of a plurality of frames obtained by shooting at a plurality of shooting positions that are spaced at equal intervals is performed. Further, the image extracting unit captures a plurality of frames obtained by capturing the second frame corresponding to each capturing time estimated by the capturing position calculating unit at a plurality of capturing positions that are equally spaced from each other. Each of them extracts the moving image stream.

  According to this, it is possible to extract a still image at an appropriate shooting position whose distance is equally spaced by shooting in a short time.

  FIG. 11 is a flowchart illustrating an operation of a still image extraction method according to another embodiment of the present technology.

  The image processing apparatus 10 of the image processing system according to the present embodiment verifies the imaging position of the resampling frame extracted by resampling, and if the imaging position is out of the resampling point, performs resampling again.

  As shown in FIG. 11, the processes in steps S301 to S304 are performed in the same manner as the process shown in FIG. In step S304, the image extraction unit 11 extracts the re-sampling frame from the moving image stream by re-sampling as each of a plurality of frames obtained by photographing by the camera 21 at the re-sampling point. The re-sampling frame corresponding to each time stamp estimated in step S303 is not necessarily a plurality of frames obtained by photographing with the camera 21 at the re-sampling point. Specifically, if the relationship between the time stamp and the angle from the reference point is not proportional between the two initial sampling points in the vicinity of the re-sampling point, the estimation in step S303 is incorrect.

  When the rotation speed of the turntable 23 is not constant, the relationship between the time stamp between the initial sampling points and the angle from the reference point is not proportional. Thereby, the time stamp estimated by the linear interpolation based on the time stamp which is not proportional and the angle from the reference point does not become the time stamp corresponding to the re-sampling point.

  In the present embodiment, the plurality of re-sampling frames extracted by re-sampling in step S304 are not directly stored in the multi-viewpoint image DB 50 as still images, and the image extraction unit 11 extracts by re-sampling from the moving image stream in step S304. The shooting positions of the plurality of re-sampling frames are specified from the frame and the initial sampling frame (step S305). That is, the operation in step S302 is performed on a plurality of resampling frames extracted by resampling. Thereby, it can be determined whether or not the estimation in step S303 is correct.

  Next, the image extraction unit 11 is an equidistant position that is one of the photographing positions identified in step S305 of the resampled frame extracted by resampling and a plurality of photographing positions whose distances are equally spaced. That is, it is determined whether or not the positional deviation from the re-sampling point is within a predetermined range (step S306). For example, the image extraction unit 11 uses one of the shooting positions specified in step S305 of the resampled frame extracted by resampling and a plurality of shooting positions at equal intervals of 22.5 degrees. It is determined whether or not a positional deviation from a certain 45-degree re-sampling point is within a predetermined range.

  When the positional deviations of all the plurality of resampling frames extracted by resampling are within a predetermined range (Yes in step S306), the image extraction unit 11 selects the plurality of resampling frames extracted by resampling as a still image. And stored in the multi-viewpoint image DB 50.

  When the positional deviation of the plurality of resampling frames extracted by resampling is not within the predetermined range (No in step S306), the image extraction unit 11 uses the imaging position and time stamp of the resampling frame extracted by resampling. Then, again, the time stamp of the frame obtained by photographing with the camera 21 at the re-sampling point is estimated. In other words, the shooting positions and time stamps of the plurality of re-sampling frames extracted by resampling are added to the shooting positions of the plurality of initial sampling frames, and the operation in step S303 in which the estimation accuracy is increased is performed. Then, the image extracting unit 11 extracts the estimated time stamp frame from the moving image stream as a frame obtained by photographing by the camera 21 at the re-sampling point. That is, the image extraction unit 11 repeatedly performs resampling using the resampling frame extracted by the immediately preceding resampling until the positional deviation of the extracted resampling frames falls within a predetermined range. As a result, it is possible to reliably extract a frame in which the positional deviation falls within a predetermined range. In other words, the frame at the resampling point can be extracted with high accuracy.

  As described above, the shooting positions of a plurality of resampling frames extracted from the moving image stream by resampling are specified based on the resampling frames. Then, it is determined whether or not the positional deviation between the imaging position of the re-sampling frame extracted by re-sampling and the re-sampling point is within a predetermined range. When the positional deviation of a plurality of re-sampling frames extracted by resampling is not within a predetermined range, shooting by the camera 21 at the re-sampling point is performed using the shooting position and time stamp of the re-sampling frame extracted by re-sampling. Estimate the time stamp of the frame obtained by Then, re-sampling is repeatedly performed using the re-sampling frame extracted by the previous re-sampling until the positional deviation of the plurality of re-sampling frames extracted by the estimated time stamp falls within a predetermined range.

  As a result, re-sampling is performed until the positional deviation of the plurality of re-sampling frames extracted by re-sampling falls within a predetermined range, so that it is possible to extract a still image at an appropriate shooting position that is spaced at regular intervals. it can.

  FIG. 12 is an explanatory diagram illustrating a method for photographing a subject according to another embodiment of the present technology. As illustrated in FIG. 12, the photographing unit 20 includes a slider 25 separately from the robot arm 22, and the camera 21 is attached to the slider 25.

  As surrounded by a one-dot chain line in FIG. 12 (part indicated by an arrow A), the robot arm 22 includes a slider 25 and a camera attached to the slider 25 in the circumferential direction on the XZ plane of FIG. 21 is held movably. As indicated by an arrow A in FIG. 12, the slider 25 holds the camera 21 attached to the slider 25 so as to be movable in the y-axis direction in FIG. A portion surrounded by a one-dot chain line in FIG. 12 is a diagram illustrating a state in which the camera 21 is moved in the y-axis direction by the slider 25 when the camera 21 is viewed from the front.

  In the embodiment described above, the camera 21 generates a moving image stream by shooting the subject 24 while moving relative to the subject 24 on the circumference around the subject 24. On the other hand, in the present embodiment, the camera 21 further moves in the tangential direction (y-axis direction) of the circumference around the subject 24, and after the movement, the subject 24 moves on the circumference around the subject 24 again. The subject 24 is photographed while moving relative to the image. Specifically, the control unit 30 performs the following control.

  The control unit 30 repeatedly controls the rotation of the turntable 23 and the movement of the camera 21 by the robot arm 22 in the same manner as in the above embodiment, so that the camera 21 is positioned from the position on the spherical surface with the subject 24 as the center. The subject 24 is photographed. Then, the control unit 30 controls the movement of the camera 21 by the slider 25 and moves the camera 21 in the y-axis direction of FIG. After moving the camera 21 in the y-axis direction, the control unit 30 causes the camera 21 to image the subject 24 again from a position on the spherical surface centered on the subject 24. Thereby, since the subject 24 is also photographed in the depth direction, a moving image stream for generating a parallax three-dimensional image obtained by photographing the subject 24 from a position on the spherical surface centering on the subject 24 is obtained.

  As described above, in this modified example, the camera 21 is moved in the tangential direction of the circumference with the subject 24 as the center for shooting.

  Thereby, a moving image stream including a frame for generating a three-dimensional image with parallax is obtained. Therefore, it is possible to extract a plurality of frames for generating a three-dimensional image with parallax.

  As described above, the embodiments have been described as examples of the technology in the present technology. For this purpose, the accompanying drawings and detailed description are provided.

  Accordingly, the constituent elements described in the accompanying drawings and the detailed description may include not only constituent elements essential for solving the problem but also constituent elements not essential for solving the problem. Therefore, it should not be immediately recognized that these non-essential components are essential as those non-essential components are described in the accompanying drawings and detailed description.

  Moreover, since the above-mentioned embodiment is for demonstrating the technique in this technique, a various change, replacement, addition, omission, etc. can be performed in a claim or its equivalent range.

  The present technology can be applied to, for example, an image processing apparatus that extracts a still image for interactively displaying an entire surrounding image of an object.

DESCRIPTION OF SYMBOLS 1 Image processing system 10 Image processing apparatus 11 Image extraction part 12 Shooting position calculation part 20 Shooting part 21 Camera 22 Robot arm 23 Turntable 24 Subject 25 Slider 30 Control part 40 Photographed image memory | storage part 50 Multiview image DB

Claims (8)

A plurality of first frames are set such that shooting times are equidistant from a moving image stream which is taken continuously in time by moving the camera relatively in one direction on the circumference or straight line centering on the subject. The action of extracting
An act of identifying a put that shooting positions on the plurality of first frames each shooting time from the plurality of first frame,
Operation and to estimate the respective imaging times plurality of frames obtained by the basis of the plurality of first frames each photographing time and photographing location, distance is taken at a plurality of photographing positions equally spaced,
A still image extraction method comprising: performing an operation of extracting each of a plurality of second frames corresponding to each of a plurality of estimated shooting times as a still image from the moving image stream. The still image extraction method according to claim 1 , wherein the number of the plurality of first frames is smaller than the number of frames included in the moving image stream . 2. The moving image stream according to claim 1, wherein the moving image stream is a moving image stream that is taken continuously in time by moving the camera at a relatively constant speed around the subject. The still image extraction method described. An image processing apparatus and a photographing position calculation unit images extractor,
The image extraction unit moves the camera relatively in one direction on a circumference or a straight line with the subject as the center so that shooting times are equidistant from a moving image stream that is taken continuously in time. An operation of extracting a plurality of first frames is performed,
The photographing position calculating unit, the imaging position location was identified from the plurality of first frame, the shooting time of each of the plurality of first frame and the imaging position in the plurality of first frames each shooting time based distance performs an operation for estimating the respective imaging times plurality of frames obtained by photographing a plurality of photographing positions equally spaced,
Before Symbol image extracting unit includes a feature to perform an operation of extracting a plurality of second frame corresponding to each of the photographing positions plurality of imaging times estimated by the calculation unit, from the moving image stream as a still image An image processing apparatus. An operation of moving a camera relative to a subject and extracting a plurality of first frames from a moving image stream captured continuously in time;
An operation of acquiring a photographing time of each of the plurality of first frames;
An operation of specifying the shooting position of each of the plurality of first frames from the plurality of first frames;
An operation of estimating the shooting time of each of a plurality of frames obtained by shooting at a plurality of shooting positions at equal intervals based on the shooting time and shooting position of each of the plurality of first frames;
An operation of extracting a plurality of second frames corresponding to each of a plurality of estimated shooting times;
An act of identifying on the basis of the shooting position of each of the plurality of second frame to the plurality of second frames,
And imaging position of each of the plurality of second frames, the operation of positional deviation of the distance is equidistant photographing position to determine whether it is within a predetermined range,
When the positional deviation is not within the predetermined of the previous SL by using a plurality of second frames each photographing position and photographing time, distance and the operation of estimating the shooting time having equal intervals photographing position again ,
A still image extraction method, comprising: extracting a plurality of second frames corresponding to a plurality of re-estimated shooting times as still images from the moving image stream. An image processing apparatus including an image extraction unit and a shooting position calculation unit,
The image extraction unit moves a camera relative to a subject, extracts a plurality of first frames from a moving image stream captured continuously in time, and extracts each of the plurality of first frames. Take action to get the shooting time,
The shooting position calculation unit specifies a shooting position at each shooting time of the plurality of first frames from the plurality of first frames, and based on the shooting time and the shooting position of each of the plurality of first frames, Performing an operation to estimate the shooting time of each of a plurality of frames obtained by shooting at a plurality of shooting positions at equal distances,
The image extraction unit performs an operation of extracting a plurality of second frames corresponding to a plurality of shooting times estimated by the shooting position calculation unit,
The shooting position calculation unit specifies a shooting position at a shooting time of each of the plurality of second frames from the plurality of second frames, and the shooting positions of the plurality of second frames are equally spaced from each other. It is determined whether or not the positional deviation with respect to the photographing position is within a predetermined range. If the positional deviation is not within the predetermined range, the photographing position and photographing time of each of the plurality of second frames are determined. Use it to re-estimate the shooting time at which the shooting positions are equally spaced,
The image processing apparatus, wherein the image extraction unit performs an operation of extracting each of a plurality of second frames corresponding to a plurality of re-estimated shooting times from the moving image stream as a still image. An operation of moving a camera relative to a subject and extracting a plurality of first frames from a moving image stream captured continuously in time;
An operation of acquiring a photographing time of each of the plurality of first frames;
An operation of specifying the shooting position of each of the plurality of first frames from the plurality of first frames;
By linear interpolation based on the photographing time and photographing location of each of the plurality of first frame, operation for estimating a plurality of frames each imaging time obtained by the distance to shoot at a plurality of photographing positions equally spaced,
A still image extraction method comprising: performing an operation of extracting each of a plurality of second frames corresponding to each of a plurality of estimated shooting times as a still image from the moving image stream. An image processing apparatus including an image extraction unit and a shooting position calculation unit,
The image extraction unit moves a camera relative to a subject, extracts a plurality of first frames from a moving image stream captured continuously in time, and extracts each of the plurality of first frames. Take action to get the shooting time,
The shooting position calculation unit specifies shooting positions at the shooting times of the plurality of first frames from the plurality of first frames, and is linear based on the shooting times and shooting positions of the plurality of first frames. By complementing, perform the operation to estimate the shooting time of each of multiple frames obtained by shooting at multiple shooting positions with equal distances,
The image extraction unit performs an operation of extracting each of a plurality of second frames corresponding to each of a plurality of shooting times estimated by the shooting position calculation unit as a still image from the moving image stream. Image processing device.

Images