JP2017103564A - Control apparatus, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly control a frame rate of a motion image to be acquired for forming a three-dimensional shape in accordance with a subject.SOLUTION: A control apparatus which acquires a frame from an imaging device imaging a motion image for forming a distance image of a subject, and controls an output of the motion image, includes: acquisition means of acquiring the distance image corresponding to a frame constructing the motion image imaged by the imaging device; calculation means of calculating a complication level of a shape of the subject in the distance image on the basis of the distance image; determination means of determining a frame rate of the motion image outputted corresponding to the complication level; and forming means of acquiring the frame from the motion image imaged by the imaging device in accordance with the frame rate acquired by the determination means, and forming the motion image of the frame rate.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for controlling photographing for generating a three-dimensional shape of a subject using a plurality of images photographed by a camera.

  2. Description of the Related Art Conventionally, a technique (Structure from motion) for acquiring a plurality of images with different viewpoints with respect to a subject and restoring the three-dimensional shape of the subject from the captured image is known. In order to generate a three-dimensional shape of a subject as a moving image, it is necessary to continuously shoot a plurality of images with different viewpoints with respect to the subject. Therefore, there is a technique for generating a three-dimensional shape of a subject based on a result of shooting a moving image of the subject. However, the frame rate of a moving image shot to generate a three-dimensional shape must consider a trade-off with the generation accuracy of the three-dimensional shape. The higher the frame rate of a moving image (the more images can be acquired), the higher the accuracy of generating a three-dimensional shape. On the other hand, the data amount and processing load increase. If the frame rate is reduced (the number of images to be acquired is reduced) giving priority to the amount of data and processing load, information necessary for generating a three-dimensional shape may be lost.

  Patent Document 1 discloses a method for dynamically controlling the frame rate of a moving image in a method for acquiring three-dimensional information of a subject viewed from a car from a moving image taken from an in-vehicle camera. Specifically, if it is determined that there is a risk of collision based on the speed of the vehicle or the amount of visual field change, the movie is shot at a high frame rate, otherwise the movie is shot at a low frame rate. To do. As a result, the amount of data to be saved is reduced while securing information necessary for grasping the situation.

JP 2010-273178 A

  However, in the method disclosed in Patent Document 1, the frame rate may not be appropriately controlled. Shoot a video at a high frame rate for a subject that can generate a 3D shape even with a low frame rate video, or a video at a low frame rate for a subject that cannot generate a 3D shape unless it is shot at a high frame rate. Sometimes I shoot.

  Therefore, an object of the present invention is to appropriately control the frame rate of a moving image acquired to generate a three-dimensional shape according to the subject.

In order to solve the above-described problems, the present invention provides a control device that acquires a frame from an imaging device that captures a moving image for generating a distance image of a subject and controls the output of the moving image, the moving image captured by the imaging device Acquisition means for acquiring a distance image corresponding to a frame constituting the image, and calculation means for calculating the complexity of the shape of the subject in the distance image based on the distance image;
A determining unit that determines a frame rate of a moving image to be output according to the complexity, and a frame is acquired from the moving image captured by the imaging device according to the frame rate acquired by the determining unit, and a moving image having the frame rate is generated. And generating means.

  As an effect of the present invention, the frame rate of a moving image acquired to generate a three-dimensional shape can be appropriately controlled according to the subject.

1 is a block diagram of an imaging control apparatus that performs three-dimensional shape generation according to a first embodiment. Flowchart of subject photographing processing according to the first embodiment Flowchart of shape generation according to the first embodiment Flow chart of complexity calculation processing and frame rate calculation processing of the first embodiment A camera that shoots a plane while translating A diagram showing the shooting range of the camera Camera and occlusion taking pictures of uneven surfaces Details of camera and occlusion shooting on uneven surface Structure of GOP in encoding Block diagram of computer The block diagram of the imaging | photography control apparatus which performs the three-dimensional shape production | generation of 2nd Embodiment Flowchart of an imaging control apparatus that performs 3D shape generation according to the second embodiment The figure which shows an example of the filter for detecting a recessed area

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. In addition, each structure shown in the following embodiment is only an example, and this invention is not limited to the structure shown in figure.

[First Embodiment]
In the first embodiment, a moving image is shot while the camera is translated at a constant speed with respect to the subject, and at least a part of the shot moving image is recorded in the storage. A distance image in the captured scene is generated based on the captured moving image, and a three-dimensional shape of a subject included in the scene is generated based on the distance image. In particular, in the present embodiment, the frame rate of a moving image that is dynamically acquired is controlled based on a distance image of a scene that is simply calculated during moving image shooting. In the present embodiment, the moving image means an image sequence using a plurality of images taken at successive times as frames. As specific use cases, various applications such as shooting a wall surface of a building with a camera mounted on a gondola or shooting a road with a camera mounted on a car are conceivable.

  FIG. 1 is a block diagram of an imaging apparatus 109 that can be applied to the first embodiment and an image processing apparatus 112 that generates a three-dimensional shape using a color image obtained from the imaging apparatus 109. In the present embodiment, the imaging device 109 is a camera capable of multi-viewpoint shooting. Specifically, a plenoptic camera is used. A plenoptic camera is a camera that separates incident light by installing a microlens array in front of an imaging sensor and can simultaneously capture multi-viewpoint images. In addition, as a camera capable of multi-view shooting, a multi-view camera in which a plurality of imaging units are arranged may be used. The imaging unit 111 is an optical system such as an imaging sensor or a microlens array. Further, the imaging device 109 has a built-in control device 110 that controls various processes and a frame rate for an image obtained by the imaging unit 111. On the other hand, the image processing device 112 is a device different from the imaging device 109 and performs image processing based on data acquired from the imaging device 109 to generate a three-dimensional shape. The image processing apparatus 112 is realized by, for example, a personal computer.

  First, the imaging device 109 will be described. In response to an instruction to start moving image shooting, the imaging unit 111 shoots a subject at a predetermined frame rate. Here, a moving image composed of color digital images composed of red (R), blue (B), and green (G) is taken. In addition, since images can be acquired from multiple viewpoints, a plurality of moving images from different viewpoints are acquired substantially. The multi-viewpoint image acquisition unit 101 sequentially acquires images of each frame constituting the moving image from the imaging unit 111. In addition, the multi-viewpoint image acquisition unit 101 acquires some frames from the moving image acquired by the imaging unit 111 according to the frame rate input from the frame rate determination unit 103, and the distance image generation unit 102 and the single-viewpoint image generation unit To 104.

  The distance image generation unit 102 generates a distance image from the acquired multi-viewpoint image. A distance image is an image that stores distance information representing the distance from the camera position to the subject in pixel units. The distance information may be a distance itself expressed in a metric unit system, or may be a value that can be converted into a distance such as parallax. The distance image generation unit 102 generates one distance image from a plurality of different viewpoint images (frames) taken at the same time among the acquired moving images. Here, for example, parallax estimation is performed by stereo matching, and a parallax image (disparity map) is generated as a distance image. Note that the distance image calculated here is the first distance image. The distance image generation unit 102 outputs the distance image to the frame rate determination unit 103.

  The frame rate determination unit 103 calculates the complexity of the subject from the distance image. The complexity of the subject is an index representing the complexity of the shape of the subject in the subject area that can be photographed from the camera. The frame rate determination unit 103 determines the frame rate of the moving image that the multi-viewpoint image acquisition unit 101 should acquire according to the complexity of the subject. The complexity and frame rate calculation processing will be described later.

  The single-viewpoint image generation unit 104 sequentially generates single-viewpoint images from a plurality of images with different viewpoints that are corresponding frames in the acquired moving image, and outputs the generated images to the encoded image storage 106. Here, the single-viewpoint image generation unit 104 generates a single-viewpoint image by calculating a value obtained by averaging pixel values for each corresponding pixel of a plurality of images having different viewpoints corresponding to the same time. Similarly to the multi-viewpoint image, the single-viewpoint image generated by the single-viewpoint image generation unit 104 is a color image composed of RGB. Since the single-viewpoint image generation unit 104 generates and outputs a single-viewpoint image in order from a plurality of frames constituting the moving image, the image output from the single-viewpoint image generation unit 104 forms a single-viewpoint moving image. It can also be said to be an image (frame) sequence. Furthermore, the moving image output from the single-viewpoint image generation unit 104 is a moving image whose frame rate is dynamically changed according to the shape of the subject. The single viewpoint image generated here is used for the subsequent three-dimensional shape generation processing. Note that the single-viewpoint image generation method may be a method of selecting a single-viewpoint image from a plurality of multi-viewpoint images, or a single sum of pixel values of each image instead of an average of pixel values of each pixel in the multi-viewpoint image. You may take the method of calculating as a pixel value in a viewpoint image.

  The image encoding unit 105 sequentially encodes the single viewpoint images output from the single viewpoint image generation unit 104 by a predetermined encoding method, and sends the encoded images to the encoded image storage 106. Here, as described above, the multi-viewpoint image generation unit 104 sequentially outputs single-viewpoint images as frames. H.264 is used for encoding. The encoding method may be one that can efficiently encode the single-view color image. For example, each single-view image may be independently encoded using JPEG.

  Next, the image processing apparatus 112 will be described. The image decoding unit 107 acquires the encoded data from the encoded image storage 106, decodes it, and outputs a single-view video. The three-dimensional shape generation unit 108 generates a distance image from a plurality of input single viewpoint images. As described above, since the imaging device 109 is translated at a constant speed, a plurality of single-viewpoint images can be regarded as images shot from different viewpoints although the shooting times are different. Therefore, the three-dimensional shape generation unit 108 generates a distance image using a multi-viewpoint image that is captured when the camera moves. The distance image calculated here is a second distance image. Furthermore, a three-dimensional shape of the subject is generated based on the second distance image.

  Next, the flow of processing in the imaging device 109 will be described. FIG. 2 shows a processing flow for realizing the imaging device 109. In step S <b> 201, the multi-viewpoint image acquisition unit 101 acquires a frame image of interest from the moving image of the subject captured by the imaging unit 111 from the imaging unit 111. The images acquired here are multi-viewpoint images taken from a plurality of different viewpoints at the same time. In step S202, the distance image generation 102 generates a distance image based on a plurality of images taken at the same time and having different viewpoints.

  In step S203, the frame rate determination unit 103 calculates the complexity of the subject from the distance image. In step S204, the frame rate determination unit 104 calculates the number M of frames of the multi-viewpoint image to be acquired until the next generation of the distance image according to the complexity of the subject. This means that the frame rate of the multi-viewpoint image is M times the frame rate of the distance image. Detailed descriptions of step S203 and step S204 will be described later. In step S205, the single viewpoint image generation unit 104 generates one single viewpoint image from a plurality of images having different viewpoints.

  In step S <b> 206, the image encoding unit 105 encodes the single viewpoint image and outputs it to the encoded image storage 106. In the present embodiment, in consideration of the fact that the frame rate of the moving image output from the imaging device 109 changes dynamically, encoding is performed as follows. As described above, the distance image is not calculated for all the frame images, but is generated at a temporally constant frame period. The distance image is generated from the acquired multi-view image of color, while the single-view image is generated from the multi-view images of all frames. For this reason, there is always a color single-viewpoint image corresponding to the same time as the distance image, and the number of frames varies between the times corresponding to the distance image. Therefore, a GOP (Group Of Picture), which is a moving image coding unit, is divided at the timing when the distance image is generated, and a single viewpoint image corresponding to the same time as the distance image is defined as an I frame (first frame) of the GOP. . At this time, the number of frames constituting the GOP is M including I frames. The state of GOP is shown in FIG. In FIG. 9, I, P, and B of the single viewpoint image are H. H.264 encoded I frame, P frame, and B frame. Normally, the frame rate of a moving image is described in the sequence header of the sequence layer, but here, since the frame rate is different for each GOP, the number M of constituent frames is described in the GOP header of the GOP layer. Further, when the frame rate is lowered, the temporal correlation between frames is lowered. For this reason, when the number of frames M falls below a predetermined threshold, all the frames in the same GOP are intra-coded.

  Steps S207, S208, and S209 are the same processes as steps S201, S205, and S206, respectively. The processing from step S207 to step S209 is repeated M-1 times in accordance with the frame rate at which the multi-viewpoint image is acquired. In step S210, the imaging apparatus 109 confirms a shooting end signal input from the outside. If shooting has ended, the processing flow ends. If not, the process returns to step S201.

  Furthermore, the flow of processing in step S203 and step S204 will be described in detail. FIG. 3 is a processing flow for realizing the frame rate determination unit 103. Step S203 corresponds to steps S302 to S304, and step S204 corresponds to steps S305 and S306.

  First, in step S301, the complexity n is initialized to zero. In step S302, the number of pixels that cannot be measured in the distance image is added to n. A pixel that cannot be measured means a pixel for which distance information could not be calculated. When a distance image is generated, an area such as an occlusion area that is captured only in one viewpoint image cannot detect a corresponding point between the multi-viewpoint images and cannot measure the distance. A subject with many occlusion areas is likely to have a complicated subject shape, and the number of non-measurable pixels correlates with the complexity of the subject shape. Therefore, the number of pixels that cannot be measured is detected and added to the complexity n.

  In step S303, edge pixels in the distance image are detected. The detection of the edge pixel can be realized by using a Sobel filter or the like for the distance image. The edge of the distance image represents a boundary where the distance is discontinuous. If there are many edge pixels in the distance image, there is a high possibility that there are various subjects with different positions, or there are many steep surface irregularities in the subject itself. In such a case, an occlusion area is likely to occur, and the subject can be regarded as having a complicated shape. In step S304, the calculated number of edge pixels is counted and added to the complexity n.

  In step S305, the frame rate F is calculated based on the calculated complexity n. First, a minimum frame rate Fmin necessary for generating a three-dimensional shape will be described with reference to FIG. FIG. 5 shows a situation in which a plane is taken as a subject and the camera is shooting while moving at a constant speed with a constant distance to the plane. In FIG. 5, the camera position indicates the captured viewpoint, and the broken line extending from the camera indicates the angle of view of the camera at each viewpoint position. When a three-dimensional shape is generated using motion parallax, it is necessary to capture at least two frames at all positions on the surface that is the subject. Therefore, FIG. 5A shows a state where the entire plane is always photographed within the angle of view of 2 frames. In principle, it is possible to generate a three-dimensional shape at the imaging interval shown in FIG. 5A. However, due to the optical properties of the camera, the spatial resolution decreases near the image edge and the angle of view. In some cases, the corresponding point cannot be detected at the boundary portion of. Therefore, as shown in FIG. 5B, the frame rate for imaging so that the entire plane always falls within the angle of view of 3 frames is set to be the minimum frame rate necessary for stable three-dimensional shape generation.

Next, the minimum required frame rate Fmin is calculated using specific parameters. Here, the distance between the camera and the plane is L [m], the moving speed of the camera is V [m / s], and the angle of view of the camera is θ [°]. Since it can be seen from FIG. 6 that one imaging is necessary for (2/3) Ltan (θ / 2) [m], the imaging cycle is (2 / 3V) Ltan (θ / 2) [s]. The frame rate Fmin [fps] can be calculated according to Equation 1.
Fmin = 3V / (2Ltan (θ / 2)) Equation 1
Here, how to obtain the parameters L, V, and θ of Equation 1 will be described. The distance L can know an approximate value between camera subjects from the distance image. The motion speed V of the camera is estimated by combining the distance L with a motion vector used in moving image encoding. Further, the angle of view θ of the camera can be obtained from the focal length of the photographing apparatus and the size of the photographing element. As described above, the frame rate Fmin is calculated based on the imaging conditions of the imaging device 109 and the acquired information. If there is an environment in which shooting is performed with the parameters V, L, and θ kept constant in advance, shooting may be performed by inputting parameters in advance.

  Furthermore, when the shape of the subject is complicated, information necessary for generating the shape of the subject is acquired by generating an image sequence having a high frame rate. For example, if there is an uneven surface as shown in FIG. 7A on the plane that is the subject, 1 is displayed on the plane B, C, E, and F when shooting under the conditions shown in FIG. There is an area that can be imaged only by the viewpoint camera. Therefore, the second distance image cannot be obtained in the areas of the surface B, the surface C, the surface E, and the surface F. However, as shown in FIG. 7B, it is wasteful to output a moving image of an image sequence having a uniform high frame rate to a subject.

  In step S305, the frame rate F is dynamically adjusted according to the complexity of the subject calculated using the first distance image obtained by the plenoptic camera. The minimum frame rate is the minimum required frame rate Fmin shown in Equation 1. The maximum value is the maximum frame rate Fmax that can be captured by the camera used for capturing. The complexity n calculated from step S302 to step S304 indicates the number of pixels representing a portion where the structure of the subject is complex, such as pixels that cannot be measured and edge pixels. Therefore, the maximum value of complexity is the number of pixels N of the entire image. In this embodiment, the frame rate F− is set as shown in Equation 2.

However, P (P ≧ 1) is a parameter. When F obtained here exceeds Fmax, F = Fmax. In step S306, the number M of frames is calculated based on the frame rate of the distance image and the frame rate F of the moving image being shot. First, if the frame rate of the distance image is Fd, the number of frames M can be calculated according to Equation 3.

In this embodiment, Fd = Fmin. Therefore, Equation 3 is transformed as Equation 4.

Above, detailed description of step S203 and step S204 is complete | finished.

  Next, the three-dimensional shape generation process executed by the image processing apparatus 112 will be described with reference to FIG. In step S401, the image decoding unit 107 decodes the encoded data. As a result of decoding, an image (frame) constituting a moving image is obtained, and a moving image captured while a single viewpoint translates at a constant speed can be acquired.

  In step S402, the three-dimensional shape generation unit 108 extracts a local feature amount from the image acquired in step S401, and searches for a corresponding feature amount on the image. Further, the external parameters of the camera in each frame are estimated based on the position of the corresponding feature amount. However, the internal parameters are obtained in advance and are common to all frames. The internal parameters generally represent the focal length of the camera, the position of the image origin, lens distortion characteristics, and the like. The external parameter is the camera posture in the global coordinate system, and is generally expressed by the global coordinate T and the optical axis direction R of the camera. Here, SIFT (Scale-Invariant Feature Transform) is used as the local feature amount. The feature amount may be any method that can search for local correspondence in an image, and SURF (Speeded Up Robust Features) or other methods may be used. Here, in addition to the calculation of local features and search for corresponding points, it is desirable to remove erroneous corresponding points using Ransac (random sample consensus) or the like.

  In step S403, based on the estimated internal / external parameters, the three-dimensional shape generation unit 108 estimates a parallax from a plurality of viewpoint image pairs with which local feature amounts are associated, and generates a second distance image. The distance value of the second distance image obtained here is projected onto a global coordinate system to generate a point group representing the surface of the subject. In step S404, a three-dimensional shape model of the subject is generated from the obtained point group. As a method for generating the three-dimensional shape model, an explicit modeling method such as Delaunay triangulation may be used, or an implicit modeling method such as Poisson method may be used. This completes the process of generating the three-dimensional shape of the subject.

  As described above, according to the first embodiment, the frame rate of the moving image for generating the distance image is controlled based on the shape of the subject. When it is predicted that the shape of the subject is complicated, a moving image is created at a high frame rate, and in other cases, a moving image is created at a low frame rate. Thereby, the data amount to preserve | save can be suppressed appropriately, acquiring the information required for calculation of a distance image.

  In the first embodiment, the complexity of the subject is calculated using the first distance image generated based on images at a plurality of viewpoints photographed at the same time. The baseline length of the plenoptic camera is relatively short depending on the width of the camera. Therefore, the accuracy of the calculated distance information is low. On the other hand, since there is no need to estimate camera motion, distance information can be obtained relatively easily. Therefore, the second distance image calculated based on higher-precision motion parallax was used to generate the three-dimensional shape while controlling the frame rate using the first distance image generated by a simple method. Accordingly, the image processing apparatus 110 only needs to acquire a moving image that is not a multi-viewpoint image but a single-viewpoint image, and the amount of output data from the imaging apparatus 109 and the amount of input data acquired by the image processing apparatus 110 can be suppressed.

  In addition, the imaging device 109 that can output a distance image includes a device that separately outputs a reliability indicating the accuracy of the distance measurement value of each pixel in the distance image. In this case, a pixel with low reliability output may be treated as a pixel that cannot be measured. Further, the complexity may be calculated based on only one of the number of non-measurable pixels and the number of edge pixels.

  Furthermore, the complexity of the subject may be calculated using an index other than the number of non-measurable pixels and the number of edge pixels. As an index other than the number of pixels that cannot be measured and the number of edge pixels, three examples will be described below. First, the complexity may be calculated based on the number of pixels in the edge region having a specific direction, instead of all the edge pixels. FIG. 8 is a diagram showing the positional relationship between the subject and the camera. 8A, 8B, and 8C, the areas shown in the respective angles of view of the viewpoints 706, 707, and 708 in FIG. 7 are represented by bold lines. It is only the viewpoint 707 shown in FIG. 8B that the joint between the surface E and the surface F is reflected in the angle of view. Therefore, the joint between the surface E and the surface F cannot obtain the second distance information. Therefore, it is desirable to photograph at a high frame rate in the vicinity of the viewpoint 707 where the concave portion including the surfaces E and F can be photographed.

  A feature of the joint between the surface E and the surface F is that the edge direction is perpendicular to the movement direction of the camera. In the case of such an edge, since there are few viewpoint positions that can be photographed so as to include this edge, it is preferable that the complexity of the joint between the surface E and the surface F be a relatively large value. Therefore, the number of pixels constituting an edge in a specific direction is detected as the complexity based on the camera movement direction. Specifically, the subsequent motion direction of the camera is estimated from the motion vector obtained as a result of encoding the single viewpoint image immediately before the distance image is acquired. The encoding method is H.264. A specific process when H.264 is used will be described. H. In H.264, a motion vector is set for each macroblock and sub-block, and a motion vector may not be given depending on the block. In order to obtain the motion of the entire screen, a weighted average corresponding to the block size of each motion vector may be taken. That is, if the motion vector of the block in which the motion vector is set and the block size thereof are Ci and Si, respectively, the average motion vector MC of the entire screen can be calculated according to Equation 5.

Using the direction of the average motion vector obtained by Equation 5, only edge pixels close to the opposite of the average motion vector are detected, and the number of detected pixels is added to the complexity. Thereby, the complexity can be derived more appropriately for a shape in which occlusion is likely to occur according to the moving direction of the camera.

  Note that only the number of edge pixels in a specific direction may be added to the complexity without calculating the number of all edge pixels described in the above embodiment. When all edge pixels are detected, it is necessary to filter at least two directions of vertical and horizontal in order to detect edges in all directions. On the other hand, when the edge direction to be detected is limited, a single filtering according to the direction is sufficient, so that the amount of calculation required for the detection process can be reduced.

  Next, an example in which a concave area is detected and complexity is calculated will be described. The occlusion area is caused by unevenness of the subject's confidence, and occlusion is likely to occur particularly in the recess. Therefore, the distance image is subjected to filter processing for detecting a concave portion, and the number of pixels of the detected concave portion is defined as the complexity. For detection of the concave region, a filter is used in which the coefficient decreases as the distance from the filter center increases. An example of a filter used for detection of the concave region is shown in FIG. FIG. 13A shows a case of 3 pixels × 3 pixels, and FIG. 13B shows a case of 5 pixels × 5 pixels. The size of the concave region that can be detected depends on the filter size. For example, a large concave area of 100 × 100 pixels on an image cannot be detected as a concave area even if a 3 × 3 filter is used. Conversely, even if a large filter is used for a concave region with a small number of pixels, it cannot be detected. Using this property, the size of the dent to be detected can be determined. In addition, it can be determined by determining a threshold value with respect to the absolute value of the value after the filter processing, and only a deep recess is detected as the threshold value is increased.

  Further, the variance in the distance image may be calculated, and the complexity may be calculated based on the variance. When the dispersion in the distance image is large, it is predicted that the distance between the subject and the camera is not uniform and the shape of the subject is complicated. Therefore, the complexity may be calculated based on the variance by associating the variance with the complexity so that the greater the variance in the distance image, the greater the complexity. In particular, when the complexity is calculated based only on the variance, the filter processing is not required for the calculation of the complexity, so that it can be realized with a simpler configuration. Note that when the complexity is calculated in combination with the number of edge pixels or the number of non-measurable pixels, it is preferable to normalize the calculated variance.

[Modification of First Embodiment]
Each unit of the control device 111 illustrated in FIG. 1 may be configured by hardware incorporated in the imaging device 109, but may be implemented as software (computer program) as an external device of the imaging device 109. In this case, the software is installed in a memory of a general computer such as a PC (personal computer). Then, when the CPU of this computer executes the installed software, this computer realizes the function as the control device of the photographing unit 111 described above. An example of the hardware configuration of a computer applicable to the control device of the photographing apparatus according to the first embodiment will be described with reference to the block diagram of FIG.

  The CPU 1001 controls the entire computer using computer programs and data stored in the RAM 1002 and the ROM 1003, and executes the above-described processes described as being performed by the imaging control apparatus.

  The RAM 1002 is an example of a computer-readable storage medium. The RAM 1002 has an area for temporarily storing computer programs and data loaded from the external storage device 1007, the storage medium drive 1008, and the network interface 1010. Furthermore, the RAM 1002 has a work area used when the CPU 1001 executes various processes. That is, the RAM 1002 can provide various areas as appropriate. The ROM 1003 is an example of a computer-readable storage medium, and stores computer setting data, a boot program, and the like.

  The keyboard 1004 and the mouse 1005 can be operated by a computer operator to input various instructions to the CPU 1001. The display device 1006 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 1001 using an image, text, or the like.

  The external storage device 1007 is an example of a computer-readable storage medium, and is a large-capacity information storage device represented by a hard disk drive device. The external storage device 1007 stores an OS (Operating System), computer programs and data for causing the CPU 1001 to perform the processes shown in FIG. 1, the above-described various tables, databases, and the like. Computer programs and data stored in the external storage device 1007 are appropriately loaded into the RAM 1002 under the control of the CPU 1001 and are processed by the CPU 1001.

  The storage medium drive 1008 reads a computer program and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and outputs the read computer program or data to the external storage device 1007 or the RAM 1002. Note that part or all of the information described as being stored in the external storage device 1007 may be recorded on this storage medium and read by this storage medium drive 1008.

  An I / F 1009 is an interface for inputting a color image, a distance image, and the like from the outside, and outputting a frame rate. For example, the I / F 1009 is a USB (Universal Serial Bus). An imaging device 109 is connected as an external device, and a color image and a distance image are output through the I / F 1007. A bus 1010 connects the above-described units.

  In the above configuration, when the computer is turned on, the CPU 1001 loads the OS from the external storage device 1007 to the RAM 1002 according to the boot program stored in the ROM 1003. As a result, an information input operation can be performed via the keyboard 1004 and the mouse 1005, and a GUI can be displayed on the display device 1006. When the user operates the keyboard 1004 or the mouse 1005 and inputs an activation instruction for the encoded application stored in the external storage device 1007, the CPU 1001 loads the program into the RAM 1002 and executes it. As a result, the computer functions as an imaging control device.

  Note that the encoding application program executed by the CPU 1001 is a program for realizing the flowchart shown in FIG. The output encoded data is stored in the external storage device 1007. Note that this computer can be similarly applied to imaging control apparatuses according to the following embodiments.

[Second Embodiment]
In the first embodiment, the case where a multi-viewpoint image is acquired using a plenoptic camera has been described as an example. In the second embodiment, an example will be described in which a single-viewpoint camera is captured in hand. As a use case, there are cases where it is necessary to shoot a subject having a very complicated shape with various camera postures, or generating an internal three-dimensional shape while traveling around a building. In the second embodiment, in order to acquire a distance image, an infrared irradiation camera is used separately from a single viewpoint camera that acquires a color image. In the first embodiment, the distance image is acquired at a constant time period, and the frame rate of the single-view image to be output is controlled. The frame rate of the single-view image to be output is equal to or higher than the frame rate of the distance image. In the second embodiment, it is assumed that the single-viewpoint image and the distance image are acquired synchronously at the same frame rate, and the frame rate determination unit controls the frame rates of both the single-viewpoint image and the distance image.

  FIG. 11 is a block diagram illustrating a configuration of an imaging device 1109 and an image processing device 1112 that can be applied to the second embodiment. Hereinafter, parts different from FIG. 1 will be described.

  The single viewpoint image acquisition unit 1101 sequentially acquires single viewpoint images from the imaging unit 1111 according to the frame rate obtained from the frame rate determination unit 1105. The single viewpoint image acquired here is image data composed of RGB. The single viewpoint image acquisition unit 1101 outputs the single viewpoint image to the image encoding unit 1104. The distance image acquisition unit 1102 acquires an infrared image from the imaging unit 1111 according to the frame rate obtained from the frame rate determination unit 1105. The distance image acquisition unit 102 generates a distance image based on the infrared image. As a method for generating a distance image from an infrared image, for example, there is a Structured Light method, a Time Of Flight (TOF) method, or the like. Note that the single viewpoint image acquired by the single viewpoint image acquisition unit 1101 and the infrared image acquired by the distance image acquisition unit 1102 are captured at substantially the same timing, and the frame rates correspond to each other. The distance image acquisition unit 1102 outputs the distance image to the frame rate determination unit 1103 and the image encoding unit 1104. The image encoding unit 1104 encodes both the single viewpoint image and the distance image. The infrared irradiation type distance image is encoded such that the distance accuracy is higher than that of the multi-viewpoint type distance image of the first embodiment and can be used for generating a three-dimensional shape in the subsequent stage. A three-dimensional shape generation unit 1107 generates a three-dimensional shape more easily using a distance image acquired simultaneously with the multi-viewpoint image.

  Next, a flow of processing executed by the imaging apparatus 1109 in the present embodiment is shown. In step S <b> 1201, the single viewpoint image acquisition unit 1101 and the distance image acquisition unit 1102 each acquire a single viewpoint image and a distance image at the same timing from the imaging unit 1111. In step S1202, the image encoding unit 1104 encodes a single viewpoint image. In step S1203, the frame rate determination unit 1103 calculates the complexity of the subject shown in the distance image from the distance image. In step S1204, the frame rate determination unit 1103 calculates a delay time until the next color image and distance image are acquired. This means that the frame rate changes in units of one frame. The update of the delay time is not necessarily performed every frame, and may be performed once every constant time, for example, or may be performed once every fixed frame. In step S1205, the image encoding unit 1104 encodes the distance image. In step S1206, the imaging apparatus 1109 confirms the shooting end signal. If the shooting is completed, the imaging apparatus 1109 ends the process. If not, the process returns to step S1201.

  Next, the image processing apparatus 1112 in the second embodiment will be described. The image processing device 1112 in the second embodiment acquires a distance image from the imaging device 1109. As in the first embodiment, the external parameter estimation S302 can estimate motion information by matching local feature amounts between color images, but it can also be estimated by aligning point groups obtained from distance images. it can.

  The dense point cloud generation S303 representing the surface of the subject can easily generate the point cloud by projecting the first distance information onto the global coordinate system using the external parameters. However, at this time, the second distance information is necessary to complement the point group of the area where distance information was not obtained from the first distance image due to problems of low reliability or subject material. It is also possible to generate and use only.

  This is the end of the description of the present embodiment. With the above configuration, the subject is photographed while the frame rate is dynamically controlled according to the complexity of the shape of the subject, as in the first embodiment. Thereby, the data amount to preserve | save can be suppressed appropriately, acquiring the information required for calculation of a distance image. Furthermore, in the present embodiment, it has become possible to greatly reduce the processing for generating the point cloud representing the subject surface required in the first embodiment.

DESCRIPTION OF SYMBOLS 101 Multi viewpoint image acquisition part 102 Distance image generation part 103 Frame rate determination part 104 Single viewpoint image generation part 105 Image encoding part 110 Control apparatus

Claims (13)

A control device that acquires a frame from an imaging device that captures a moving image for generating a distance image of a subject and controls output of the moving image,
Acquisition means for acquiring a distance image corresponding to a frame constituting a moving image captured by the imaging device;
Calculation means for calculating the complexity of the shape of the subject in the distance image based on the distance image;
Determining means for determining a frame rate of a moving image to be output according to the complexity;
A control device comprising: a generating unit configured to acquire a frame from a moving image captured by the imaging device according to the frame rate acquired by the determining unit and generate a moving image at the frame rate.   The control device according to claim 1, wherein the determination unit determines the frame rate of the moving image to be a higher frame rate as the complexity increases. The determining means determines the number of frames to be acquired from the imaging device after the frame corresponding to the distance image acquired by the acquiring means based on the complexity,
The control device according to claim 1, wherein the generation unit acquires the number of frames from the frame corresponding to the distance image acquired by the acquisition unit. The acquisition means acquires a distance image at a predetermined time among distance images corresponding to frames constituting a moving image captured by the imaging device,
The determination means determines the number of frames each time the acquisition means acquires a distance image,
The said generation means acquires the frame of the said frame number between the time when the said acquisition means acquired the distance image, and produces | generates the moving image from which the said frame number changes dynamically. Control device.   5. The control device according to claim 1, wherein the calculation unit calculates the complexity based on a number of non-measurable pixels in the distance image.   The control device according to claim 1, wherein the calculation unit detects an edge pixel in the distance image and calculates the complexity based on the number of the edge pixels.   The control device according to claim 1, wherein the calculation unit estimates a movement direction of the imaging device and detects an edge pixel in a specific direction according to the movement direction. .   8. The generation unit according to claim 1, wherein the generation unit encodes a distance image acquired by the acquisition unit and a frame from a moving image captured by the imaging device and outputs the frame as a moving image. Control device.   The generating means sets one GOP from a frame captured at the same time as the distance image acquired by the acquiring means to a frame immediately before a frame imaged at the same time as the distance image acquired by the acquiring means, 9. The control apparatus according to claim 8, wherein an image acquired simultaneously with the distance image is encoded as an I frame.   The control device according to claim 9, wherein the generation unit stores information indicating the frame rate calculated by the calculation unit in a header of the GOP.   The said acquisition means acquires the several image imaged from the mutually different viewpoint from the said imaging device, and produces | generates the said distance image based on the said several image, The one of Claims 1 thru | or 10 characterized by the above-mentioned. The control device described in 1.   A computer program for causing a computer to function as the control device according to any one of claims 1 to 11 by being read and executed by a computer. A control method for acquiring a frame from an imaging device that captures a moving image for generating a distance image of a subject and controlling the output of the moving image,
Obtaining a distance image corresponding to a frame constituting a moving image captured by the imaging device;
Based on the distance image, calculate the complexity of the shape of the subject in the distance image,
According to the complexity, determine the frame rate of the output video,
A control method comprising: obtaining a frame from a moving image captured by the imaging device according to the frame rate, and generating a moving image at the frame rate.