JP5724677B2 - Moving image photographing apparatus and moving image photographing method - Google Patents

Description

  The present disclosure relates to a moving image capturing apparatus and a moving image capturing method having two image sensors.

  An imaging element mounted on an imaging apparatus such as a portable terminal or an electronic camera may be overheated if driven for a long time. In order to prevent overheating of the image sensor, conventionally, there has been proposed a technique for limiting the continuous shooting time of moving images by a control program such as an electronic camera.

  In addition, when using a videophone service using a mobile terminal having a plurality of image sensors, a technique for limiting the duration of a mode in which a plurality of image sensors are simultaneously driven based on a temperature time table has been proposed. (See Patent Document 1).

JP 2007-195043 A

  By the way, in order to capture a three-dimensional (3D) moving image, a moving image capturing device having two imaging units arranged with parallax has been put into practical use. The two imaging units mounted on such a moving image capturing apparatus can also be used for obtaining a two-dimensional (2D) moving image, respectively.

  An object of the apparatus and method disclosed herein is to realize long-time two-dimensional moving image shooting with a moving image shooting apparatus having two imaging units arranged with parallax.

A moving image capturing apparatus according to one aspect causes two imaging units that capture a common subject from different viewpoints and two imaging units to alternately perform an imaging operation, and outputs an image obtained by the imaging unit that is performing the imaging operation. A generation unit that includes a switching unit and generates two-dimensional moving image data using an image output by the switching unit; and a common subject included in an image obtained by the two imaging units prior to switching. A detection unit that detects a timing at which the parallax becomes smaller than a predetermined threshold as a switching timing, and the switching unit controls switching of imaging operations by the two imaging units based on the switching timing detected by the detection unit. .
The detection unit operates the other imaging unit at a predetermined time interval in a predetermined search period provided before the operation duration time of the imaging unit whose imaging operation is performed by the switching unit reaches a predetermined time limit. A first acquisition unit that acquires an image from the other imaging unit, an evaluation unit that evaluates a parallax between the image obtained by the first acquisition unit and the image output by the switching unit, and an evaluation unit And a specifying unit that specifies the timing at which the parallax between the images obtained by the two imaging units becomes smaller than a predetermined threshold as the switching timing. Alternatively, the generation unit operates the other imaging unit in synchronization with the imaging unit that is imaged by the switching unit in a predetermined transition period set based on the switching timing detected by the detection unit. In the transition period, a second acquisition unit that acquires an image from the other imaging unit, an interpolation unit that generates an interpolation image based on the image output by the switching unit and the image obtained by the second acquisition unit, And an output control unit that outputs two-dimensional moving image data including an interpolation image obtained by the interpolation unit, instead of the image output by the switching unit.

A moving image capturing method according to another aspect is configured such that, when two-dimensional moving image data is acquired by alternately switching two imaging units that capture a common subject from different viewpoints, A timing at which the parallax between images obtained by two imaging units becomes smaller than a predetermined threshold is detected as a switching timing, and switching of operations of the two imaging units is controlled based on the detected switching timing.
In detecting the switching timing, in the predetermined search period provided before the operation duration time of the imaging unit operated by switching reaches the predetermined time limit, the other imaging unit is set at a predetermined time interval. To obtain the image by the other imaging unit, evaluate the parallax between the acquired image and the image obtained by the imaging unit operated by switching, and based on the evaluation result A timing at which the parallax between images obtained by the two imaging units becomes smaller than a predetermined threshold is specified as a switching timing. Alternatively, when acquiring the two-dimensional moving image data, in the predetermined transition period set based on the detected switching timing, the other imaging unit is synchronized with the imaging unit operated by switching. By operating, an image is acquired by the other imaging unit, and an interpolated image is generated based on the acquired image and an image obtained by the imaging unit that is operated by switching. The two-dimensional moving image data including the generated interpolated image is output instead of the image obtained by the image pickup unit operated by.

  According to the apparatus and method of the present disclosure, it is possible to capture a long-time two-dimensional moving image by using a moving image capturing apparatus having two imaging units arranged with parallax.

It is a figure which shows one Embodiment of a moving image imaging device. It is a figure explaining a moving image imaging operation. It is a figure which shows an example of the state transition diagram of a moving image imaging device. It is a figure which shows an example of the hardware constitutions of a moving image imaging device. It is a flowchart of the process which switches an imaging part. It is a figure explaining the process which evaluates parallax. It is a flowchart of the process which evaluates parallax. It is a figure explaining the production | generation process of an interpolation image. It is a flowchart of the process which produces | generates an interpolation image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an embodiment of a moving image photographing apparatus.

  The moving image capturing apparatus illustrated in FIG. 1 includes two imaging units 101L and 101R, a generation unit 110, and a detection unit 120. The two imaging units 101L and 101R are arranged so as to have a parallax corresponding to the left and right eyes of a human for the purpose of capturing a three-dimensional moving image or a three-dimensional still image, for example.

  The generation unit 110 causes one of the two imaging units 101L and 101R to perform an imaging operation and sets the other to a standby state, thereby switching the two imaging units 101L and 101R to operate. Further, the generation unit 110 generates a two-dimensional moving image mv based on images obtained by the imaging units 101L and 101R. Then, the generation unit 110 outputs the generated two-dimensional moving image mv via the output terminal Vout.

  Based on the images obtained by the two imaging units 101L and 101R, for example, the detection unit 120 switches the timing at which the parallax for the subject captured in common by the imaging units 101L and 101R is equal to or less than a predetermined threshold. Detected as timing Tc. For example, when the imaging unit 101L is operating, the detection unit 120 switches to the imaging unit 101R before the continuous operation time of the imaging unit 101L reaches the time limit set to prevent overheating. The timing Tc may be detected.

  Then, based on the switching timing Tc detected by the detection unit 120, the generation unit 110 controls switching of the operation states of the two imaging units 101L and 101R. Thereby, for example, when the parallax of the subject common to two images obtained by the imaging unit 101L in operation and the imaging unit 101R in standby decreases, the generation unit 110 puts the imaging unit 101L in a standby state. Switching, the imaging unit 101R can be switched to the operating state.

  As described above, the moving image capturing apparatus disclosed herein searches for a suitable switching timing Tc prior to switching between the two imaging units 101L and 101R, and performs two imaging based on the switching timing Tc detected in this search. Switching between the units 101L and 101R is executed. Thereby, the production | generation part 110 of the moving image imaging device of this indication can produce | generate the two-dimensional moving image mv with few discontinuities by switching.

  In this way, the moving image capturing apparatus disclosed in the present disclosure is a long-time two-dimensional image in which changes in images before and after switching are suppressed while switching the two imaging units 101L and 101R arranged with parallax. A moving image can be taken.

  Next, the generation unit 110 and the detection unit 120 included in the moving image shooting apparatus disclosed herein will be described.

  The generation unit 110 includes a switching unit 102, a second acquisition unit 111, an interpolation unit 112, and an output control unit 113.

  The switching unit 102 alternately drives the two imaging units 101L and 101R mounted on the moving image capturing apparatus. In the following description, of the two image capturing units 101L and 101R, the image capturing operation performed by the switching unit 102 is referred to as an active image capturing unit 101, and the other is referred to as a standby image capturing unit 101. The switching unit 102 controls switching of the operation states of the two imaging units 101L and 101R based on the switching timing Tc detected by the detection unit 120.

  In addition, the switching unit 102 acquires the image of each frame from the operating imaging unit 101 by driving the operating imaging unit 101 at a predetermined frame rate for moving image shooting, for example. Are output sequentially. For example, when the imaging unit 101L is the active imaging unit 101, the switching unit 102 sequentially outputs the images GL obtained by the imaging unit 101L at every frame interval tf corresponding to the above-described frame rate. Note that the frame rate applied to the imaging unit 101 in which the switching unit 102 is operating may be, for example, 30 frames per second.

  The image output by the switching unit 102 is input to the output control unit 113 and the interpolation unit 112.

  In addition, the second acquisition unit 111 changes the standby imaging unit 101 to the imaging operation of the imaging unit 101 that is operating over a transition period having a predetermined time set after the switching timing Tc detected by the detection unit 120. Drive synchronously. Accordingly, the second acquisition unit 111 acquires, from the standby imaging unit 101, an image that is captured simultaneously with the operating imaging unit 101 over the transition period described above.

  The interpolating unit 112 generates an interpolated image Gint by combining the image acquired by the second acquiring unit 111 and the image obtained by the imaging unit 101 in operation output by the switching unit 102. In the following description, an image obtained by the imaging unit 101 in operation among the images to be combined by the interpolation unit 112 is referred to as a switching source image GB. On the other hand, the image acquired by the second acquisition unit 111 is an image obtained by the standby imaging unit 101 that is the switching destination, and is referred to as a switching destination image GF.

  Further, the output control unit 113 illustrated in FIG. 1 switches between the image output by the switching unit 102 and the interpolation image Gint generated by the interpolation unit 112 in accordance with the switching timing Tc detected by the detection unit 120. Output. The output control unit 113 normally generates the two-dimensional moving image mv by sequentially outputting the image of each frame output by the switching unit 102. On the other hand, over the transition period described above, the output control unit 113 sequentially outputs the interpolation image Gint of each frame obtained by the interpolation unit 112 as a part of the two-dimensional moving image mv instead of the output of the switching unit 102.

  FIG. 2 is a diagram illustrating a moving image shooting operation. In FIG. 2, an image GL obtained by the imaging operation by the imaging unit 101L is shown by a white rectangle. A shaded rectangle shown in FIG. 2 indicates an image GR obtained by an imaging operation by the imaging unit 101R.

  FIG. 2 shows images GL and GR obtained by the respective imaging units 101L and 101R in the process of switching the imaging unit 101 in operation from the imaging unit 101L to the imaging unit 101R, and a two-dimensional video image output from the video imaging device. The relationship with mv is shown.

  Moreover, the code | symbol Tc shown in FIG. 2 shows the switching timing detected by the detection part 120. FIG. In the example illustrated in FIG. 2, the two-dimensional moving image mv output by the output control unit 113 in the period before the switching timing Tc is an image GL obtained by the imaging unit 101L that is the imaging unit 101 in operation. Based on.

  For example, as illustrated in FIG. 2, the second acquisition unit 111 drives the standby imaging unit 101 at a frame rate equivalent to that of the operating imaging unit 101 after a predetermined time tp has elapsed from the switching timing Tc. The operation may be started. In the example of FIG. 2, the transition period is a period of time tr after the time tp has elapsed from the switching timing Tc. The length of the transition period may be, for example, a natural number m times the frame interval tf described above. In this transition period, the second acquisition unit 111 acquires an image of each frame obtained for each frame interval tf by the standby imaging unit 101 as a switching destination image GF.

  Then, as shown in FIG. 2, the interpolation unit 112 shown in FIG. 1 converts the images GL and GR obtained from the two imaging units 101L and 101R at the frame intervals tf, respectively, into the switching source image GB and the switching destination, respectively. Is synthesized as an image GF. By this synthesis processing, the interpolation unit 112 generates an interpolated image Gint corresponding to a virtual imaging unit having an intermediate viewpoint between the two imaging units 101L and 101R at every frame interval tf.

  Over the transition period described above, the output control unit 113 outputs the interpolation image Gint generated by the interpolation unit 112 as a part of the two-dimensional moving image mv instead of the image output by the switching unit 102. In the example of FIG. 2, a rectangle represented by a double line indicates an interpolation image Gint generated by the interpolation unit 112 during the transition period.

  Thus, after the transition period elapses, the switching unit 102 switches the operation states of the two imaging units 101L and 101R. For example, the switching unit 102 stops the driving operation for the imaging unit 101 that has been operating until then, and starts the driving operation for the imaging unit 101 that is on standby.

  For example, in the example illustrated in FIG. 2, an arrow with a symbol Ta indicates a timing at which switching from the imaging unit 101L to the imaging unit 101R is executed. At this timing Ta, the switching unit 102 stops the imaging operation of the imaging unit 101L, while continuing the imaging operation of the imaging unit 101R that started the operation during the transition period. As a result, as shown in FIG. 2, after the timing Ta described above, the image GR obtained by the imaging unit 101R is output as a part of the two-dimensional moving image mv at every frame interval tf.

  On the other hand, the detection unit 120 illustrated in FIG. 1 includes a first acquisition unit 121, an evaluation unit 122, a specification unit 123, and a timer 124.

  For example, the first acquisition unit 121 starts an operation of periodically driving the imaging unit 101 that is on standby in response to the control signal ST from the timer 124. The first acquisition unit 121 periodically acquires an image obtained by the standby imaging unit 101 by, for example, driving the standby imaging unit 101 at every predetermined time interval td. The time interval td may be set to a natural number n times the frame interval tf, for example.

  The timer 124 shown in FIG. 1 holds an internal value indicating the remaining time Trem that allows the imaging unit 101 in operation to continue the imaging operation, and the internal value is decreased as time elapses. I do. The timer 124 corresponds to, for example, a time limit set in order to prevent the imaging unit 101 from being overheated according to the control signal RS indicating the timing at which the switching unit 102 described above performs switching of the imaging unit 101. A value may be set as an initial value. In addition, the timer 124 starts the operation of the first acquisition unit 121 by outputting the control signal ST when the internal value reduced with the passage of time reaches a value indicating the predetermined time Ds. Let

  For example, the symbol Ts illustrated in FIG. 2 indicates the timing at which the time during which the imaging unit 101L that is operating continues the imaging operation reaches the time limit set to prevent the imaging unit 101 from overheating. . The above-described timer 124 starts the operation of the first acquisition unit 121 by outputting the control signal ST at a time point that is earlier than the timing Ts by the above-described time Ds.

  In this way, while the imaging operation by the imaging unit 101 in operation can be continued, the operation of acquiring the image of the imaging unit 101 on standby by the first acquisition unit 121 can be started.

  In addition, the evaluation unit 122 illustrated in FIG. 1 compares the image output by the switching unit 102 with the image acquired by the first acquisition unit 121 so that the two imaging units 101L and 101R can capture the same. Evaluate the parallax for the subject. Based on the evaluation result by the evaluation unit 122, the specifying unit 123 specifies, as the switching timing Tc, a timing at which the parallax for the subject captured by the two image pickup units 101L and 101R in common is smaller than a predetermined threshold. In this way, the detection unit 120 can detect the switching timing Tc at which the parallax for the subject captured by the two imaging units 101L and 101R in common is smaller than the predetermined threshold.

  Reference numerals T1, T2, and T3 illustrated in FIG. 2 indicate examples of timing at which the above-described first acquisition unit 121 acquires the image GR from the imaging unit 101R that is on standby. The example of FIG. 2 illustrates a case where the switching timing Tc is detected based on the result of comparing the images GL and GR respectively obtained from the two imaging units 101L and 101R at the timing T3.

  Based on the switching timing Tc detected in this way, the generation unit 110 causes the two imaging units 101L and 101R to pass through a transition period in which the interpolation image Gint generated by the interpolation unit 112 is output as described above. Switch.

  By providing such a transition period prior to switching between the two imaging units 101L and 101R mounted on the moving image capturing apparatus, the influence of the change in the viewpoint caused by the switching on the output two-dimensional moving image mv is affected. Further suppression can be achieved. As a result, the moving image capturing device disclosed herein can output a two-dimensional moving image maintaining natural continuity while switching between the two imaging units 101L and 101R having different viewpoints.

  FIG. 3 shows an example of a state transition diagram of the moving image shooting apparatus disclosed herein.

  A symbol PL illustrated in FIG. 3 indicates a state where the imaging unit 101L illustrated in FIG. A symbol Ps (L → R) indicates a state in which the switching timing from the imaging unit 101L to the imaging unit 101R is searched. A symbol Pt (L → R) indicates a state corresponding to a transition period prior to switching from the imaging unit 101L to the imaging unit 101R. Reference sign PR indicates a state in which the imaging unit 101R illustrated in FIG. A symbol Ps (R → L) indicates a state in which the switching timing from the imaging unit 101R to the imaging unit 101L is being searched for. A symbol Pt (R → L) indicates a state corresponding to a transition period prior to switching from the imaging unit 101R to the imaging unit 101L.

  As shown in FIG. 3, the state of the moving image capturing device disclosed herein is as follows: state PL, state Ps (L → R), state Pt (L → R), state PR, state Ps (R → L), state In the process of making a cyclic transition of Pt (R → L), the two imaging units 101L and 101R are switched. As a result, the moving image shooting device disclosed herein can realize long-time two-dimensional moving image shooting, regardless of restrictions on the continuous operation time for preventing overheating set in the individual imaging units 101L and 101R. Can do. In addition, the code | symbol PL shown in FIG. 2 shows the period when the state of a moving image imaging device is the state PL. Moreover, the code | symbol Ps (L-> R) shown in FIG. 2 shows the period when the state of a moving image imaging device is state Ps (L-> R). Similarly, a symbol Pt (L → R) illustrated in FIG. 2 indicates a period in which the state of the moving image capturing apparatus is the state Pt (L → R). 2 indicates a period in which the state of the moving image capturing apparatus is the state PR.

  FIG. 4 shows an example of the hardware configuration of the moving image capturing apparatus.

  The moving image capturing apparatus illustrated in FIG. 4 includes two imaging units 101L and 101R, an image processing processor 21, a memory 22, a recording processing unit 23, and a memory controller 24. The two imaging units 101L and 101R, the processor 21, the memory 22, and the recording processing unit 23 are connected to each other via a memory controller 24.

  The moving image shooting apparatus illustrated in FIG. 4 is, for example, a digital video camera or a digital camera having a three-dimensional moving image shooting function. The two imaging units 101L and 101R are arranged so as to have a parallax corresponding to the left and right eyes of a human for the purpose of capturing a three-dimensional moving image or a three-dimensional still image, for example. The recording processing unit 23 performs recording processing and reading processing of image data and the like with a detachable recording medium such as a memory card, for example.

  Further, the moving image photographing apparatus may include a processor 28 for controlling the apparatus, an external interface (external I / F) 29, and an operation panel 30. The processor 28, the external interface 29, the operation panel 30, and the memory controller 24 are connected to each other via a bus. The external interface 29 mediates exchange of data between, for example, a personal computer and the memory 22 under the control of the processor 28.

  The memory 22 stores a switching processing program for operating the two imaging units 101L and 101R alternately. The processor 21 implements the functions of the generation unit 110 and the detection unit 120 illustrated in FIG. 1 by controlling the operations of the imaging units 101L and 101R according to the switching processing program.

  For example, the switching process program may be read using a personal computer or the like using the external interface 29, and the read switching process program may be stored in the memory 22.

  The imaging unit 101R illustrated in FIG. 4 includes an imaging element 25R, a drive control unit 26R, and a format conversion unit 27R. Similarly, the imaging unit 101L includes an imaging element 25L, a drive control unit 26L, and a format conversion unit 27L.

  The drive controllers 26R and 26L drive the imaging operation by the imaging elements 25R and 25L, respectively. Further, the drive control units 26R and 26L generate images GR and GL from signals obtained by the imaging operation by the imaging elements 25R and 25L. The format conversion units 27R and 27L convert the data format of the images GR and GL generated by the drive control units 26R and 26L into an output data format such as a YCbCr format.

  When the moving image capturing apparatus captures a three-dimensional moving image, the imaging units 101R and 101L operate in parallel at a predetermined frame rate for moving image capturing. Then, the format conversion units 27R and 27L store the images GR and GL converted into the output data format in the memory 22 via the memory controller 24 for each frame interval tf corresponding to the frame rate for moving image shooting. Store. In addition, the recording processing unit 23 performs a process of recording the images GR and GL in the output data format stored in the memory 22 on a recording medium. Note that the recording processing unit 23 may perform compression processing based on, for example, the MPEG method in the process of recording the recording medium of the images GR and GL.

  On the other hand, when the moving image capturing apparatus captures a two-dimensional moving image, the processor 21 performs processing for switching the image capturing units 101R and 101L alternately.

  Note that the moving image capturing apparatus, for example, a mode for capturing a three-dimensional moving image and a two-dimensional moving image in response to a mode switching instruction input by the photographer via the operation panel 30 or the like under the control of the processor 28. The operation mode may be switched to a mode for shooting. Further, the processor 28 may capture a 3D still image and a 2D still image by controlling each part of the moving image capturing apparatus.

  For example, when switching to a mode for capturing a two-dimensional moving image is instructed by a mode switching instruction, the processor 21 performs a process of generating a two-dimensional moving image by alternately switching the two imaging units 101L and 101R. Start.

  When the continuous operation time of one imaging unit 101 approaches the time limit set for preventing overheating, the processor 21 starts executing the switching process program. For example, as described for the timer 124 shown in FIG. 1, the processor 21 starts executing the switching processing program when the remaining time Trem obtained by subtracting the continuous operation time from the time limit becomes the time Ds. May be. Note that the time Ds may be determined in consideration of, for example, the time required to search for a suitable timing for switching between the two imaging units 101L and 101R and the transition period described above.

  FIG. 5 shows a flowchart of processing for switching the imaging units 101R and 101L.

  First, the processor 21 sets an initial value 0 to a variable t for controlling the timing of acquiring an image from the imaging unit 101 that is on standby (step S11). Next, the processor 21 outputs an image obtained by the imaging unit 101 in operation as a part of the two-dimensional moving image mv (step S12). In step S <b> 12, the processor 21 may cause the format conversion unit 27 included in the operating imaging unit 101 to output the converted image via the memory controller 24.

  Thereafter, the processor 21 determines whether or not the variable t is less than the time interval td for acquiring an image from the imaging unit 101 that is on standby (step S13). The time interval td may be an integer n times the frame interval tf as shown in FIG. 2, for example. When the value of the variable t is less than the time interval td (Yes determination in step S13), the processor 21 adds the frame interval tf to the variable t in step S14, and then returns to the process of step S12.

  When the steps S11 to S14 shown in FIG. 5 are repeated and an image of n frames obtained by the imaging unit 101 in operation is output, the value of the variable t described above becomes equal to the time interval td. At this time, the processor 21 proceeds to the process of step S15 as an affirmative determination of step S13.

  In step S <b> 15, the processor 21 drives the imaging unit 101 that is on standby, and acquires an image of one frame. For example, the processor 21 temporarily drives the imaging device 25 via the drive control unit 26 included in the standby imaging unit 101, the format conversion unit 27 included in the standby imaging unit 101, the memory 22, and the like. An image of one frame may be acquired via the memory controller 24. Thus, the processor 21 may realize the function of the first acquisition unit 121 illustrated in FIG. 1 by executing the process of step S15.

  Next, the processor 21 evaluates the parallax of the two imaging units 101R and L with respect to the subject captured in common in these images based on the image obtained in step S15 and the image obtained in step S12. (Step S16). Thus, the processor 21 may implement the function of the evaluation unit 122 illustrated in FIG. 1 by executing the process of step S16. Details of the process in step S16 will be described later.

  Then, the processor 21 can switch from the active imaging unit 101 to the standby imaging unit 101 based on the comparison result between the magnitude of the parallax indicated by the evaluation result obtained in step S16 and a predetermined threshold value. It is determined whether or not (step S17). Note that, in step S17, a method for determining a threshold value for the processor 21 to compare with the evaluation result obtained in step S16 will be described later.

  When the magnitude of the parallax indicated by the evaluation result obtained in step S16 is larger than the above-described threshold value (negative determination in step S17), the processor 21 proceeds to step S18. In step S <b> 18, the processor 21 determines whether or not the remaining time Trem in which the imaging unit 101 in operation can be continuously operated is larger than the above-described time De. In step S18, the time De that the processor 21 compares with the remaining time Trem is set to a time that is shorter than the time Ds shown in FIG. 2 and longer than the time tr corresponding to the transition period, for example. May be.

  When the remaining time Trem corresponding to the imaging unit 101 in operation is larger than the above-described time De (positive determination in step S18), the processor 21 returns to the process in step S11. Then, the processor 21 starts a process of acquiring a new image from the imaging unit 101 that is on standby.

  In this way, by repeating the processing of step S11 to step S18, the processor 21 acquires the image GR obtained by the imaging unit 101R that is the standby imaging unit 101 at the time interval td described above.

  For example, when the evaluation result obtained for the image GR and the image GL acquired at timing T3 in FIG. 2 is equal to or less than the above-described threshold value (Yes determination in step S17), the processor 21 captures an image from the image capturing unit 101L. It is determined that switching to the unit 101R is possible. As described above, the function of the specifying unit 123 illustrated in FIG. 1 may be realized by the processor 21 executing the process of step S17.

  If the determination in step S17 is affirmative, the processor 21 proceeds to the process of step S19 shown in FIG. In step S <b> 19, the processor 21 acquires an image obtained by the imaging unit 101 in operation as the switching source image GB and acquires an image obtained by the standby imaging unit 101 as the switching destination image GF. For example, the processor 21 may drive the imaging element 25 at a frame rate for moving image shooting via the drive control unit 26 included in the standby imaging unit 101. The processor 21 acquires the image GR obtained by the drive control unit 26R by this drive processing as the switching destination image GF. Thus, the processor 21 may realize the function of the second acquisition unit 111 illustrated in FIG. 1 by executing the process of step S19.

  Next, the processor 21 generates an interpolated image Gint based on the switching source image GB and the switching destination image GF acquired in step S19 (step S20). Thus, the processor 21 may implement the function of the interpolation unit 112 illustrated in FIG. 1 by executing the process of step S20. Details of the processing in step S20 will be described later.

  Next, the processor 21 outputs the interpolated image Gint generated in step S20 as part of the two-dimensional moving image data instead of the switching source image GB (step S21). Thus, the function of the output control unit 113 shown in FIG. 1 may be realized by the processor 21 executing the process of step S21.

  After that, the processor 21 repeats the processing from step S19 to step S22 until it is determined in step S22 that the transition period having the time tr illustrated in FIG. 2 has ended.

  For example, in the transition period Pt (L → R) shown in FIG. 2, the processor 21 repeats the processing from step S19 to step S22. In the course of this process, the processor 21 generates the interpolated image Gint of each frame by executing the process of step S20 each time the image GL and the image GR of each frame are acquired from the imaging unit 101L and the imaging unit 101R. To do. Then, the processor 21 outputs the interpolation image Gint of each frame generated in this way as a part of the two-dimensional moving image mv.

  Note that the processor 21 may provide a predetermined spare time tp as shown in the example of FIG. 2 prior to driving the standby imaging unit 101 at the moving image shooting frame rate. In this case, the processor 21 starts the transition process from step S19 to step S22 after the preliminary time tp has elapsed since the switchable timing T3 was detected as described above. On the other hand, when the above-described preliminary time tp is not necessary, the processor 21 may start the transition process of step S19 to step S22 when detecting a switchable timing in step S17.

  When the time tr elapses after the start of the transition process from step S19 to step S22 (affirmative determination in step S22), the processor 21 performs a process of switching the operation states of the two imaging units 101R and 101L (step S23). ). For example, the processor 21 puts the imaging unit 101L into a sleep state via the drive control unit 26L in the operating imaging unit 101L. On the other hand, the processor 21 continues the imaging operation by the imaging unit 101R via the drive control unit 26R in the imaging unit 101R. In this way, the state of the moving image capturing apparatus after the roles of the imaging unit 101L and the imaging unit 101R are switched by the processor 21 transitions to the single operation state PR of the imaging unit 101R.

  In the single operation period PR shown in FIG. 3, the imaging unit 101R generates an image GR at a frame interval tf corresponding to the frame rate for moving image shooting, instead of the imaging unit 101L. This image GR is stored in the memory 22 via the memory controller 24 as it is as a part of the two-dimensional moving image mv.

  As described above, the moving image shooting apparatus disclosed herein can realize switching from the imaging unit 101L to the imaging unit 101R. Then, the imaging unit 101R is set as the active imaging unit 101, the imaging unit 101L is operated as the standby imaging unit 101, and then the switching process described above is performed again, whereby the switching from the imaging unit 101R to the imaging unit 101L is performed. Can be realized.

  As described above, the moving image capturing apparatus having the hardware configuration illustrated in FIG. 4 can be operated while the two imaging units 101L and 101R are alternately switched in the process of cyclically changing the six states illustrated in FIG. It is possible to realize a moving image shooting apparatus that performs time-based two-dimensional moving image shooting.

  Note that, as in the flowchart illustrated in FIG. 5, even in the case of a negative determination in step S <b> 17, when it is determined that the remaining time Trem is equal to or less than the time De (negative determination in step S <b> 18), the processor 21 performs steps S <b> 19 to S <b> 19. You may perform the transfer process of step S22.

  For example, a time obtained by adding a predetermined grace time to the time tr corresponding to the transition period described above may be set as the time De. In this case, the processor 21 reliably completes switching to the standby imaging unit 101 before the continuous operation time of the imaging unit 101 in operation reaches the time limit set to prevent overheating. be able to.

  Further, when the evaluation result obtained in step S16 indicates that the parallax between the two imaging units 101L and 101R is extremely small, the processor 21 may skip the transition process from step S19 to step S22. Good. For example, when photographing a landscape such as a distant mountain range, the parallax between the two imaging units 101L and 101R with respect to the landscape that is the main subject becomes very small. Therefore, the processor 21 compares the evaluation result obtained in step S16 with the threshold corresponding to the result of evaluating the parallax of the two imaging units 101L and 101R in such a case, thereby performing the transition process described above. It may be determined whether or not to skip.

  Further, instead of the processor 21 illustrated in FIG. 4, hardware such as a digital signal processor (DSP) incorporating a program for switching the imaging unit illustrated in FIG. 5 may be used.

  Next, processing for evaluating the parallax between the two imaging units 101L and 101R will be described.

  FIG. 6 illustrates a process for evaluating the parallax. In FIG. 6, symbols GL and GR indicate images GL and GR obtained by the imaging units 101L and 101R, respectively. Further, in each image GL shown in FIG. 6, the code MB is one of blocks obtained by dividing the image GL into small regions of N × N pixels. The size of each block may be, for example, 16 pixels × 16 pixels. In FIG. 6, the size of each block is shown enlarged for easy viewing.

  FIGS. 6A and 6B are examples of images GL and GR obtained by simultaneously photographing a person at a short distance with a landscape including a tree as a background by the two imaging units 101L and 101R.

  For example, the parallax regarding the person included in common in the images GL and GR shown in FIGS. 6A and 6B performs block matching with the image GR for the block MB included in the person portion of the image GL. Can be obtained based on the parallax vector Vd obtained in this way. In FIG. 6B, a symbol MB ′ indicates an area detected from the image GR by block matching for the block MB included in the image GL. The disparity vector Vd with respect to the image GR for the block MB of the image GL is obtained, for example, as a vector indicating the position of the upper left pixel of the region MB ′ of the image GR with reference to the position of the upper left pixel of the block MB of the image GL. be able to.

  FIGS. 6C and 6D are examples of images GL and GR obtained by simultaneously capturing images of a vehicle that is close to a background tree by the two imaging units 101L and 101R. Further, a code symbol MB ′ illustrated in FIG. 6D indicates a region detected from the image GR by block matching for the block MB included in the image GL. And the code | symbol Vd shown in FIG.6 (C) shows the parallax vector about the block part MB containing a part of image of the vehicle contained in the image GL.

  As can be seen from the comparison of the disparity vectors Vd shown in FIGS. 6A and 6C, compared to the disparity for the person included in the images GL and GR shown in FIGS. The parallax for the vehicles included in the images GL and GR shown in 6 (C) and (D) is small. As described above, when the parallax of the subject included in common in the images GL and GR obtained simultaneously by the two imaging units 101L and 101R is small, if the imaging units 101L and 101R are switched, the images before and after the switching are suddenly changed. Change can be suppressed.

  By the way, when the tree parts commonly included in the images GL and GR shown in FIGS. 6A and 6B are compared with each other, it is understood that the parallax for the tree part is smaller than the parallax for the person. . Similarly, in the images GL and GR shown in FIGS. 6C and 6D, the parallax for the tree portion is different from the parallax for the vehicle.

  As described above, since the parallax of the subjects included in common in the images GL and GR obtained simultaneously by the two imaging units 101L and 101R is different, the parallax between the images GL and GR is different for each subject. It is desirable to evaluate based on the parallax vector.

  The magnitude of the parallax between the image GL and the image GR can be evaluated based on, for example, the parallax vector Vd obtained for each block included in the image GL. For example, an average value of the parallax vectors Vd obtained for each block may be used as an evaluation index indicating the magnitude of the parallax between the image GL and the image GR.

  In this way, by using the average value of the parallax vectors Vd obtained for each block as one of the evaluation indexes, the parallax for the images GL and GR in which the subject is present at a short distance can be obtained, similar to the human visual impression. It can be determined that the position of the subject is larger than when the subject is close to the background.

  FIG. 7 shows a flowchart of processing for evaluating the parallax between the two imaging units 101L and 101R. The process of step S31 to step S37 illustrated in FIG. 7 is an example of the process of step S16 illustrated in FIG.

  First, the processor 21 divides the switching source image GB obtained by the imaging unit 101 in operation into blocks of N × N pixels (step S31). For example, the processor 21 may divide the switching source image GB into blocks of 16 × 16 pixels or 8 × 8 pixels.

  Next, the processor 21 performs block matching processing on each block obtained by dividing the switching source image GB with the switching destination image GF obtained by the standby imaging unit 101 (step S32). In the process of S32, the processor 21 first sequentially selects one of the blocks included in the switching source image GB. Next, for this block MB, the processor 21 detects the most similar area MB ′ from the switching destination image GF. Then, based on the position of the block MB included in the switching source image GB and the position of the region MB ′ included in the switching destination image GF, a disparity vector Vd corresponding to the block MB is calculated. At this time, the processor 21 adds the absolute value of the difference between each pixel included in the block MB and each pixel included in the region MB ′ to all the pixels included in the block MB, thereby calculating the difference absolute value sum SAD. Calculate (Sum of Absolute Difference).

  Next, the processor 21 holds the disparity vector Vd calculated in the process of step S32, for example, corresponding to the block number for identifying the block MB (step S33). For example, the processor 21 secures an area for holding the disparity vector Vd corresponding to all blocks included in the switching source image GB in the memory 22, and stores the disparity vector Vd calculated in step S32 in this area. May be. Next, the processor 21 adds the difference absolute value sum SAD calculated in step S32 to the total sum S (step S34). Note that the processor 21 may execute the processes in steps S33 and S34 described above in the reverse order.

  Thereafter, the processor 21 determines whether or not the processing from step S32 to step S34 described above has been completed for all blocks included in the switching source image GB (step S35).

  When there is an unprocessed block (No at Step S35), the processor 21 returns to the process at Step S32. And the process of step S32-step S34 is performed about the newly selected block.

  In this way, the processor 21 repeatedly executes the processing from step S32 to step S34 for each block included in the switching source image GB. Then, when the processing in steps S32 to S34 is completed for all the blocks (Yes determination in step S35), the processor 21 proceeds to processing in step S36.

  In step S36, the processor 21 calculates the average parallax vector Vdav based on the parallax vector Vd held for each block in step S33 described above. For example, the processor 21 may obtain the average disparity vector Vdav by reading the disparity vector Vd stored in the memory 22 corresponding to the block number of each block and obtaining the average value of the read disparity vectors Vd.

  Thereafter, the processor 21 outputs the average parallax vector Vdav obtained in step S36 and the sum S as indices indicating the magnitudes of the parallaxes of the two images GB and GF (step S37). Note that the total sum S is obtained by integrating the sum of absolute differences SAD obtained for each block in the process in which the processor 21 executes the processing of steps S32 to S34 described above for all the blocks included in one frame. Can be obtained.

  The average parallax vector Vdav obtained in this way indicates the magnitude of parallax between the switching source image GB and the switching destination image GF, as described with reference to FIG. Further, the sum S, which is a value obtained by integrating the sum of absolute differences SAD included in one frame, becomes a small value when the disparity vector obtained for each block is likely. Conversely, for example, when the likelihood of the disparity vector obtained for each block is low, such as when the photographer's finger is placed on one of the imaging units 101L and 101R, the sum S of the absolute difference value sum SAD is low. Is a large value.

By using the average parallax vector Vdav thus obtained and the sum S of the absolute difference sum SAD as an evaluation index, the processor 21 performs processing for the two images GB and GF in step S17 shown in FIG. Whether or not the parallax is small can be determined with high accuracy.
For example, the processor 21 may use the threshold Thv for the average parallax vector Vdav and the threshold Ths for the sum S of the absolute difference sum SAD as the threshold to be compared with the evaluation result in the process of step S17.

  It should be noted that a value suitable for the threshold Thv for the average parallax vector Vdav and a value suitable for the threshold Ths for the total sum S of absolute differences are, for example, the relationship between the optical axes of the lenses provided in the two image sensors and the imaging It depends on the resolution of the element. For this reason, at the product development stage or the like, threshold values Thv and Ths are set for each moving image shooting device based on a 3D still image obtained by shooting a scene with parallax by a moving image shooting device and a 3D still image obtained by shooting a scene without parallax. It is desirable to decide. For example, the range of the average parallax vector Vd obtained for a 3D still image obtained by photographing a scene with parallax and the range of the distribution of the average parallax vector Vd obtained for a 3D still image obtained by photographing a scene without parallax are obtained. deep. Then, the threshold value Thv for the average parallax vector Vdav may be determined based on the result of comparing these ranges with each other. Similarly, the range of the sum S of the absolute difference sum SAD corresponding to the 3D still image obtained by photographing the scene with parallax and the sum S of the absolute difference sum SAD corresponding to the 3D still image obtained by photographing the scene without the parallax. Find each range. Then, based on the result of comparing these ranges with each other, the threshold Ths for the sum S of the absolute difference value sum SAD may be determined.

  The moving image capturing apparatus disclosed herein determines the switching timing of the two imaging units 101L and 101R based on the evaluation index obtained as described above. Thereby, according to the moving image imaging device of the present disclosure, the two imaging units 101L and 101R can be switched at a timing at which the continuity of the output two-dimensional moving image is easily maintained. Therefore, according to the moving image capturing device of the present disclosure, it is possible to capture a long-time two-dimensional moving image with little discomfort while switching between the two imaging units 101L and 101R.

  Next, an interpolation image generation process will be described.

  FIG. 8 is a diagram illustrating processing for generating an interpolation image. In FIG. 8, symbol GB indicates a switching source image obtained by the imaging unit 101 in operation. Reference numeral GF indicates a switching destination image obtained by the imaging unit 101 in standby. A symbol Gint indicates an interpolation image generated based on the two images GB and GF.

  The code MB shown in FIG. 8 is a block included in the interpolation image Gint to be generated. In the image GB, a block at a position corresponding to the block MB is indicated by a symbol MBb, and in the image GF, a block at a position corresponding to the block MB is indicated by a symbol MBf. Moreover, the code | symbol Vd shown in FIG. 8 shows the parallax vector detected by performing the block matching process with the image GF about the block MBb contained in the image GB.

  A block of the interpolated image Gint is generated by associating the viewpoint of the interpolation image Gint to be generated with a position that internally divides the viewpoint of the switching source image GB and the viewpoint of the switching destination image GF by a ratio α: 1−α. A method for generating each pixel included in the MB will be described.

  In the switching source image GB, the vector Vb from the block MBb to the reference image Bb corresponding to the block MB is expressed by Expression (1) using the parallax vector Vd and the coefficient α. In the switching destination image GF, the vector Vf from the block MBf to the reference image Bf corresponding to the block MB is expressed by the equation (2) using the parallax vector Vd and the coefficient α.

Vf = Vd × (1.0−α) (1)
Vb = −Vd × α (2)
That is, based on the vectors Vb and Vf expressed by the above-described equations (1) and (2), the reference image Bb in the image GB and the reference in the image Gf that are referred to when the block MB of the interpolation image Gint to be generated is generated. Each of the images Bf can be obtained.

  The pixels P (x, y) included in the block MB of the interpolated image Gint are the pixels Pb (x, y) included in the reference image Bb in the image GB and the pixels included in the reference image Bf in the image Gf. Using Pf (x, y), it is expressed as in equation (3).

P (x, y) = β × Pf (x, y) + (1.0−β) × Pb (x, y) (3)
In Equation (3), the value of the coefficient β may be set so as to increase monotonously in the range of 0 to 1 as the coefficient α is increased. By adjusting the coefficients α and β in this way, the interpolation is performed while adjusting the synthesis ratio of the switching source image GB and the switching destination image GF so as to approach the switching destination image GF as the transition period starts and ends. An image Gint can be generated.

  FIG. 9 shows a flowchart of processing for generating an interpolation image. The process of step S41 to step S48 illustrated in FIG. 9 is an example of the process of step S20 illustrated in FIG.

  In step S41, the processor 21 first divides the switching source image GB and the generation target interpolation image Gint acquired in step S19 into blocks of N pixels × N pixels, respectively.

  Next, the processor 21 determines the values of the coefficient α and the coefficient β to be applied to the generation target interpolation image Gint (step S42). Based on the position occupied by the interpolation image Gint to be generated in the transition period in the process of step S42, the processor 21 adjusts the coefficient α and the coefficient α so as to approach the numerical value 1 from the value close to the numerical value 0 as the transition period elapses. The coefficient β may be determined. For example, when generating an M-frame interpolation image during the transition period, the processor 21 determines the values of the coefficients α and β to be applied to the k-th generated interpolation image Gint based on the value k / (M + 1). May be.

  Next, the processor 21 sequentially obtains each disparity vector Vd by performing block matching with the switching destination image GF for the block MBb included in the switching source image GB (step S43). For example, the processor 21 selects one of the blocks MBb included in the switching source image GB as a processing target based on the scanning order. By performing block matching for the selected block MBb, the processor 21 detects the most similar region in the switching destination image GF. Then, the processor 21 may obtain a disparity vector Vd indicating the position of the detected region with reference to the position of the block corresponding to the block MBb described above in the switching destination image GF.

  Next, the processor 21 uses the above-described equations (1) and (2) to determine the positions of the reference images Bb and Bf corresponding to the generation target block MB in the switching source image GB and the switching destination image GF, respectively. The indicated vectors Vb and Vf are obtained (step S44). In the process of step S44, the processor 21 desirably obtains the vectors Vb and Vf with decimal precision.

  Next, the processor 21 generates reference images Bb and Bf based on the vectors Vb and Vf obtained in step S44 (step S45). For example, the processor 21 converts the pixels Pb (x, y) and Pf (x, y) included in the reference images Bb and Bf indicated by the vectors Vb and Vf obtained with decimal precision into the switching source image GB and the switching. It is desirable to obtain the previous image GF by applying an interpolation filter.

  Based on the pixels Pb (x, y) and Pf (x, y) included in the reference images Bb and Bf generated in this way, the processor 21 uses the above-described equation (3) to calculate the interpolated image Gint. Each pixel P (x, y) included in the block MB is generated (step S46).

  Next, the processor 21 determines whether or not the generation of all the pixels included in the generation target block MB has been completed (step S47). When there is an ungenerated pixel (No determination in step S47), the processor 21 returns to the process of step S46 and executes a process of generating the next pixel.

  When the generation of all the pixels included in the generation target block is completed by repeating steps S46 and S47 (Yes determination in step S47), the processor 21 proceeds to the process of step S48. In step S48, the processor 21 determines whether or not the generation of all the blocks MB included in the interpolation image Gint has been completed.

  When there is a block MB that has not yet been generated (No determination in step S48), the processor 21 returns to the process of step S43 and starts a process of generating the next block MB.

  When the generation process shown in steps S43 to S47 is completed for all the blocks MB included in the interpolation image Gint (Yes determination in step S48), the processor 21 ends the generation process of the interpolation image Gint. .

  In this way, the moving image capturing device disclosed herein can generate an interpolated image Gint that gradually approaches the switching destination image GF from the switching source image GB as the transition period elapses. As a result, the moving image capturing device disclosed herein suppresses the change in the viewpoint when the two image capturing units 101L and 101R are switched from appearing as a change in the output image in long-time two-dimensional moving image capturing. be able to. Therefore, according to the moving image capturing device disclosed herein, it is possible to capture a long-time two-dimensional moving image that does not make the viewer feel uncomfortable while switching between the two imaging units 101L and 101R.

101L, 101R ... Imaging unit; 110 ... Generation unit; 111 ... Second acquisition unit; 112 ... Interpolation unit; 113 ... Output control unit; 120 ... Detection unit; 121 ... First acquisition unit; 122 ... Evaluation unit; Unit: 21, 28 ... Processor; 22 ... Memory; 23 ... Recording processing unit; 24 ... Memory controller; 25 ... Imaging device; 26 ... Drive control unit; 27 ... Format conversion unit: 29 ... External interface (I / F); 30 ... Operation panel

Claims (5)

Two imagers that image a common subject from different viewpoints;
A switching unit that alternately images the two imaging units and outputs an image obtained by the imaging unit that is performing the imaging operation;
A generating unit that generates two-dimensional moving image data using the image output by the switching unit;
Prior to the switching, including a detection unit that detects, as a switching timing, a timing at which a parallax for the common subject included in images obtained by the two imaging units is smaller than a predetermined threshold,
The switching unit controls switching of the imaging operation by the two imaging units based on the switching timing detected by the detection unit,
The detector is
The other imaging unit is operated at a predetermined time interval in a predetermined search period provided before the operation continuation time of the imaging unit that is imaged by the switching unit reaches a predetermined time limit. A first acquisition unit that acquires an image from the other imaging unit;
An evaluation unit that evaluates the parallax between the image obtained by the first acquisition unit and the image output by the switching unit;
A moving image comprising: a specifying unit that specifies, as the switching timing, a timing at which a parallax between images obtained by the two imaging units becomes smaller than a predetermined threshold based on an evaluation result by the evaluation unit. Shooting device. Two imagers that image a common subject from different viewpoints;
A switching unit that alternately images the two imaging units and outputs an image obtained by the imaging unit that is performing the imaging operation;
A generating unit that generates two-dimensional moving image data using the image output by the switching unit;
Prior to the switching, including a detection unit that detects, as a switching timing, a timing at which a parallax for the common subject included in images obtained by the two imaging units is smaller than a predetermined threshold,
The switching unit controls switching of the imaging operation by the two imaging units based on the switching timing detected by the detection unit,
The generator is
By operating the other imaging unit in synchronization with the imaging unit being imaged by the switching unit in a predetermined transition period set based on the switching timing detected by the detection unit A second acquisition unit that acquires an image from the other imaging unit;
An interpolation unit that generates an interpolation image based on the image output by the switching unit and the image obtained by the second acquisition unit;
In the transition period, in place of the image output by the switching unit, an output control unit that outputs two-dimensional moving image data including an interpolation image obtained by the interpolation unit is provided. . The moving image photographing device according to claim 2 ,
The interpolation unit
The synthesis ratio of the image output by the switching unit and the image acquired by the second acquisition unit so as to approach the image obtained from the viewpoint of the other imaging unit as it approaches the end from the start of the transition period. A moving image photographing device characterized by adjusting. When acquiring two-dimensional moving image data by alternately switching and operating two imaging units that capture a common subject from different viewpoints,
Prior to the switching, a timing at which the parallax between images obtained by the two imaging units is smaller than a predetermined threshold is detected as a switching timing,
Based on the detected switching timing, control the switching of the operations of the two imaging units ,
In detecting the switching timing,
By operating the other imaging unit at a predetermined time interval in a predetermined search period provided before the operation duration time of the imaging unit that is operated by the switching reaches a predetermined time limit , Obtaining an image by the other imaging unit,
Evaluate the parallax between the acquired image and the image obtained by the imaging unit that is operated by the switching,
A moving image capturing method characterized in that , based on an evaluation result, a timing at which a parallax between images obtained by the two imaging units becomes smaller than a predetermined threshold is specified as the switching timing . When acquiring two-dimensional moving image data by alternately switching and operating two imaging units that capture a common subject from different viewpoints,
  Prior to the switching, a timing at which the parallax between images obtained by the two imaging units is smaller than a predetermined threshold is detected as a switching timing,
  Based on the detected switching timing, control the switching of the operations of the two imaging units,
  In obtaining the two-dimensional moving image data,
  In the predetermined transition period set based on the detected switching timing, the other imaging unit is operated by operating the other imaging unit in synchronization with the imaging unit operated by the switching. Part of the image,
  Generating an interpolated image based on the acquired image and the image obtained by the imaging unit operated by the switching,
  In the transition period, two-dimensional moving image data including an interpolation image to be generated is output instead of an image obtained by the imaging unit that is operated by the switching.
  A moving image photographing method characterized by the above.

Images