JP2023177556A - Imaging apparatus and control method for the same - Google Patents

Abstract

To make it possible to achieve a stable frame rate on top of synchronizing a plurality of imaging units or synchronizing the imaging units and a sensing unit.SOLUTION: An imaging apparatus has a first imaging unit that takes an image by exposure during a first exposure period, a second imaging unit that takes an image by exposure during a second exposure period, a first timestamp generation unit for generating a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit, and a second timestamp generation unit for generating a second timestamp indicating an exposure timing within the second exposure period of the second imaging unit.SELECTED DRAWING: Figure 5

Description

The present disclosure relates to an imaging device and a method of controlling the imaging device.

In recent years, mixed reality, or so-called MR (Mixed Reality) technology, has become known as a technology that seamlessly fuses the real world and the virtual world in real time. BACKGROUND ART As one of the MR techniques, a technique is known in which a video see-through HMD (Head Mounted Display) is used to image an object that substantially matches the object observed from the pupil position of the HMD user using a video camera or the like. With this technology, an HMD user can observe an image in which CG (computer graphics) is superimposed on the captured image.

In such technology, the position and orientation of the HMD is determined by performing arithmetic processing using a plurality of captured images and various sensor information, thereby obtaining position information of the space experienced by the HMD user. If the position information of the HMD user can be accurately acquired, CG can be drawn in an appropriate space, and the HMD user can be immersed in the MR experience.

In order to accurately obtain positional information, it is desirable to operate the plurality of imaging means and various sensors in a synchronized state as much as possible. On the other hand, Patent Document 1 describes a technique for synchronizing a plurality of imaging means by aligning the centers of gravity of their exposure times. Further, Patent Document 2 describes a technique for synchronizing a plurality of imaging means and various sensors.

Japanese Patent Application Publication No. 2006-5608 JP 2021-117057 Publication

However, although the technologies described in Patent Documents 1 and 2 can synchronize multiple captured images and various sensors on a frame-by-frame basis, the frame rate fluctuates and is not constant due to synchronization. This results in an MR image that feels strange to the user.

An object of the present disclosure is to enable a stable frame rate to be achieved by synchronizing a plurality of imaging units or an imaging unit and a sensing unit.

The imaging device includes a first imaging unit that performs imaging by exposure in a first exposure period, a second imaging unit that performs imaging by exposure in a second exposure period, and the first imaging unit of the first imaging unit. a first timestamp generating section that generates a first timestamp indicating an exposure timing within the exposure period of the second imaging section; and a second timestamp indicating the exposure timing within the second exposure period of the second imaging section. and a second timestamp generation unit that generates.

According to the present disclosure, a stable frame rate can be achieved by synchronizing a plurality of imaging units or an imaging unit and a sensing unit.

1 is a diagram showing a configuration example of an image display system. It is a diagram showing an example of functional configuration of an HMD and an image processing device. 5 is a timing chart showing processing of an imaging unit. It is a flowchart which shows the control method of HMD. It is a timing chart which shows the control method of HMD. It is a diagram showing an example of functional configuration of an HMD and an image processing device. It is a timing chart which shows the control method of HMD.

Below, preferred embodiments will be described in detail based on the drawings.
(First embodiment)
FIG. 1 is a diagram showing a configuration example of an image display system 100 according to the first embodiment. The image display system 100 includes a head mounted display (hereinafter referred to as HMD) 101 , a display device 102 , an operation unit 104 , and an image processing device 103 . A display device 102 and an operation unit 104 are connected to the image processing device 103 . The image processing device 103 inputs operation information of the user through the operation unit 104 and displays images and information on the display device 102 .

The HMD 101 is worn on the user's head and displays images generated by the image processing device 103. The HMD 101 communicates with an image processing device 103 that is wirelessly connected via a small-scale network. The small-scale network is, for example, WLAN (Wireless Local Area Network) or WPAN (Wireless Personal Area Network). Communication between the HMD 101 and the image processing device 103 is not limited to wireless communication, and a wired communication method may be used. Further, although the image processing device 103 and the HMD 101 have separate hardware configurations, it is also possible to implement all the functions of the image processing device 103 within the HMD 101 and integrate them.

FIG. 2 is a block diagram showing an example of the functional configuration of the HMD 101 and image processing device 103 shown in FIG. 1. The HMD 101 includes imaging units 201 and 202, a display unit 203, a sensing unit 204, a motion calculation unit 205, an imaging processing unit 206, a calculation unit 207, an image synthesis unit 208, a synchronization unit 220, and a timer. It has a stamp management section 221.

The imaging units 201 and 202 each have an objective optical system and an image sensor for imaging the outside world. The display unit 203 includes an eyepiece optical system and a display for presenting images to the user of the HMD 101. The sensing unit 204 senses the operation of the HMD 101. The motion calculation unit 205 calculates the motion of the HMD 101 from the sensing result of the sensing unit 204. The imaging processing unit 206 performs image processing on the captured images from the imaging units 201 and 202. The calculation unit 207 performs calculations. The image composition unit 208 composes the captured image and the CG image. The synchronization unit 220 synchronizes the imaging units 201 and 202 and the sensing unit 204. The time stamp management unit 221 manages time stamps.

The image processing device 103 includes a position/orientation calculation section 209, a CG generation section 210, and a content DB (database) 211. The position and orientation calculation unit 209 calculates the position and orientation of the HMD 101 from the captured image and the movement of the HMD 101. The content DB 211 stores CG data. The CG generation unit 210 reads CG data from the content DB 211 based on the position and orientation information calculated by the position and orientation calculation unit 209, and then generates a CG image.

Next, a method for a user to experience MR images will be described using FIG. 2. Imaging units 201 and 202 each image the outside world. The imaging units 201 and 202 may be configured to use a rolling shutter type image sensor or a global shutter type image sensor depending on the purpose, taking into account various factors such as the number of pixels, image quality, noise, sensor size, power consumption, and cost. They can be used in combination. For example, when capturing an image to be combined with an image in the virtual space, the imaging unit 201 uses a rolling shutter type imaging element that can obtain a higher quality image. When capturing an image of the outside world in order to calculate the position and orientation of the HMD 101, the imaging unit 202 uses a global shutter type imaging element without image drift. Image drift is a phenomenon that occurs due to the operating principle of a rolling shutter method in which exposure processing is started sequentially for each line in the scanning direction.

FIG. 3 is a timing chart showing a processing example of the rolling shutter type imaging unit 201 and the global shutter type imaging unit 202. The horizontal axis in FIG. 3 indicates time. An example will be described in which the imaging unit 201 uses a rolling shutter type imaging element and the imaging unit 202 uses a rolling shutter type imaging element.

Exposure (rolling) of the imaging unit 201 indicates the exposure time of each line imaged by a rolling shutter type imaging element. In the rolling shutter type imaging unit 201, because there is a time lag in the exposure timing of each line, when the imaging unit 201 or the subject moves during the exposure time, the subject may be deformed and recorded in a flowing manner. A phenomenon occurs. This phenomenon is image drift.

Exposure (global) of the imaging unit 202 indicates the exposure time of each line imaged by a global shutter type imaging device. In the global shutter type imaging unit 202, exposure processing for all lines is performed at the same time, so there is no time lag in the exposure timing of each line, and image blur does not occur. In this embodiment, the imaging unit 201 uses a rolling shutter type imaging device, and the imaging unit 202 uses a global shutter type imaging device.

In FIG. 2, imaging units 201 and 202 output captured images of the outside world to a time stamp management unit 221. The time stamp management unit 221 issues a time stamp when a captured image is input. The detailed operation regarding this will be described later.

The imaging processing unit 206 performs image processing on the captured images captured by the imaging units 201 and 202. The image processing performed by the imaging processing unit 206 includes AE (Auto Exposure) control, demosaic processing, shading correction, noise reduction, distortion correction, and the like. The imaging processing unit 206 performs image processing on the captured images captured by the imaging units 201 and 202 to improve the quality of the images. Thereafter, the imaging processing unit 206 outputs an image obtained by processing the captured image captured by the imaging unit 202 to the position and orientation calculation unit 209 in order to specify the CG drawing position after the image processing. Further, the imaging processing unit 206 outputs an image obtained by processing the captured image captured by the imaging unit 201 to the image combining unit 208 in order to combine it with a CG image.

The sensing unit 204 senses the operation of the HMD 101. The sensor of the sensing unit 204 is an IMU (Inertial Measurement Unit), an acceleration sensor, an angular velocity sensor, or the like. In this embodiment, the sensing unit 204 is, for example, an IMU. The motion calculation unit 205 calculates the movement, tilt, rotation, etc. of the HMD 101 from the sensing data output from the IMU, and outputs them to the position and orientation calculation unit 209.

The position and orientation calculation unit 209 calculates the relationship between the world coordinate system representing the real world and the camera coordinate system representing the image observed through the HMD 101 from the captured image of the imaging unit 202 after image processing and the calculation result of the motion calculation unit 205. do. Then, the position/orientation calculation unit 209 calculates the position/orientation of the HMD 101 with respect to the real world based on this relationship. To calculate the position and orientation, for example, a marker or the like is placed as a reference in the real world, an image is acquired by the imaging unit 202 having a stereo camera configuration, and the position and orientation of the HMD 101 is calculated from the positional relationship of the markers in the image. There is a way to calculate it. Alternatively, there is a method of calculating the feature amount of a stationary object in the real world from an image, and estimating the self-position of the HMD 101 and creating an environment map based on the temporal change in the feature amount. In this embodiment, the method of calculating the position and orientation is not particularly limited. The position and orientation calculation unit 209 outputs information on the position and orientation calculated in this manner to the CG generation unit 210.

The CG generation unit 210 calculates in which position and in which direction of the captured image the CG is to be superimposed, based on the position and orientation information from the position and orientation calculation unit 209 . Further, the CG generation unit 210 performs CG rendering by reading CG data from the content DB 211. Thereby, the CG generation unit 210 generates a CG image that matches the direction in which the captured image captured by the imaging unit 202 is facing, and outputs the CG image to the image composition unit 208.

The image synthesis unit 208 generates an MR image by synthesizing the image captured by the imaging unit 201 after image processing and the CG image obtained from the CG generation unit 210, and presents (displays) the MR image on the display unit 203. . By displaying the MR image on the display unit 203, the user of the HMD 101 can visually recognize objects that do not exist in the outside world as CG and have an MR experience.

At this time, as described above, the CG image generated by the CG generation unit 210 is generated based on the image captured by the image capture unit 202 after image synthesis. On the other hand, the captured image synthesized by the image synthesizing unit 208 is the captured image of the imaging unit 201 after image processing. That is, if the temporal timings at which images are captured by the imaging unit 201 and the imaging unit 202 are different, an MR image that looks strange will be generated when the image synthesis unit 208 performs image synthesis.

Therefore, a method of aligning the exposure centers of gravity of the plurality of imaging units 201 and 202 will be described using the timing chart of FIG. 3. If the exposure centers of gravity of the imaging units 201 and 202 can be made to coincide with each other, there will be no difference in the temporal timing when images are captured by the imaging units 201 and 202.

FIG. 3 shows an operation in which the external synchronization input from the synchronization unit 220 to the two imaging units 201 and 202 is changed so that the exposure centers of gravity of the imaging units 201 and 202 are made to coincide. FIG. 3 shows a timing chart of imaging exposure of the imaging unit 201 and a timing chart of imaging exposure of the imaging unit 202.

When an external synchronization input is received from the synchronization unit 220, the imaging unit 201 starts exposure after an exposure start waiting time A1_f0 that changes according to the set shutter speed (for example, 1/120 s in the case of Frame 0). do. At this time, since the imaging unit 201 uses a rolling shutter type imaging device, it sequentially starts exposure for each line. Then, when the imaging unit 201 has been exposed for the set shutter speed time, it sequentially ends the exposure for each line. Then, when the exposure is completed, the imaging unit 201 outputs the image data in raster order. When outputting image data, the imaging unit 201 also outputs synchronization signals, such as Vsync indicating the beginning of a frame and Hsync indicating the beginning of a line, in addition to the image data. When inputting image data, the imaging processing unit 206 acquires it as image data based on synchronization signals such as Vsync and Hsync.

Similar to the imaging unit 201, when an external synchronization input is performed from the synchronization unit 220, the imaging unit 202 sets an exposure start waiting time that changes depending on the set shutter speed (for example, 1/120 s in the case of Frame 0). Exposure is started after passing through A2_f0. At this time, since the imaging unit 202 uses a global shutter type imaging element, it starts exposing all lines at the same time. When the imaging unit 202 has been exposed for the set shutter speed time, it ends the exposure for all lines at the same time. Then, when the exposure is completed, the imaging unit 202 outputs the image data in raster order.

As described above, the imaging unit 201 uses a rolling shutter type imaging element, and the imaging unit 202 uses a global shutter type imaging element. Therefore, in order to match the exposure centers of gravity of the imaging units 201 and 202, it is necessary to relatively shift the exposure start times of the imaging units 201 and 202, taking into consideration the shutter speed settings of the imaging units 201 and 202. be. Therefore, in FIG. 3, by changing the timing of the external synchronization input to the imaging unit 202 and the external synchronization input to the imaging unit 201, the above-mentioned coincidence of the exposure center of gravity is achieved. In this way, the synchronization unit 220 changes the external synchronization input timing for each of the imaging units 201 and 202 so that the exposure centers of gravity in the plurality of imaging units 201 and 202 coincide.

However, according to the above method, the external synchronization input timing changes depending on the shutter speed of the imaging units 201 and 202. Since the time from external synchronization input to image data output is constant, if the external synchronization input timing is not constant, the image data output time (frame rate) between frames will vary. When the frame rate of the captured images output from the imaging units 201 and 202 changes, the frame rate of updating the image displayed on the display unit 203 also changes, making it difficult for the user to comfortably experience the MR experience. There is a problem that cannot be done.

Therefore, in the present embodiment, the HMD 101 uses the synchronization unit 220 and the time stamp management unit 221 to share external synchronization input between the plurality of imaging units 201 and 202. Then, the HMD 101 attempts to solve the above problem by associating the shift of the exposure center of gravity of the imaging units 201 and 202 with the image data as time stamp information.

Next, the synchronization unit 220 of this embodiment will be explained. As shown in FIG. 2, the synchronization unit 220 generates an external synchronization signal to synchronize the imaging unit 201 and the imaging unit 202. When the synchronization unit 220 receives the vertical synchronization signal Vsync from the image synthesis unit 208, it generates an external synchronization signal after a certain offset time has elapsed. This is because by setting the imaging rates of the imaging units 201 and 202 and the update rate of the display unit 203 to have the same cycle, it is possible to provide MR images smoothly with low delay. However, the method of generating the external synchronization signal is not particularly limited to this. The synchronization unit 220 automatically generates a timing signal internally and outputs the timing signal as an external synchronization signal, or generates an external synchronization signal using a vertical synchronization signal Vsync of a specific imaging unit 201 or 202 as a trigger. You may also use a method such as outputting the

Next, the time stamp management unit 221 of this embodiment will be explained using FIGS. 4 and 5. FIG. 4 is a flowchart illustrating an example of processing by the timestamp management unit 221. The time stamp management unit 221 has an internal time counter Tcnt.

In step S401, the time stamp management unit 221 starts the operation of the time counter Tcnt. The time counter Tcnt starts counting the reference time without being affected by other operations. Note that the time stamp management unit 221 may use the value of an internal counter as the value of the time counter Tcnt, or may generate the value of the time counter Tcnt based on a reference signal input from an external device.

In step S402, the time stamp management unit 221 determines whether the external synchronization signal output by the synchronization unit 220 has been input. The time stamp management unit 221 returns to step S402 when the external synchronization signal output from the synchronization unit 220 is not input, and returns to step S403 when the external synchronization signal output from the synchronization unit 220 is input. Proceed to.

In step S403, the time stamp management unit 221 holds the value of the time counter Tcnt at the time when the external synchronization signal is input from the synchronization unit 220 as the external synchronization time T(n). The synchronization unit 220 also outputs this external synchronization signal to the imaging units 201 and 202. The imaging units 201 and 202 enter an exposure preparation state from the same time as the external synchronization time T(n).

In step S404, the time stamp management unit 221 obtains the exposure time parameters set in the imaging units 201 and 202. The exposure time parameters include shutter speed values, exposure mode settings, etc. that are set in registers for the imaging units 201 and 202. This allows the time stamp management unit 221 to acquire information regarding the exposure time in each of the imaging units 201 and 202.

In step S405, the time stamp management unit 221 specifies an offset value Toffset1 of the exposure center of gravity of the imaging unit 201 and an offset value Toffset2 of the exposure center of gravity of the imaging unit 202 from the exposure time parameters of the imaging units 201 and 202 described above. The offset values Toffset1 and Toffset2 are values indicating the time from the external synchronization time T(n) to the center of gravity of exposure of the imaging units 201 and 202, respectively, and vary depending on the characteristics of the imaging units 201 and 202 and the set shutter speed. This is a uniquely determined value. The offset value Toffset1 can be expressed as, for example, Toffset1(n)=A1_f(n)+B1_f(n). The offset value Toffset2 can be expressed as, for example, Toffset2(n)=A2_f(n)+B2_f(n). A1_f(n) and A2_f(n) are the times from the input of the external synchronization signal to the start of exposure, respectively, like A1_f0 and A2_f0 in FIG. 5, for example. B1_f(n) and B2_f(n) are the times from the start of exposure to the time when the exposure center of gravity is reached, as shown in B1_f0 and B2_f0 in FIG. 5, respectively, for example. n indicates a frame number. The times A1_f(n), A2_f(n), B1_f(n), and B2_f(n) are all determined by the characteristics of the imaging unit 201 or 202 and the set shutter speed. Therefore, the times A1_f(n), A2_f(n), B1_f(n), and B2_f(n) may be calculated according to the shutter speed set for each frame by the calculation unit 207, or may be calculated based on the shutter speed candidates to be set. It may also be obtained by creating a table and referring to it. In this manner, the time stamp management unit 221 identifies the offset value Toffset of the exposure center of gravity of the imaging unit 201 and the offset value Toffset2 of the exposure center of gravity of the imaging unit 202 from the exposure time parameters of the imaging units 201 and 202. The identification method is not particularly limited.

In step S406, the timestamp management unit 221 issues the timestamp S1(n) of the imaging unit 201 and the timestamp S2(n) of the imaging unit 202, and holds the timestamps S1(n) and S2(n). . The timestamp management unit 221 calculates the timestamp S1(n) using S1(n)=T(n)+Toffset1(n), and calculates the timestamp S2(n) using S2(n)=T(n)+Toffset2(n). ) is calculated. Time stamps S1(n) and S2(n) indicate the exposure gravity center times of the imaging units 201 and 202, respectively.

In step S407, the time stamp management unit 221 determines whether the vertical synchronization signal Vsync has been input from the imaging unit 201 or 202. If the time stamp management unit 221 has not received the vertical synchronization signal Vsync from the imaging unit 201 or 202, the process returns to step S407, and if the vertical synchronization signal Vsync has been input from the imaging unit 201 or 202, it returns to step S408. Proceed to.

In step S408, when the vertical synchronization signal Vsync is input from the imaging unit 201, the time stamp management unit 221 calculates the timestamp S1(n) of the imaging unit 201 and the image data D1(n) input from the imaging unit 201. Output in association with. Further, when the vertical synchronization signal Vsync is input from the imaging unit 202, the timestamp management unit 221 stores the timestamp S2(n) of the imaging unit 202 and the image data D2(n) input from the imaging unit 202. Output in correspondence. Thereby, the time stamp management unit 221 completes the image processing in units of one frame, and returns to step S402.

Thereafter, the time stamp management unit 221 repeats the same process for the next frame to obtain the image data D1(n) and D2 for each frame and the time stamps S1(n) and S2(n) indicating the exposure center of gravity. and can be output in association with each other.

FIG. 5 is a timing chart showing an example of a method for controlling the HMD 101 according to this embodiment. The time stamp management unit 221 performs the processing shown in the flowchart of FIG. Hereinafter, the explanation of the process of the flowchart explained in FIG. 4 will be supplemented with reference to FIG. 5.

Time E0 is the time when the synchronization unit 220 outputs an external synchronization signal to the imaging units 201 and 202, the sensing unit 204, and the time stamp management unit 221. The imaging unit 201, the imaging unit 202, the sensing unit 204, and the time stamp management unit 221 input a common external synchronization signal. The timestamp management unit 221 holds the value of the time counter Tcnt at time E0 as external synchronization time T0.

At time T0, the shutter speed setting of the imaging unit 201 is 1/120 s, and the imaging unit 201 uses a rolling shutter type imaging element. The time A1_f0 is the time from when the imaging unit 201 inputs the external synchronization signal until it starts exposure. The time B1_f0 is the time from the start of exposure of the imaging unit 201 until reaching the exposure center of gravity. The times A1_f0 and B1_f0 are values uniquely determined by the characteristics of the imaging unit 201 and the set shutter speed. The timestamp management unit 221 calculates the timestamp S1(0) by S1(0)=T0+A1_f0+B1_f0.

Time R0 is the time when the imaging unit 201 outputs the vertical synchronization signal Vsync, and is the time when the imaging unit 201 starts outputting the image data D1(0). At time R0, the time stamp management unit 221 outputs the time stamp S1(0) and the image data D1(0) in association with each other.

Further, at time T0, the shutter speed setting of the imaging unit 202 is 1/120 s, and the imaging unit 202 uses a global shutter type imaging element. Time A2_f0 is the time from when the imaging unit 202 inputs the external synchronization signal until it starts exposure. The time B2_f0 is the time from the start of exposure of the imaging unit 202 to the time when the exposure center of gravity is reached. The times A2_f0 and B2_f0 are values uniquely determined by the characteristics of the imaging unit 202 and the set shutter speed. The time stamp management unit 221 calculates the time stamp S2(0) by S2(0)=T0+A2_f0+B2_f0.

Time R1 is the time when the imaging unit 202 outputs the vertical synchronization signal Vsync, and is the time when the imaging unit 202 starts outputting the image data D2(0). At time R1, the time stamp management unit 221 outputs the time stamp S2(0) and the image data D2(0) in association with each other.

The imaging unit 201 uses a rolling shutter type imaging element. The imaging unit 202 uses a global shutter type imaging element. Therefore, it can be seen that although the shutter speeds of the imaging units 201 and 202 are the same, the times B1_f0 and B2_f0 are different from each other.

Next, the effects of this embodiment will be explained. As described above, the time stamp management unit 221 receives the common external synchronization signal of the imaging units 201 and 202, and generates the time stamps S1(n) and S2(n) according to the set values. The time difference ΔT(n) is the difference between the time stamp S1(n) and the time stamp S2(n), and is the shift time of the exposure center of gravity of the imaging units 201 and 202.

The CG image generated by the CG generation unit 210 is generated based on the image captured by the image capture unit 202 after image processing. On the other hand, the captured image input to the image combining unit 208 is the captured image of the imaging unit 201 after image processing. That is, if the temporal timing of exposure in the imaging unit 201 and the imaging unit 202 is different, when the image combining unit 208 performs image combining, an MR image that looks strange will be generated. In this embodiment, the HMD 101 calculates the time difference ΔT(n) from the time stamps S1(n) and S2(n), and generates a CG image by taking the time difference ΔT(n) into consideration.

In FIG. 5, the timestamp value in Frame 0 of the imaging unit 201 is S1(0). The timestamp value in Frame 0 of the imaging unit 202 is S2(0). The time difference ΔT0 is the time difference between the exposure center of gravity of the imaging section 201 and the exposure center of gravity of the imaging section 202. For example, the imaging processing unit 206 calculates the time difference ΔT0 by ΔT0=|S1(0)−S2(0)|, and outputs the time difference ΔT0 and the captured image of the imaging unit 202 after image processing to the position and orientation calculation unit 209. .

The position/orientation calculation unit 209 considers the deviation of the exposure center of gravity between the imaging unit 201 and the imaging unit 202 based on the time difference ΔT0, the captured image of the imaging unit 202 after image processing, and the calculation result of the motion calculation unit 205. The position and orientation of the HMD 101 is calculated. As described above, the position and orientation calculation unit 209 calculates the relationship between the world coordinate system representing the real world and the camera coordinate system representing the image observed through the HMD 101, and calculates the position and orientation of the HMD 101 with respect to the real world. Calculate. By using the time difference ΔT and the calculation result of the motion calculation unit 205, the position and orientation calculation unit 209 can correct the time error of the time difference ΔT0 in consideration of the movement of the HMD 101. Specifically, the position and orientation calculation unit 209 can know the state of movement, tilt, rotation, acceleration, angular velocity, etc. of the HMD 101 based on the calculation results of the motion calculation unit 205. That is, the position and orientation calculation unit 209 calculates the amount of difference between the camera coordinates represented by the captured image of the imaging unit 201 and the camera coordinates represented by the captured image of the imaging unit 202, based on the change in the HMD 101 in the real world and the time difference ΔT0. Since this is known, the position and orientation of the HMD 101 is calculated taking this into consideration. The position and orientation calculation unit 209 outputs information on the position and orientation of the HMD 101 to the CG generation unit 210.

The CG generation unit 210 reads CG data from the content DB 211 based on the information on the position and orientation of the HMD 101 calculated by the position and orientation calculation unit 209, and adjusts it to the direction in which the captured image captured by the imaging unit 202 is facing. Generate a CG image. Then, the CG generation unit 210 outputs the CG image to the image composition unit 208.

The image synthesizing unit 208 receives an image captured by the imaging unit 201 after image processing from the imaging processing unit 206, and receives a CG image from the CG generation unit 210. The image combining unit 208 generates an MR image by combining the image captured by the imaging unit 201 after image processing and the CG image, and displays the MR image on the display unit 203. By displaying the MR image on the display unit 203, the user of the HMD 101 can visually recognize objects that do not exist in the outside world as CG and have an MR experience.

As described above, the position and orientation calculation unit 209 calculates the position and orientation of the HMD 101 using the time difference ΔT0 and the calculation result of the motion calculation unit 205. Thereby, the position and orientation calculation unit 209 can calculate the position and orientation of the HMD 101 after correcting the time error of the time difference ΔT0 in consideration of the movement of the HMD 101. In this way, even if the exposure centers of gravity of the imaging units 201 and 202 do not match, the image combining unit 208 combines the image captured by the imaging unit 201 after image processing and the CG image at the correct position. be able to.

The HMD 101 can accurately determine the time difference ΔT0 between the exposure centers of the imaging units 201 and 202 based on the characteristics of the plurality of imaging units 201 and 202 and the set shutter speed value. Thereby, the HMD 101 can achieve a stable frame rate by synchronizing the plurality of imaging units 201 and 202 and the sensing unit 204 using the synchronization unit 220.

As described above, the imaging device includes the HMD 101 and the image processing device 103. HMD 101 is a head-mounted device that has a display section 203.

The imaging unit 201 performs imaging by exposure during the first exposure period. The first exposure period is, for example, B1_f0+C1_f0 in FIG. 5. The imaging unit 202 performs imaging by exposure during the second exposure period. The second exposure period is, for example, B2_f0+C2_f0 in FIG. 5. The time stamp management unit 221 is a time stamp generation unit, and generates a first time stamp indicating the exposure timing of the imaging unit 201 within the first exposure period. The first timestamp is, for example, S1(0) in FIG. Furthermore, the time stamp management unit 221 generates a second time stamp indicating the exposure timing of the imaging unit 202 within the second exposure period. The second timestamp is, for example, S2(0) in FIG.

Specifically, the time stamp management unit 221 generates a first time stamp indicating the center of gravity or the center of the first exposure period of the imaging unit 201. Furthermore, the time stamp management unit 221 generates a second time stamp indicating the center of gravity or the center of the exposure period of the imaging unit 202.

The imaging unit 201 captures images using the first shutter method. The imaging unit 202 captures images using a second shutter method that is different from the first shutter method. The first shutter method is, for example, a rolling shutter method. The second shutter method is, for example, a global shutter method.

In frame 1 of FIG. 5, the first exposure period of the imaging unit 201 is, for example, B1_f1+C1_f1. The second exposure period of the imaging unit 202 is, for example, B2_f1+C2_f1. In this case, the length of the first exposure period and the length of the second exposure period are different.

The sensing unit 204 acquires sensor data. For example, the sensing unit 204 acquires sensor data indicating the movement of the HMD 101.

The image synthesis unit 208, the position and orientation calculation unit 209, and the CG generation unit 210 are image processing units, and process an image captured by the imaging unit 201, an image captured by the imaging unit 202, a first time stamp, Image processing is performed based on the second time stamp.

The position and orientation calculation unit 209 and the CG generation unit 210 generate a computer graphics image based on the image captured by the imaging unit 202, the first time stamp, the second time stamp, and sensor data. . The image synthesis unit 208 performs image synthesis based on the image captured by the imaging unit 201 and the computer graphics image described above.

Note that the imaging unit 201 may perform exposure in the first exposure period based on the vertical synchronization signal Vsync. In that case, the synchronization unit 220 generates an external synchronization signal based on the vertical synchronization signal Vsync of the imaging unit 201. The imaging unit 202 performs exposure in the second exposure period based on the external synchronization signal generated by the synchronization unit 220.

As described above, according to the present embodiment, the HMD 101 and the image processing device 103 can synchronize the imaging unit 201, the imaging unit 202, and the sensing unit 204, and then realize a stable frame rate.

(Second embodiment)
The second embodiment will be described below with reference to FIG. FIG. 6 is a block diagram showing an example of the functional configuration of the HMD 101 and the image processing device 103 according to the second embodiment. The HMD 101 in FIG. 6 is the HMD 101 in FIG. 2 with an image modification unit 601 added thereto. The image processing device 103 in FIG. 6 is similar to the image processing device 103 in FIG. Hereinafter, the differences between the second embodiment and the first embodiment will be explained.

The throughput of the CG generation unit 210 may vary depending on the drawing load status of CG data read from the content DB, and the latency may increase. In this case, since the timing at which the image synthesis unit 208 receives the CG image from the CG generation unit 210 is delayed, the position of the CG image may not match the position of the captured image, resulting in an unnatural MR image.

Therefore, the image modification section 601 transforms the CG image generated by the CG generation section 210 based on the calculation result of the motion calculation section 205. Since the image captured by the imaging unit 201 changes in accordance with the operation of the HMD 101, the image transformation unit 601 transforms the CG image so as to correct the delay of the CG image. The image transformation method is, for example, an image transformation method such as homography transformation. The image transformation unit 601 outputs the transformed CG image to the image composition unit 208.

The image synthesizing unit 208 inputs the image captured by the image capturing unit 201 after image processing from the image capturing processing unit 206, and inputs the transformed CG image from the image transforming unit 601. The image combining unit 208 generates an MR image by combining the image taken by the imaging unit 201 after image processing and the deformed CG image, and presents (displays) the MR image on the display unit 203.

Thereby, the HMD 101 can display on the display unit 203 an MR image that is less uncomfortable for the user of the HMD 101, even if the latency increases due to CG image generation in the CG generation unit 210.

The motion calculation unit 205 calculates the movement, tilt, rotation, etc. of the HMD 101 based on the sensor data output from the sensing unit 204. The image modification unit 601 transforms the CG image based on the calculation result of the motion calculation unit 205. The image synthesis unit 208 synthesizes the image captured by the image capture unit 201 after image processing and the transformed CG image. Therefore, the HMD 101 needs to accurately grasp the data acquisition timing of the imaging units 201 and 202 and the sensing unit 204. Therefore, the time stamp management unit 221 outputs the sensor data output by the sensing unit 204 in association with a time stamp. The details will be explained below.

FIG. 7 is a timing chart showing an example of a method for controlling the HMD 101 according to the second embodiment. The operation in the second embodiment will be explained using the timing chart in FIG. The timing chart of the imaging units 201 and 202 in FIG. 7 is the same as the timing chart of the imaging units 201 and 202 in FIG. 5. The timing chart of the sensing unit 204 in FIG. 7 will be mainly described below.

Time E0 is the time when the synchronization unit 220 outputs an external synchronization signal to the imaging units 201 and 202, the sensing unit 204, and the time stamp management unit 221. The imaging unit 201, the imaging unit 202, the sensing unit 204, and the time stamp management unit 221 input a common external synchronization signal. The timestamp management unit 221 holds the value of the time counter Tcnt at time E0 as external synchronization time T0.

The sensing unit 204 samples sensor data at sampling timing based on the settings. The time from time E0 to the sampling timing is a value uniquely determined by the settings of the sensing unit 204, and is A3_f0+B3_f0. The time stamp management unit 221 calculates the time stamp S3(0) by S3(0)=T0+A3_f0+B3_f0. After a time C3_f0 from the sampling timing, the sensing unit 204 starts outputting the sampled sensor data D3(0). The timestamp management unit 221 associates the timestamp S3(0) and the sensor data D3(0) and outputs them to the motion calculation unit 205.

The operation calculation unit 205 calculates the sensor data acquisition timing based on the time stamp S1(0) of the imaging unit 201, the time stamp S2(0) of the imaging unit 202, and the time stamp S3(0) of the sensing unit 204. The deviation amounts ΔT0_13 and ΔT0_23 are calculated.

For example, the operation calculation unit 205 calculates the deviation amount ΔT0_13 between the exposure center of gravity of the imaging unit 201 and the data acquisition timing of the sensing unit 204 using ΔT0_13=|S1(0)−S3(0)|. Further, the operation calculation unit 205 calculates the amount of deviation ΔT0_23 between the exposure center of gravity of the imaging unit 202 and the data acquisition timing of the sensing unit 204 using ΔT0_23=|S2(0)−S3(0)|. The motion calculation unit 205 outputs the deviation amounts ΔT0_13 and ΔT0_23 to the image modification unit 601.

The image modification unit 601 transforms the CG image generated by the CG generation unit 210 based on the deviation amounts ΔT0_13 and ΔT0_23 so as to correct the delay of the CG image with respect to the captured image.

The image synthesizing unit 208 inputs the image captured by the image capturing unit 201 after image processing from the image capturing processing unit 206, and inputs the transformed CG image from the image transforming unit 601. The image combining unit 208 generates an MR image by combining the image taken by the imaging unit 201 after image processing and the transformed CG image, and displays the MR image on the display unit 203. Thereby, the display unit 203 can present (display) an image that is less uncomfortable for the user of the HMD 101.

As described above, the sensing unit 204 acquires sensor data by sampling at the sampling time. The time stamp management unit 221 generates a third time stamp indicating the sampling time of the sensing unit 204. The third timestamp is, for example, S3(0) in FIG.

The position and orientation calculation unit 209 and the CG generation unit 210 generate a computer graphics image based on the image captured by the imaging unit 202, the first time stamp, the second time stamp, and sensor data. . The first timestamp is, for example, S1(0) in FIG. The second timestamp is, for example, S2(0) in FIG.

The image modification unit 601 transforms the computer graphics image based on the first time stamp, the second time stamp, and the third time stamp. The image synthesis unit 208 performs image synthesis based on the image captured by the imaging unit 201 and the computer graphics image transformed by the image transformation unit 601.

As described above, according to the present embodiment, the HMD 101 and the image processing device 103 can synchronize the imaging unit 201, the imaging unit 202, and the sensing unit 204, and then realize a stable frame rate.

(Other embodiments)
The present disclosure provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although the embodiments have been described above, the present disclosure is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist.

The disclosure of this embodiment includes the following configuration and method.
(Configuration 1)
a first imaging unit that performs imaging by exposure during a first exposure period;
a second imaging unit that performs imaging by exposure in a second exposure period;
a first time stamp generation unit that generates a first time stamp indicating an exposure timing within the first exposure period of the first imaging unit;
An imaging device comprising: a second time stamp generation section that generates a second time stamp indicating exposure timing within the second exposure period of the second imaging section.
(Configuration 2)
The first time stamp generation unit generates a first time stamp indicating a center of gravity or a center of the first exposure period of the first imaging unit,
The imaging device according to configuration 1, wherein the second time stamp generation unit generates a second time stamp indicating a center of gravity or a center of the second exposure period of the second imaging unit.
(Configuration 3)
The first imaging unit captures an image using a first shutter method,
The imaging device according to configuration 1 or 2, wherein the second imaging unit takes an image using a second shutter method different from the first shutter method.
(Configuration 4)
The first shutter method is a rolling shutter method,
The imaging device according to configuration 3, wherein the second shutter method is a global shutter method.
(Configuration 5)
5. The imaging device according to any one of configurations 1 to 4, wherein the length of the first exposure period and the length of the second exposure period are different.
(Configuration 6)
The imaging device according to any one of configurations 1 to 5, further comprising a sensing unit that acquires sensor data.
(Configuration 7)
The imaging device according to configuration 6, wherein the sensing unit acquires sensor data indicating movement of the imaging device.
(Configuration 8)
An image on which image processing is performed based on an image captured by the first imaging unit, an image captured by the second imaging unit, the first time stamp, and the second time stamp. 8. The imaging device according to any one of configurations 1 to 7, further comprising a processing section.
(Configuration 9)
a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first time stamp, the second time stamp, and the sensor data;
8. The imaging device according to configuration 6 or 7, further comprising an image synthesis section that performs image synthesis based on the image captured by the first imaging section and the computer graphics image.
(Configuration 10)
The sensing unit acquires sensor data by sampling at a sampling time,
8. The imaging device according to configuration 6 or 7, further comprising a third time stamp generation section that generates a third time stamp indicating the sampling time of the sensing section.
(Configuration 11)
a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first time stamp, the second time stamp, and the sensor data;
an image transformation unit that transforms the computer graphics image based on the first time stamp, the second time stamp, and the third time stamp;
The imaging device according to configuration 10, further comprising an image synthesis unit that performs image synthesis based on the image captured by the first imaging unit and the computer graphics image transformed by the image transformation unit. .
(Configuration 12)
The first imaging unit performs exposure in the first exposure period based on a first synchronization signal,
The second imaging unit performs exposure in the second exposure period based on a second synchronization signal,
The imaging device according to any one of configurations 1 to 11, wherein the imaging device further includes a synchronization unit that generates the second synchronization signal based on the first synchronization signal.
(Configuration 13)
The imaging device according to any one of configurations 1 to 12, wherein the imaging device is a head-mounted device having a display section.
(Configuration 14)
an imaging unit that captures an image by exposure during an exposure period;
a sensing unit that acquires sensor data by sampling at a sampling time;
a first time stamp generation unit that generates a first time stamp indicating an exposure timing within the exposure period of the imaging unit;
An imaging device comprising: a second time stamp generation section that generates a second time stamp indicating the sampling time of the sensing section.
(Configuration 15)
15. The imaging device according to configuration 14, wherein the first time stamp generation unit generates a first time stamp indicating a center of gravity or a center of the exposure period of the imaging unit.
(Configuration 16)
16. The imaging device according to configuration 14 or 15, wherein the sensing unit acquires sensor data indicating movement of the imaging device.
(Configuration 17)
17. The imaging device according to any one of configurations 14 to 16, wherein the imaging device is a head-mounted device having a display section.
(Method 1)
a first imaging unit that performs imaging by exposure during a first exposure period;
A method for controlling an imaging device including a second imaging unit that performs imaging by exposure in a second exposure period,
a first timestamp generation step of generating a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit;
A method for controlling an imaging device, comprising: a second timestamp generation step of generating a second timestamp indicating an exposure timing of the second imaging unit within the second exposure period.
(Method 2)
an imaging unit that captures an image by exposure during an exposure period;
A method for controlling an imaging device including a sensing unit that acquires sensor data by sampling at a sampling time, the method comprising:
a first time stamp generation step of generating a first time stamp indicating an exposure timing of the imaging unit within the exposure period;
a second time stamp generation step of generating a second time stamp indicating the sampling time of the sensing unit.

Reference Signs List 101 HMD, 102 display device, 103 image processing device, 104 operation unit, 201 imaging unit, 202 imaging unit, 203 display unit, 204 sensing unit, 205 motion calculation unit, 206 imaging processing unit, 207 calculation unit, 208 image composition unit , 209 position and orientation calculation unit, 210 CG generation unit, 211 content DB, 220 synchronization unit, 221 time stamp management unit, 601 image transformation unit

Claims (19)

a first imaging unit that performs imaging by exposure during a first exposure period;
a second imaging unit that performs imaging by exposure in a second exposure period;
a first time stamp generation unit that generates a first time stamp indicating an exposure timing within the first exposure period of the first imaging unit;
An imaging device comprising: a second time stamp generation section that generates a second time stamp indicating exposure timing within the second exposure period of the second imaging section. The first time stamp generation unit generates a first time stamp indicating a center of gravity or a center of the first exposure period of the first imaging unit,
The imaging device according to claim 1, wherein the second time stamp generation unit generates a second time stamp indicating a center of gravity or a center of the second exposure period of the second imaging unit. . The first imaging unit captures an image using a first shutter method,
The imaging device according to claim 1, wherein the second imaging unit takes an image using a second shutter method different from the first shutter method. The first shutter method is a rolling shutter method,
The imaging device according to claim 3, wherein the second shutter method is a global shutter method. The imaging device according to claim 1, wherein the length of the first exposure period and the length of the second exposure period are different. The imaging device according to claim 1, further comprising a sensing unit that acquires sensor data. The imaging device according to claim 6, wherein the sensing unit acquires sensor data indicating movement of the imaging device. An image on which image processing is performed based on an image captured by the first imaging unit, an image captured by the second imaging unit, the first time stamp, and the second time stamp. The imaging device according to claim 1, further comprising a processing section. a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first time stamp, the second time stamp, and the sensor data;
7. The imaging device according to claim 6, further comprising an image synthesis section that performs image synthesis based on the image captured by the first imaging section and the computer graphics image. The sensing unit acquires sensor data by sampling at a sampling time,
The imaging device according to claim 6, further comprising a third time stamp generation section that generates a third time stamp indicating the sampling time of the sensing section. a generation unit that generates a computer graphics image based on the image captured by the second imaging unit, the first time stamp, the second time stamp, and the sensor data;
an image transformation unit that transforms the computer graphics image based on the first time stamp, the second time stamp, and the third time stamp;
The imaging device according to claim 10, further comprising an image synthesis unit that performs image synthesis based on the image captured by the first imaging unit and the computer graphics image transformed by the image transformation unit. Device. The first imaging unit performs exposure in the first exposure period based on a first synchronization signal,
The second imaging unit performs exposure in the second exposure period based on a second synchronization signal,
The imaging device according to claim 1, further comprising a synchronization unit that generates the second synchronization signal based on the first synchronization signal. The imaging device according to claim 1, wherein the imaging device is a head-mounted device having a display section. an imaging unit that captures an image by exposure during an exposure period;
a sensing unit that acquires sensor data by sampling at a sampling time;
a first time stamp generation unit that generates a first time stamp indicating an exposure timing within the exposure period of the imaging unit;
An imaging device comprising: a second time stamp generation section that generates a second time stamp indicating the sampling time of the sensing section. The imaging device according to claim 14, wherein the first time stamp generation unit generates a first time stamp indicating a center of gravity or a center of the exposure period of the imaging unit. The imaging device according to claim 14, wherein the sensing unit acquires sensor data indicating movement of the imaging device. The imaging device according to claim 14, wherein the imaging device is a head-mounted device having a display section. a first imaging unit that performs imaging by exposure during a first exposure period;
A method for controlling an imaging device including a second imaging unit that performs imaging by exposure in a second exposure period,
a first timestamp generation step of generating a first timestamp indicating an exposure timing within the first exposure period of the first imaging unit;
A method for controlling an imaging device, comprising: a second timestamp generation step of generating a second timestamp indicating an exposure timing of the second imaging unit within the second exposure period. an imaging unit that captures an image by exposure during an exposure period;
A method for controlling an imaging device including a sensing unit that acquires sensor data by sampling at a sampling time, the method comprising:
a first time stamp generation step of generating a first time stamp indicating an exposure timing of the imaging unit within the exposure period;
a second time stamp generation step of generating a second time stamp indicating the sampling time of the sensing unit.

Images