JP2021061504A - Imaging apparatus, image display apparatus, image generation apparatus, and image generation program - Google Patents

Abstract

To display an image viewed from an arbitrary direction.SOLUTION: An imaging apparatus comprises: two imaging units which are connected by a connection part and image a subject from directions different from each other, and where the overlapping range of imaging ranges is to be changed; an image generation unit which, on the basis of an image of the overlapping range of a plurality of images obtained by imaging the subject from a plurality of directions by the imaging units, generates an image of the subject viewed from a direction different from the plurality of directions; and a display unit for displaying the image of the subject viewed from the different direction.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image pickup device, an image display device, an image generation device, and an image generation program.

A compound eye camera that acquires parallax images for the right eye and the left eye, respectively, is known (see Patent Document 1). Conventionally, it has not been possible to obtain an image viewed from an arbitrary direction using a parallax image taken by a camera as described above.

Japanese Unexamined Patent Publication No. 2009-188931

The image pickup apparatus according to the first aspect of the present invention has two image pickup units connected by a connecting portion, images of a subject from different directions, and the overlapping range of shooting ranges is changed, and the image pickup unit provides a plurality of directions. An image generation unit that generates an image of the subject viewed from a direction different from the plurality of directions based on the images of the overlapping range of the plurality of captured images obtained by photographing the subject from the above, and from the different directions. It is provided with a display unit that displays an image of the subject as seen.
The image display device according to the second aspect of the present invention includes a distance calculation unit that calculates a distance to the subject based on a plurality of captured images obtained by imaging a subject from a plurality of different directions, and the plurality of images. It includes an image generation unit that generates an image of the subject as viewed from a direction different from the captured image, the distance, and the plurality of directions, and a display control unit that displays the image on the display unit.
The image generation device according to the third aspect of the present invention is an image generation device that generates an image viewed from a set direction based on a plurality of captured images viewed from two or more directions, and is an image generating device of the plurality of captured images. Among each, the distance calculation unit that calculates the distance to the subject in the overlapping areas based on the images of the overlapping areas, and the plurality of captured images, the distance, and the set direction. It includes an image generation unit that generates an image of an object viewed from a set direction.
The image generation program according to the fourth aspect of the present invention is an image generation program for generating an image viewed from a set direction based on a plurality of captured images viewed from two or more directions, and the plurality of images are captured. The process of calculating the distance to the subject in the overlapping regions based on the images of the overlapping regions of each of the images, and the processing based on the plurality of captured images, the distance, and the set direction. Let the computer execute the process of generating an image of the object viewed from the set direction.

FIG. 1A is a schematic view of the image display device according to the present embodiment as viewed from the front, and FIG. 1B is a schematic view of the image display device as viewed from the side. 2 (a) to 2 (c) are schematic views of the two imaging units of FIG. 1 as viewed from above, and are views showing different open states of the two imaging units. 3 (a) to 3 (e) are schematic views showing the orientations of the two imaging units in different open states. It is a figure explaining the structure of an image display device. It is a schematic diagram explaining the storage of the display unit, FIG. 5A is a diagram showing the storage time, and FIG. 5B is a diagram showing the time of use. It is a schematic diagram which shows an image display device and an object. It is a schematic diagram explaining the shooting range by an image display device. FIG. 8 (a) is a diagram showing an overlapping range DUP when the opening angle θ = 360 degrees, and FIG. 8 (b) is a diagram showing an overlapping range DUP when the opening angle θ = 270 degrees, FIG. 8 (c). Is a diagram showing an overlapping range DUP when the opening angle θ = 180 degrees, and FIG. 8D is a diagram showing an overlapping range DUP when the opening angle θ = 90 degrees. It is a figure explaining the display example of the image before and after changing the viewing direction, FIG. 9A is a figure which shows before the change, and FIG. 9B is a figure which shows after the change. It is a figure explaining another display example of the image before and after changing the viewing direction, FIG. 10A is a figure which shows before the change, and FIG. 10B is a figure which shows after the change. It is a figure explaining another display example of the image before and after changing the viewing direction, FIG. 11A is a figure which shows before the change, and FIG. 11B is a figure which shows after the change. It is a figure explaining another display example of the image before and after changing the viewing direction, FIG. 12A is a figure which shows before the change, and FIG. 12B is a figure which shows after the change. It is a flowchart explaining the flow of the image display process which generates and displays the image which changed the viewing direction. FIG. 14 (a) is a diagram for explaining the initial display of the modified example 5, and FIG. 14 (b) is a diagram for explaining the initial display of the modified example 6.

The image display device according to the embodiment of the present invention generates an image viewed from an arbitrary direction based on the parallax image captured and acquired by the compound eye camera, and displays the generated image on the display unit. In the following description, an image display device as an image pickup device for acquiring a parallax image will be described first, and then an image display device for generating and displaying an image viewed from an arbitrary direction based on the parallax image will be described. ..

<Appearance of imaging device>
FIG. 1A is a schematic view of the image pickup apparatus 100 according to the present embodiment as viewed from the front, and FIG. 1B is a schematic view of the image pickup apparatus 100 as viewed from the side. 2 (a) to 2 (c) are schematic views of the two image pickup units of the image pickup apparatus 100 of FIG. 1 as viewed from above, exemplifying different open states of the two image pickup units. In the image pickup apparatus 100, the first image pickup unit 200 and the second image pickup unit 300 are supported by the connection unit 400. The connecting portion 400 has a function similar to that of a so-called hinge, and the first imaging unit 200 and the second imaging unit 300 can rotate around the axis AX. With this configuration, the open state of the second imaging unit 300 with respect to the first imaging unit 200 (opening angle θ around the axis AX) can be adjusted. In this example, the opening angle θ can be adjusted in the range of 360 degrees to 0 degrees.

Both FIGS. 1 (a) and 1 (b) exemplify an open state in which the opening angle θ = 360 degrees. 2 (a) to 2 (c) exemplify the opening states of the opening angles θ = 270 degrees, θ = 180 degrees, and θ = 90 degrees, respectively.

The image pickup apparatus 100 is provided with a rod-shaped grip portion 450. The grip portion 450 is provided along an axis GX parallel to the axis AX of the connection portion 400. The reason why the shaft GX of the grip portion 450 is not provided coaxially with the shaft AX of the connection portion 400 is that the user's hand holding the grip portion 450 is reflected in the first imaging unit 200 and the second imaging unit 300. This is to avoid.

When using the image pickup device 100, for example, the user holds the grip portion 450 of the image pickup device 100 by hand and holds it in front of the body, and directs the connection portion 400 toward the main subject (in the direction of the arrow in FIG. 1A). .. Assuming that the main subject side (arrow direction) is forward when viewed from the connection portion 400, the grip portion 450 is located behind the connection portion 400, so that the user's hand holding the grip portion 450 is the first imaging unit 200. It becomes difficult to capture the image on the second imaging unit 300. At this time, the user keeps the grip portion 450 in the direction perpendicular to the ground (floor surface) and directs the connection portion 400 of the image pickup apparatus 100 toward the main subject.
If it is permissible for the user's hand holding the grip portion 450 to be reflected in the first image pickup unit 200 and the second image pickup unit 300, the axis GX of the grip portion 450 is connected to the axis AX of the connection portion 400. It may be provided coaxially with.

The release button 810 is provided on the surface of the grip portion 450 that faces the connection portion 400, that is, the surface that faces the main subject (in the direction of the arrow) during shooting. The release button 810 can be operated with the index finger. Further, in the grip portion 450, an open / close button 820 is provided on the opposite side (rear side) of the connection portion 400. The open / close button 820 can be operated with the thumb. The operation unit 820 is an operation unit for opening and closing the first imaging unit 200 and the second imaging unit 300 around the axis AX of the connection unit 400. The open / close button 820 is composed of, for example, a rocker type operating member. When one end is pressed, a signal for changing the opening angle θ is output, and when the other end is pressed, the open / close button 820 is described. A signal for changing the opening angle θ in a direction to be reduced is output. The first imaging unit 200 and the second imaging unit 300 may be manually opened and closed (the angle of θ is changed).

The first imaging unit 200 is provided with a fisheye lens 210, and the second imaging unit 300 is provided with a fisheye lens 310. Both the fisheye lens 210 and the fisheye lens 310 have an angle of view of more than 180 degrees. Therefore, each of the first imaging unit 200 and the second imaging unit 300 can image a wider range than the hemisphere. As a result, when the whole celestial sphere is photographed at 360 degrees in the horizontal and vertical directions at an opening angle of θ = 360 degrees (in the first open state), the image acquired by the first imaging unit 200 is used. It is possible to secure a margin for connecting the image and the image acquired by the second imaging unit 300. Further, when the opening angle θ = 180 degrees (the third open state is set), the same direction in a wider range than the hemisphere can be imaged from different positions.

<Various open states>
3 (a) to 3 (e) are schematic views showing directions of imaging by the first imaging unit 200 and the second imaging unit 300 in various open states. The first image pickup unit 200 includes a fisheye lens 210 and an image pickup element 220 arranged on the image plane of the fisheye lens 210. The second image pickup unit 300 includes a fisheye lens 310 and an image pickup device 320 arranged on the image plane of the fisheye lens 310.

FIG. 3A is a diagram showing a first open state, and corresponds to the cases of FIGS. 1A and 1B. The direction of imaging by the first imaging unit 200 and the direction of imaging by the second imaging unit 300 are opposite to each other and differ by 180 degrees. Therefore, the optical axis X200 of the fisheye lens 210 that constitutes the optical system of the first imaging unit 200 and the optical axis X300 of the fisheye lens 310 that constitutes the optical system of the second imaging unit 300 are coaxial. As described above, in the first open state, an image of the whole celestial sphere can be obtained by joining the image acquired by the first imaging unit 200 and the image acquired by the second imaging unit 300. ..

FIG. 3B is a diagram showing a second open state (180 degrees <opening angle θ <360 degrees), and is a direction in which the first image pickup unit 200 takes an image and a direction in which the second image pickup unit 300 takes an image. Is different from θ-180 degrees. For example, when the opening angle θ = 270 degrees, it corresponds to the case of FIG. 2A, and the direction of imaging by the first imaging unit 200 and the direction of imaging by the second imaging unit 300 are 270-180 = 90. Different degrees.

FIG. 3C is a diagram showing a third open state, and corresponds to the case of FIG. 2B. The direction of imaging by the first imaging unit 200 and the direction of imaging by the second imaging unit 300 are parallel. That is, the optical axis X200 of the fisheye lens 210 that constitutes the optical system of the first imaging unit 200 and the optical axis X300 of the fisheye lens 310 that constitutes the optical system of the second imaging unit 300 are parallel and do not intersect.

FIG. 3D is a diagram showing a fourth open state (0 degree <open angle θ <180 degrees), and is a direction of imaging by the first imaging unit 200 and a direction of imaging by the second imaging unit 300. Is different from θ-180 degrees. For example, when the opening angle θ = 90 degrees, it corresponds to the case of FIG. 2C, and the direction of imaging by the first imaging unit 200 and the direction of imaging by the second imaging unit 300 are 90-180 =-. 90 degrees different. When the imaging directions of the first imaging unit 200 and the second imaging unit 300 differ by a negative angle, the optical axes X200 and X300 of the imaging units intersect each other.

FIG. 3E is a diagram showing a fifth open state (opening angle θ = 0 degrees), and the direction of shooting by the first imaging unit 200 and the direction of shooting by the second imaging unit 300 are opposite to each other. To do. That is, the optical axis X200 of the fisheye lens 210 that constitutes the optical system of the first imaging unit 200 and the optical axis X300 of the fisheye lens 310 that constitutes the optical system of the second imaging unit 300 are coaxial.

<Configuration of imaging device>
The configuration of the image pickup apparatus 100 will be described with reference to the block diagram illustrated in FIG. The image pickup apparatus 100 includes a first image pickup unit 200, a second image pickup unit 300, a connection unit 400, a control unit 500, a display control unit 600, a display unit 700, and an operation member 800, and stores the image. The medium 900 is detachably configured.

(First imaging unit 200, second imaging unit 300)
When the opening angle θ of the first imaging unit 200 and the second imaging unit 300 is set to 180 degrees as shown in FIG. 3C, the optical axes X200 and X300 are at a predetermined distance L (baseline). It constitutes a stereo camera arranged with a distance (called a length). The fisheye lenses 210 and 310 are composed of, for example, equidistant projection type fisheye lenses. With this configuration, in the images obtained by the image sensors 220 and 320, the distance from each of the optical axes X200 and X300 represents the direction of the object (subject). Further, the difference in the position on the image of the same object reflected in the images obtained by the image sensors 220 and 320 represents the parallax. Therefore, the distance to the object can be obtained by using the principle of triangulation based on the image signal by the image sensor 220 and the image signal by the image sensor 320. Obtaining distance information from images obtained by the image sensors 220 and 320 of a stereo camera is also referred to as stereo matching.
The light from the object passes through the fisheye lens 210 and is incident on the image sensor 220, and is transmitted through the fisheye lens 310 and is incident on the image sensor 320. The image signals output from the image sensor 220 of the first image sensor 200 and the image sensor 320 of the second image sensor 300 are sent to the control unit 500, respectively.

(Connection part 400)
The connection unit 400 is provided with an angle sensor 410 that detects the open state (opening angle θ around the axis AX) of the first image pickup unit 200 and the second image pickup unit 300, and a drive unit 420 that changes the open state. ing. The detection signal output from the angle sensor 410 is sent to the control unit 500.
The drive unit 420 includes a motor and a deceleration mechanism that transmits the driving force of the motor to the first and second imaging units 200 and 300. Based on the drive signal sent from the control unit 500, the drive unit 420 drives the motor to rotate the first imaging unit 200 and the second imaging unit 300 around the shaft AX (FIG. 1) via the deceleration mechanism. And are rotated at equal angles in opposite directions. As a result, the opening angle θ around the axis AX is controlled.

(Control unit 500)
The control unit 500 is composed of a CPU, a ROM, a RAM, and the like, and controls the operation of each unit of the image pickup apparatus 100 based on a control program. The control unit 500 includes an image pickup control unit 510, a signal processing unit 520, a recording / reading control unit 530, and an open / close control unit 540.

The image pickup control unit 510 causes the first image pickup unit 200 and the second image pickup unit 300 to perform an image pickup operation, respectively, based on the operation signal from the release button 810 of the operation member 800. The signal processing unit 520 performs predetermined signal processing on the image signals output from the image sensor 220 of the first image sensor 200 and the image sensor 320 of the second image sensor 300, respectively. In addition to interpolation processing and gradation correction processing, the signal processing unit 520 can also perform conversion processing for converting a celestial sphere image into a two-dimensional image, for example, an equirectangular projection image.
The imaging control unit 510 can determine whether or not imaging is possible based on the opening angle θ detected by the angle sensor 410. For example, in the first open state, the second open state, the third open state, and the fourth open state, the first image pickup unit 200 and the second image pickup unit 300 are allowed to take an image in any of the states. , In the fifth open state (opening angle θ = 0 degree), control is performed so as to prohibit imaging. The reason why imaging is prohibited in the fifth open state is that the first imaging unit 200 and the second imaging unit 300 only image each other, and cannot substantially capture another subject.

The recording / reading control unit 530 is used for recording processing for recording image data captured by the first imaging unit 200 and the second imaging unit 300 and signal-processed by the signal processing unit 520 on the storage medium 900, respectively. A read-out process for reading out image data recorded on the storage medium 900 is performed.

In the storage medium 900, the data of the first image captured by the first imaging unit 200, the data of the second image captured by the second imaging unit 300, and the data of the second image captured by the second imaging unit 300 are detected by the angle sensor 410 at the time of imaging. Information indicating the opening angle θ is recorded in association with the information. By recording the information indicating the opening angle θ at the time of imaging, the first imaging described later will be performed based on the opening angle θ, the design information of the optical systems of the first imaging unit 200 and the second imaging unit 300, and the like. It is possible to obtain an overlapping range DUP (FIGS. 7 and 8) in which the photographing range B200 by the unit 200 and the photographing range B300 by the second imaging unit 300 overlap.

The open / close control unit 540 sends a drive signal to the drive unit 420 based on the operation signal from the open / close button 820 of the operation member 800. By driving the motor of the drive unit 420, the first and second imaging units 200 and 300 rotate in opposite directions around the axis AX of the connection unit 400, and the opening angle θ changes. As described above, the open / close control unit 540 sends a drive signal for driving the drive unit 420 in a direction of increasing the opening angle θ, or reduces the opening angle θ, based on the operation signal from the open / close button 820. A drive signal for driving the drive unit 420 is sent in the direction.

For example, when the user operates the open / close button 820, the motor starts driving and the first and second imaging units 200 and 300 rotate. In the image pickup apparatus 100 of the embodiment, while the user continuously operates the open / close button 820, the operation of changing the opening angle θ is continued, and the user recognizes the first to fifth opening states. When the operation of the open / close button 820 is stopped, the operation of changing the opening angle θ is stopped.
When the user operates the open / close button 820 once, the motor may be automatically continued to be driven until the angle sensor 410 detects the preset first to fifth opening angles. ..
The setting to the first to fifth open states is not limited to the above example. For example, even if the open / close button 820 is continuously operated in the direction of increasing the opening angle θ, when the angle sensor 410 detects an angle corresponding to each opening state, the drive of the motor is temporarily stopped, and the motor is driven again. When the open / close button 820 is operated, the driving of the motor may be restarted until the next opening angle is detected.
Further, the opening angle is not limited to the first to fifth opening angles, and may be stopped at an arbitrary opening angle between the first to fifth opening angles.

(Display control unit 600)
The display control unit 600 is a first image captured and acquired by the first image pickup unit 200 and the second image pickup unit 300 based on an instruction transmitted from the control unit 500 by the user operating the operation member 800. And, based on at least one of the data of the second image or at least one of the data of the first and second images recorded on the storage medium 900, a display image to be displayed on the display unit 700 is generated. ..

(Display unit 700)
The display unit 700 is composed of, for example, a liquid crystal display panel, an organic EL display panel, or the like, and the liquid crystal display panel or the like is configured to be openable and closable as shown in FIG. The display unit 700 displays a display image generated by the display control unit 600, a setting menu screen, and the like based on an instruction transmitted from the control unit 500.

The display unit 700 is provided with an operation detection unit 710 such as a touch panel switch on the display surface thereof. The operation detection unit 710 detects a contact operation (touch operation, swipe operation, etc.) on the display surface by the user, and sends an operation signal indicating the contacted position to the display control unit 600.
The operation signal from the operation detection unit 710 may be sent to the control unit 500.

(Operating member 800)
The operation member 800 includes buttons and switches such as the release button 810 and the open / close button 820 of FIG. 1, and sends an operation signal corresponding to the operation of the user to the control unit 500.

(Storage medium 900)
The storage medium 900 is composed of, for example, a memory card that can be attached to and detached from the image pickup apparatus 100. The recording and reading of data on the storage medium 900 is controlled by the recording / reading control unit 530 described above.

<Storage of display>
FIG. 5 is a schematic diagram illustrating the operation of the display unit 700. FIG. 5A is a diagram showing a stored state (when not in use), and FIG. 5B is a diagram showing a display unit 700 in use. The display unit 700 can be housed on the back side of the first image pickup unit 200 or on the back side of the second image pickup unit 300. As illustrated in FIG. 5A, in the present embodiment, the display unit 700 is housed on the back side of the first imaging unit 200. Therefore, the display unit 700 does not appear in the appearance in the first open state (opening angle θ = 360 degrees).

In the second opening state (180 degrees <opening angle θ <360 degrees), the display unit 700 gradually appears in the appearance as the opening angle θ becomes smaller. Then, for example, when the opening angle θ is smaller than 270 degrees, the display unit 700 can be opened in the direction of the arrow as illustrated in FIG. 5 (b).

<Overlapping shooting range>
The overlap (overlap) of the imaging ranges by the first imaging unit 200 and the second imaging unit 300 will be described with reference to FIG. FIG. 6 is a schematic view showing the image pickup apparatus 100 and the object P. The image pickup apparatus 100 of FIG. 6 corresponds to the fourth open state (FIG. 3 (d)). Therefore, the position where the object P is imaged differs between the first imaging unit 200 and the second imaging unit 300. That is, when the image of the object P captured by the first imaging unit 200 and the image of the object P captured by the second imaging unit 300 are compared, the direction in which the object P is viewed is different.

FIG. 7 is a schematic diagram illustrating an imaging range imaged by the first imaging unit 200 and the second imaging unit 300, where reference numeral B200 is the imaging range taken by the first imaging unit 200 and reference numeral B300 is the second. This is the shooting range by the imaging unit 300. Further, the reference numeral DUP is an overlapping range in which the shooting ranges B200 and B300 overlap. In the present embodiment, since the open state of the first imaging unit 200 and the second imaging unit 300 can be changed, the imaging range B200 by the first imaging unit 200 and the imaging by the second imaging unit 300 The amount of overlap (that is, the size of the overlap range DUP) can be changed by widening or narrowing the overlap range DUP that overlaps with the range B300.

FIG. 8 is a diagram illustrating the relationship between the opening state of the second imaging unit 300 with respect to the first imaging unit 200 (opening angle θ around the axis AX) and the overlapping range DUP.
8 (a) shows, as shown in FIG. 3 (a), a first opening in which the optical axes X200 and X300 of the first and second imaging units 200 and 300 are coaxial at an opening angle θ = 360 degrees. It is a figure which shows the overlap range DUP in a state, and each small area of the photographing range B200 and the photographing range B300 is included in the overlap range DUP as the above-mentioned glue margin. Since both the fisheye lens 210 and the fisheye lens 310 have an angle of view exceeding 180 degrees, there are two overlapping range DUPs on both sides of the shooting range B200 and the shooting range B300. The right side of the shooting range B200 shown in FIG. 8A and the left side of the shooting range B300 overlap (the central portion of FIG. 8A is shown overlapping). Further, the left side of the shooting range B200 and the right side of the shooting range B300 also overlap (in FIG. 8A, both sides are shown separately).
In FIG. 8B, as shown in FIGS. 2A and 3B, the optical axes X200 and X300 of the first and second imaging units 200 and 300 intersect at an opening angle θ = 270 degrees. It is a figure which shows the overlap range DUP in the 2nd open state, and about half of each of the photographing range B200 and the photographing range B300 is included in the overlap range DUP. Unlike when the opening angle θ = 360 degrees (FIG. 8A), the overlapping range DUP is one place.
In FIG. 8 (c), as shown in FIGS. 2 (b) and 3 (c), the optical axes X200 and X300 of the first and second imaging units 200 and 300 are coaxial with each other at an opening angle θ = 180 degrees. It is a figure which shows the overlap range DUP in the 3rd open state, and the whole area of each of the imaging range B200 and the imaging range B300 is included in the overlap range DUP.
In FIG. 8 (d), as shown in FIGS. 2 (c) and 3 (d), the optical axes X200 and X300 of the first and second imaging units 200 and 300 intersect at an opening angle θ = 90 degrees. It is a figure which shows the overlap range DUP in the 4th open state, and about half of each of the photographing range B200 and the photographing range B300 is included in the overlap range DUP.
The overlap range of the first to fourth open states DUP is the second of the first open state (θ = 360 degrees) <the second open state (θ = 270 degrees). The amount of overlap is the third amount of overlap in the third open state (θ = 180 degrees). The second overlapping amount and the fourth overlapping amount in the fourth open state (θ = 90 degrees) are equivalent.
It should be noted that the horizontally long rectangular shooting range (effective shooting area of the shooting elements 220 and 320) of FIGS. 7 and 8 exceeds 180 degrees taken by the first image pickup unit 200 and the second image pickup unit 300. An image acquired through the fisheye lens 210 having an angle of view and the fisheye lens 310 appears.

In the overlapping range DUP of the imaging range taken by the imaging device 100, the data of the first image captured by the first imaging unit 200 and the data of the second image captured by the second imaging unit 300 are obtained. Be done. Since the data of the first image and the data of the second image have parallax information in the overlap range DUP, the overlap range DUP is a region having the depth information. The user can stereoscopically view (in other words, obtain a sense of depth based on parallax) by observing the first image and the second image of the overlapping range DUP with an external device capable of stereoscopic viewing.

When, for example, a head-mounted display (HMD) (not shown) as a stereoscopic image observation device is connected to the image pickup device 100 by wireless communication or wired communication, the control unit 500 is recorded in association with the storage medium 900. The data of the image of 1 and the data of the second image are read out. Then, the control unit 500 transmits the image data corresponding to the overlapping range DUP among the data of the first image and the data of the second image to the head-mounted display. As described above, the image data corresponding to the overlap range DUP includes the opening angle θ recorded in the storage medium 900 in association with the image data, and the optics of the first imaging unit 200 and the second imaging unit 300. This is image display data calculated based on the system design information.

With this configuration, of the first image captured by the first imaging unit 200, the image in the region overlapping with the data of the second image captured by the second imaging unit 300 is head-mounted. Of the second images displayed on the display unit for the left eye of the display, observed by the left eye of the user, and captured by the second imaging unit 300, the first image captured by the first imaging unit 200. The image of the area overlapping with the data of the above is displayed on the display unit for the right eye of the head mount display and observed by the right eye of the user. The user can perform stereoscopic viewing by observing the images displayed on the display unit for the right eye and the display unit for the left eye of the head-mounted display.
As described above, the distance from the image pickup apparatus 100 to the object can be obtained for the object reflected in the region having the depth information (overlapping range DUP) by the principle of triangulation.

On the other hand, among the data of the first image captured by the first imaging unit 200, the data of the image in the region other than the overlapping range DUP and the second image captured by the second imaging unit 300. It does not have parallax information with the image data of the area other than the overlapping range DUP among the data of. That is, the region of the imaging range taken by the image pickup apparatus 100 excluding the overlapping range DUP is a region that does not have Depth information.

When the stereoscopic image observation device such as the head mount display is not connected to the image pickup device 100 by wireless communication or wired communication, the control unit 500 is the first of the shooting range B200 imaged by the first image pickup unit 200. Only the image of the above or only the second image of the shooting range B300 captured by the second imaging unit 300 is displayed on the display unit 700, and the flat image (2D image) is displayed to the user without using an external device. Let them observe. In this way, the user can also observe a flat image that is not a stereoscopic image based on the data of the image taken by the image pickup apparatus 100.
It is also possible to output the image data to be displayed on the display unit 700 to an external device and display the image data on the display unit. A display device capable of observing a stereoscopic image (stereoscopic viewing) may be used as the display unit 700.

<Image display device that generates and displays images viewed from any direction>
The image display device 100 of the embodiment is based on an image having parallax acquired from a plurality of directions (parallax image) recorded in association with the storage medium 900, and the direction set by the setting unit 620 described later. It has a function of displaying the image viewed from the front on the display unit 700. The generation and display of an image viewed from an arbitrary direction based on such a parallax image will be described below.

For image generation viewed from an arbitrary direction, an image having an overlapping range DUP in which the shooting range B200 by the first imaging unit 200 and the shooting range B300 by the second imaging unit 300 overlap is used. In the following description, the image display device 100 as the image pickup device described above sets the opening angle θ between the first image pickup unit 200 and the second image pickup unit 300 to, for example, one of the ranges of 270 degrees to 90 degrees. It is assumed that images of the object viewed from a plurality of directions are taken in this state, and the plurality of images are associated and recorded on the storage medium 900. The opening angle θ in the range of 270 degrees to 90 degrees means that the opening states of the first imaging unit 200 and the second imaging unit 300 are the states shown in FIGS. 2 (a) to 2 (c). The overlapping range DUPs in which the imaging range B200 of the first imaging unit 200 and the imaging range B300 of the second imaging unit 300 overlap in the state are shown in FIGS. 8 (b) to 8 (d), respectively.

The configuration of the display control unit 600 will be described with reference to FIG. The display control unit 600 includes an object determination unit 610, a setting unit 620, a generation unit 630, a composition unit 640 that synthesizes an image of the generated object, and a distance calculation unit 650.

The target determination unit 610 captures the first direction and the second image of the first image pickup unit 200 among the images displayed on the display unit 700 (the image of the overlap range DUP in the first image or the second image). A region to be converted into an image of an object viewed from a third direction different from the second direction of the unit 300 is determined. The target determination unit 610 determines, for example, a region of an object or a region having a predetermined shape including the object as a region to be converted. Specifically, the target determination unit 610 detects an area of a desired object by performing subject recognition processing on the image displayed on the display unit 700, and determines the detected area as the conversion target area.
The subject recognition process is, for example, a process of extracting a person's face by a known face recognition process if it is a person's face. In the imaging device of the embodiment, when a mode for displaying an image of the appearance of a subject viewed from a plurality of directions is selected, a menu screen for specifying an object to be recognized is displayed, and from the menu screen, for example, a person's face. Select.

Further, the target determination unit 610 may determine the conversion target area based on the operation signal from the operation detection unit 710 such as the touch panel switch provided on the display surface of the display unit 700. Specifically, among the images displayed on the display unit 700, the area surrounded by the user's finger on the display surface is determined by the target determination unit 610 as the area to be converted.
That is, the object for which an image viewed from an arbitrary direction is desired may be automatically selected, or may be manually selected.
In this embodiment, the selected object may be referred to as a main subject.

The setting unit 620 sets the direction in which the object is viewed based on the operation signal from the operation detection unit 710 provided on the display surface of the display unit 700, that is, based on the user's operation. The direction set by the setting unit 620 is set by the generation unit 630 that generates an image of the object viewed from a third direction different from the first direction of the first imaging unit 200 and the second direction of the second imaging unit 300. Used. The setting unit 620 changes (moves) the direction in which the object is viewed so that the object rotates in the direction in which the user swipes the finger on the display surface of the display unit 700, for example. The setting unit 620 further sets the amount of change (movement amount) in the direction in which the object is viewed to be larger as the swipe operation amount is larger. The setting unit 620 may set the direction directly specified by the user on the display surface of the display unit 700 as the direction for viewing the object.

The generation unit 630 generates an image of an object viewed from the direction set by the setting unit 620 for the area of the conversion target determined by the target determination unit 610 among the images displayed on the display unit 700. In the generation unit 630, the more the direction of change in the direction set by the setting unit 620 is toward the right side of the screen, the more the image viewed from the set direction is captured by the second imaging unit 300 and the second image is acquired. As the direction of change in the direction set by the setting unit 620 is toward the left side of the screen, the image viewed from the set direction approaches the first image captured and acquired by the first imaging unit 200. As described above, an image of an object viewed from a set direction is generated. An example of the generated image will be described later with reference to FIGS. 9-12.

The synthesis unit 640 is the target determination unit 610 of the data of the first image captured and acquired by the first imaging unit 200 or the data of the second image acquired by the second imaging unit 300. The data of the area to be converted determined in is converted into the data of the image generated by the generation unit 630 (the image seen from the direction set by the setting unit 620), and the data of the image of the background area other than the target area is combined with the data of the image. Perform the process of synthesizing.

The distance calculation unit 650 calculates the distance from the image display device 100 to the object using the principle of triangulation. The specific description is as follows. The object in the first and second images is formed of a plurality of feature points (parts representing the features of the object) that constitute the object. In the first image and the second image, the same feature points are specified by the so-called stereo matching method, and the distances to those feature points are calculated by the principle of triangulation. Therefore, the distance to the object is the distance to a plurality of feature points. The calculation and storage of the distances to the plurality of feature points in this way is to change the distances to the plurality of feature points in each arbitrary direction in order to generate an image viewed from any plurality of directions, which will be described later. This is to make you.

<Generation of an image of an object viewed from the direction set by the setting unit 620>
As described above, the generation unit 630 generates an image viewed from an arbitrary direction from images viewed from a plurality of directions (two images in this example). The images viewed from a plurality of directions are two images viewed from the first direction of the first imaging unit 200 and the second direction of the third imaging unit 300 in the embodiment. The image viewed from an arbitrary direction is an image viewed from a third direction (direction set by the setting unit 620 in this example) different from the first and second directions. Specifically, the generation unit 630 performs the following processes (1) and (2).
(1) The first image of the object captured from the first direction by the first imaging unit 200 and the second image of the object captured from the second direction by the second imaging unit 300. Based on this, the correspondence between the two images is obtained by using the feature points appearing in both images. The correspondence relationship is a correspondence relationship between each feature point of the object in the first and second images and the distance to the feature point calculated by the distance calculation unit 650.
(2) A third direction between the first direction and the second direction based on the first and second images, the distance to the object calculated by the distance calculation unit 650, and the direction set by the setting unit 620. Generate an image viewed from the direction.
In this embodiment, the third direction may be referred to as a virtual viewpoint, an arbitrary viewpoint, or an intermediate viewpoint.
A technique for generating an image viewed from an arbitrary direction using a plurality of images in which the object is viewed (photographed) in different directions is known (for example, Japanese Patent Application Laid-Open No. 2005-215724). Generating an image viewed from a third direction located between the first and second directions based on images viewed from the first and second directions I will omit the detailed explanation and refer to the direction in which the object is viewed. The following explanation will be given focusing on displaying the changed image.

9 (a) and 9 (b) are diagrams for explaining display examples of images before and after changing the viewing direction of the object, and FIG. 9 (a) is a diagram showing before the change, FIG. 9 (b). ) Is a diagram showing after the change. In the example of FIG. 9A, it is assumed that the region of the object Pa in the image displayed on the display unit 700 is determined as the region to be converted by the target determination unit 610.

The control unit 500 generates an image viewed from an arbitrary direction from images viewed from a plurality of directions, and when a display mode is selected, the display control unit 600 causes the display unit 700 to perform initial display. The initial display is a process of displaying an image of the overlapping range DUP in the first image by the first imaging unit 200 or an image of the overlapping range DUP in the second image by the second imaging unit 300 on the display unit 700. Is.

In the present embodiment, the generation unit 630 displays a display image (FIG. 9A) generated based on, for example, the data of the second image by the second imaging unit 300 as the initial display before changing the viewing direction. It is displayed on the display unit 700. The display image shown in FIG. 9A is an image of the object P viewed from the second direction (direction from the center of the image sensor of the second image pickup unit 300 to the object P).

As an initial display before the direction is changed by the setting unit 620, a display image (FIG. 9B) generated based on the data of the first image by the first image sensor 200, that is, the first direction (first direction). The image of the object P viewed from the center of the image sensor of the image pickup unit 200 of 1 toward the object P) may be displayed on the display unit 700.

Further, the user may be able to select which of the data of the first and second images the display image generated based on the data is displayed on the display unit 700 as the initial display. The selection operation by the user can be performed, for example, on the setting menu screen. The mode for generating and displaying an image viewed from an arbitrary direction from the above-mentioned images viewed from a plurality of directions is also selected on the setting menu screen.

In the initial display of FIG. 9A, the setting unit 620 sets (changes) the viewing direction of the object P as follows. The generation unit 630 includes a display image generated based on the data of the second image which is the initial display of the object P viewed from the second direction (the object viewed from the second direction is Pa), and the first direction. The mark 81 indicating the direction of is displayed on the display unit 700.

The mark 81 in FIG. 9A is displayed on the left side of the object Pa when viewed from the second direction, and indicates that the direction in which the object P is viewed can be changed to the direction of the mark 81. The user starts a swipe operation to move the position of the finger shown in FIG. 9A in the right direction (arrow direction), which is the direction in which the object Pa is rotated. When the setting unit 620 changes the direction in which the object P is viewed to the direction of the mark 81 based on the swipe operation amount, the object Pa rotates to the right. Based on the operation signal from the operation detection unit 710 provided on the display surface of the display unit 700, the setting unit 620 moves the direction of viewing the object P to the left by an amount based on the movement amount of the finger.

The setting unit 620 calculates the direction in which the object P is viewed based on the movement amount (swipe operation amount) of the finger moved by the swipe operation. In this example in which the image viewed from the second direction (FIG. 9A) is displayed as the initial display, when the swipe operation amount reaches, for example, about 1/3 of the length of the display screen in the left-right direction, the operation is performed. It is determined that the amount is the maximum, and the direction in which the object P is viewed is moved to the imaging direction (first direction) of the first imaging unit 200.

FIG. 9B shows a display image when the setting unit 620 changes the viewing direction of the object P from the second direction to the first direction in FIG. 9A, and the generation unit 630 is the first imaging unit. It was created using 200 image data. When the direction in which the object P is viewed is changed from the first image pickup unit 200 to the first direction, the generation unit 630 generates a display based on the data of the first image captured by the first image pickup unit 200. An image (FIG. 9 (b)) is displayed on the display unit 700 (the object viewed from the first direction is Pb). Then, in order to inform the user of the direction in which the direction in which the object P can be viewed can be changed, the mark 82 indicating the direction in the second direction is placed on the display unit 700 together with the display image generated based on the data of the first image. To display. The mark 82 is displayed on the right side of the object Pb when viewed from the first direction, and indicates that the direction in which the object P is viewed can be changed to the direction of the mark 82.

Further, in the state where the swipe operation amount is less than 1/3 of the length in the left-right direction of the display screen, the setting unit 620 has the second direction and the first from the second image pickup unit 300 according to the swipe operation amount. The direction in which the object P is viewed is set in the direction between the image pickup unit 200 and the first direction.

When the position of the finger by the swipe operation is moved to the intermediate position between the position of the finger shown in FIG. 9A and the position of the finger shown in FIG. 9B, the generation unit 630 moves the setting unit 620 to the intermediate position. The image of the object P viewed from the direction set corresponding to the intermediate position is captured by the data of the first image captured by the first imaging unit 200 and the second image captured by the second imaging unit 300. Generated based on the image data of. That is, based on the above-mentioned correspondence, the distance of the feature points of the object is converted into a distance according to the viewing direction to generate an image viewed from an arbitrary direction.

In general, an object located at a short distance from the image display device 100 has a large difference between an image viewed from the first direction and an image viewed from the second direction. On the contrary, the object located at a distance from the image display device 100 has a small difference between the images. Therefore, the generation unit 630 increases the change in the direction of the object P per unit movement amount of moving the finger on the display screen as the object is closer to the image display device 100, and the object is farther from the image display device 100. The larger the object, the smaller the change in the direction of the object P per unit movement amount. With this configuration, the orientation of the object P can be changed appropriately.
The change in the orientation of the object P is a change in the rotation angle centered on the rotation axis Q in FIGS. 9A and 9B.

The synthesis unit 640 replaces the data of the region to be converted in the data of the second image captured by the second image pickup unit 200 with the data of the image generated by the generation unit 630. That is, the background area image around the area of the object P and the image of the object P viewed from an arbitrary direction are combined.
Here, the setting of the viewing direction of the object P by the setting unit 620, the generation of the image of the object P by the generation unit 630, and the composition processing by the composition unit 640 follow the swipe operation of the user. It is said. That is, the orientation of the displayed object P gradually changes from the display of the object Pa to the display of the object Pb as the finger moves in the swipe operation. When the position of the finger is returned to the left by the swipe operation, the direction of the displayed object P gradually changes from the display of the object Pb to the display of the object Pa as the finger moves in the swipe operation. Change.

In FIGS. 10A and 10B, for example, the attributes (categories) related to the object selected on the menu screen are box-shaped, and three box-shaped shapes are included in the image captured by the image display device 100. An example is shown in which the objects P1 to P3 of the above exist, and the region of one of the objects P2 is determined as the region to be converted.
FIG. 10A shows a display image of the initial display. The initial image is the same as that described with reference to FIGS. 9A and 9B, for example, the image of the objects P1 to P3 viewed from the second direction by the second imaging unit 300, that is, the data of the second image. It was generated based on. The objects P1 to P3 viewed from the second direction are P1a, P2a, and P3a, respectively.

In the example of FIG. 10A, since the target determination unit 610 sets the region of the target P2a as the region to be converted, the mark 81 is displayed on the left side of the target P2a when viewed from the second direction. In FIG. 10A, the mark 81 informs the user that the direction in which the object P2 is viewed can be changed in the direction of the mark 81, that is, the object P2a can be rotated to the right. The user who sees the mark 81 can start a swipe operation for moving the position of the finger shown in FIG. 10A, for example, in the right direction (arrow direction), which is the direction in which the object P2a is rotated.

In the examples shown in FIGS. 10A and 10B, the arithmetic processing performed by the setting unit 620, the generating unit 630, and the synthesizing unit 640 is the same as that described in FIGS. 9A and 9B. Omit. Further, the setting unit 620 sets the viewing direction of the object P2, the generation unit 630 generates an image of the object P2, and the composition unit 640 performs the composition process following the swipe operation of the user. The same is true. Therefore, even in the examples of FIGS. 10A and 10B, the orientation of the displayed object P2 gradually changes from the display of the object P2a to the display of the object P2b as the finger moves in the swipe operation. To do. When the position of the finger is returned to the left by the swipe operation, the display of the object P2b is gradually changed to the display of the object P2a as the finger is moved by the swipe operation, and the direction of the displayed object P2 is gradually changed. Change.

In FIGS. 11 (a) and 11 (b), similarly to FIGS. 10 (a) and 10 (b), for example, the attribute (category) related to the object selected on the menu screen is box-shaped, and the image display device. Three box-shaped objects P1 to P3 are present in the image captured by 100, and the regions of the objects P1, P2, and P3 in the image displayed on the display unit 700 are the target determination units, respectively. An example is shown in which the region to be converted is determined by 610.
FIG. 11A shows a display image of the initial display. The initial image is the same as that described with reference to FIGS. 9A and 9B, for example, the image of the objects P1 to P3 viewed from the second direction by the second imaging unit 300, that is, the data of the second image. It was generated based on. The objects P1 to P3 viewed from the second direction are P1a, P2a, and P3a, respectively.

In the example of FIG. 11A, since the target determination unit 610 sets the regions of the objects P1a, P2a, and P3a as the conversion target regions, the mark 81 is displayed on the left side of the object P1a when viewed from the second direction. Will be done. In FIG. 11A, the mark 81 can change the direction in which the objects P1 to P3 are viewed in the direction of the mark 81, that is, the marks 81 can rotate to the right with respect to the objects P1a, P2a, and P3a, respectively. Tell the user that. The user who sees the mark 81 starts a swipe operation for moving the position of the finger shown in FIG. 11A in the right direction (arrow direction), which is the direction in which the objects P1a, P2a, and P3a are rotated. be able to.
Here, the setting of the direction of viewing the objects P1 to P3 by the setting unit 620, the generation of the images of the objects P1, P2, and P3 by the generation unit 630, and the composition processing by the composition unit 640 are swipes of the user. It is performed according to the operation. That is, the directions of the displayed objects P1, P2, and P3 gradually change from the display of the objects P1a, P2a, and P3a to the display of the objects P1b, P2b, and P3b as the finger moves in the swipe operation. To do. When the position of the finger is returned to the left by the swipe operation, the display of the objects P1b, P2b, P3b is displayed on the display of the objects P1a, P2a, P3a as the finger moves in the swipe operation. The directions of the objects P1, P2, and P3 gradually change.

12 (a) and 12 (b) have a box-shaped attribute (category) related to the object selected on the menu screen, as in the case of FIGS. 11 (a) and 11 (b), and are image display devices. Three box-shaped objects P1 to P3 exist in the image captured by 100, and the regions of the objects P1, P2, and P3 in the image displayed on the display unit 700 are combined into one. An example is shown in which the area to be converted is determined by the target determination unit 610.
FIG. 12A shows a display image of the initial display. The initial image is the same as that described with reference to FIGS. 9A and 9B, for example, the image of the objects P1 to P3 viewed from the second direction by the second imaging unit 300, that is, the data of the second image. It was generated based on. The objects P1 to P3 viewed from the second direction are P1a, P2a, and P3a, respectively.

In the example of FIG. 12A, since the target determination unit 610 collectively sets the regions of the objects P1a, P2a, and P3a as the regions to be converted, the mark 81 is the objects P1a, P2a, viewed from the second direction. It is displayed on the left side of the area containing P3a. In FIG. 12A, the mark 81 can change the viewing direction of the objects P1 to P3 in the direction of the mark 81, that is, can rotate the objects P1a, P2a, and P3a to the right. To the user. The user who sees the mark 81 starts a swipe operation for moving the position of the finger shown in FIG. 12A, for example, in the right direction (arrow direction), which is the direction in which the objects P1a, P2a, and P3a are rotated. be able to.

Here, the setting of the viewing direction of the objects P1 to P3 by the setting unit 620, the generation of the images of the objects P1, P2, and P3 by the generation unit 630, and the composition processing by the composition unit 640 are swipes of the user. It is performed according to the operation. That is, as the finger moves in the swipe operation, the display of the objects P1a, P2a, and P3a gradually changes to the display of the objects P1c, P2c, and P3c. .. When the position of the finger is returned to the left by the swipe operation, the display of the objects P1c, P2c, and P3c is displayed on the display of the objects P1a, P2a, and P3a as the finger moves in the swipe operation. The directions of the objects P1 to P3 collectively change gradually.

When the regions of the objects P1a, P2a, and P3a are collectively determined as the region to be converted, the generation unit 630 makes the display images of the objects P1c, P2c, and P3c different in size. In the example of FIG. 12A, the sizes of the displayed images of the objects P1a, P2a, and P3a before the viewing direction is changed are the same. However, in the example of FIG. 12B, the size of the display image of the objects P1c, P2c, and P3c after changing the viewing direction is set so that the relationship of the objects P1c>P2c> P3c is established. .. In this way, the display unit 700 is observed by making the size of the display image larger in the foreground of the screen and smaller toward the back (P2c is smaller than the object P1c and P3c is smaller than the object P2c). You can give the user a sense of depth.
As shown in FIGS. 12A and 12B, when the regions of a plurality of objects P1c, P2c, and P3c are collectively determined as the region to be converted, the central axis Q of the combined region is used. Rotate around the center. Therefore, the rotation angle with respect to the swipe operation amount is set in advance. Then, the size of each object is displayed smaller as the object in the back corresponds to the rotation angle.

Various display images of the object generated by the generation unit 630 have been described with reference to FIGS. 9 to 12. When the size of the image of the object generated after changing the viewing direction is smaller than the size of the image of the object before changing the direction due to the shape of the object, the generation unit 630 appears by reducing the size. The background part of the object is subjected to creation processing or interpolation processing using the background information around the object.
Further, instead of performing the creation process and the interpolation process, the generation unit 630 performs a direction so that the size of the image of the object generated after the viewing direction is changed is not smaller than the size of the image of the object before the direction change. The size of the image of the object generated after the change may be changed to be larger than the size of the image of the object before the direction change.

Furthermore, when the generation unit 630 makes the user feel an unnatural sense of depth in the generated image (the image of the object after changing the viewing direction) due to the shape of the object, the generation unit 630 gives a natural sense of depth. The keystone correction process for making the person feel may be applied to the image of the object after the direction is changed.

<Explanation of flowchart>
The flow of the image display process for generating and displaying an image in which the viewing direction is changed will be described with reference to the flowchart of FIG. The control unit 500 performs an operation of displaying an image whose viewing direction is changed, for example, when a mode of generating an image viewed from an arbitrary direction from images viewed from a plurality of directions is selected, the process of FIG. 13 is performed. Run the program you want to do. The generation of the image viewed from an arbitrary direction is performed by the generation unit 630 based on the data of the image captured by the image display device 100.

In step S10 of FIG. 13, the control unit 500 causes the display unit 700 to perform the initial display. Specifically, the control unit 500 sends an instruction to the display control unit 600, and the display image used for the initial display is based on the image data of the overlap range DUP in the second image captured by the second image pickup unit 300. Is generated and displayed on the display unit 700. In this example, the image initially displayed on the display unit 700 is an image viewed from the second direction.

In step S20, the control unit 500 causes the target determination unit 610 to determine the area to be converted based on the operation of the user, and proceeds to step S30. In step S30, the control unit 500 determines whether or not to change the viewing direction. When the user swipes the display surface of the display unit 700, the control unit 500 positively determines step S30 and proceeds to step S40. If the swipe operation is not performed, the control unit 500 waits until the swipe operation is performed.

In step S40, the control unit 500 instructs the setting unit 620 to set an arbitrary direction based on the swipe operation amount, and proceeds to step S50. The setting unit 620 sets the amount of change in the viewing direction as the swipe operation amount increases so that the object displayed on the display unit 700 rotates in the swipe operation direction.

In step S50, the control unit 500 causes the generation unit 630 to generate an image of an object having a different viewing direction for the conversion region determined in step S20 of the images displayed on the display unit 700, and proceeds to step S60. move on.

In step S60, the control unit 500 determines in step S20 the image of the conversion region among the images displayed on the display unit 700 before the initial display image is displayed by the synthesis unit 640 in step S50. A compositing process for replacing the generated image (combining the image data of the area of the object and the image data of the background) is performed, and the process proceeds to step S70. The image displayed on the display unit 700 after the compositing process is an image after changing the viewing direction.

In step S70, the control unit 500 determines whether or not to end the process of FIG. When the end operation is performed by the user, the control unit 500 positively determines step S70 and ends the process. Further, when the user swipes the display surface of the display unit 700 or touches the display surface of the display unit 700, the control unit 500 negatively determines step S70 and proceeds to step S80. move on.

In step S80, the control unit 500 determines whether to change the viewing direction or the conversion target area. When the user swipes the display surface of the display unit 700, the control unit 500 determines that the viewing direction has been changed, returns to step S40, and causes the setting unit 620 to set the viewing direction. Further, when the user touches the display surface of the display unit 700, the control unit 500 determines that the conversion target has been changed, returns to step S20, and determines the conversion target area by the target determination unit 610. Let me. After that, the above-mentioned process is repeated.
When the finger touching the display screen is stopped for a predetermined time or longer, it is determined to be a touch operation. When it is detected that the finger on the screen has moved within a predetermined time, it is judged as a swipe operation.

According to the image display device 100 described above, the following effects can be obtained.
(1) The image display device 100 is based on two images (first image and second image) obtained by imaging the object P from two different directions (first direction and second direction). , The distance calculation unit 650 that calculates the distance to the object P in the overlapping range DUP where these two images overlap, and the first and second directions are different based on the distances to the two images and the object P. It includes a generation unit 630 that generates an image of the object P viewed from the direction, and a display control unit 600 that displays the generated image on the display unit 700. With this configuration, the display unit 700 can display an image of the object P that can be seen from any direction different from the first and second directions.

(2) The image display device 100 includes a setting unit 620 for setting different directions, and the generation unit 630 generates an image of the object P set by the setting unit 620 and viewed from different directions. With this configuration, the display unit 700 can display an image of the object P that can be seen from the direction set by the setting unit 620.

(3) The image display device 100 has two imaging units (first imaging unit 200, second imaging unit 200, second imaging unit) for capturing two images in a direction in which different directions can be set (for example, a left direction or a right direction). A setting unit 620 is provided as a calculation unit that calculates based on the direction (first direction, second direction) in which the unit 300) captures an image. With this configuration, it is possible to calculate whether the viewing direction can be changed to the left or right.

(4) The image display device 100 has a generation unit 630 as a notification unit that notifies the direction calculated by the setting unit 620 as the calculation unit. With this configuration, for example, the marks 81 and 82 can be displayed on the display unit 700 to inform the user of the direction in which the viewing direction can be set (changed).

(5) Since the generation unit 630 as the notification unit displays the marks 81 and 82 indicating the calculated directions on the display unit 700, the direction in which the viewing direction can be changed for the user can be visually displayed. I can tell you.

(6) Since the distance calculation unit 650 calculates the distance by performing stereo matching processing between the two images (first image and second image), it emits light or radio waves for distance measurement. The distance to the object P can be obtained without.

(7) Since the generation unit 630 changes the direction of the object P in the image according to the change in the viewing direction, the direction of the displayed object P can be smoothly changed.

(8) The generation unit 630 changes the rotation angle (direction) around the axis Q per unit movement amount in different directions in the image displayed on the display unit 700 as the object P becomes closer. The orientation of the object P can be changed with an appropriate perspective.

(9) The image display device 100 recognizes a region of an object P or a region of a predetermined shape including an object P based on two images (a first image and a second image) of a target determination unit 610. Be prepared. The generation unit 630 generates an image having a different viewing direction with respect to the region recognized by the target determination unit 610. With this configuration, it is possible to reduce the processing load as compared with the case of generating an image in which the viewing direction is different for the entire image.

(10) The image display device 100 as an imaging device is connected by a connecting unit 400, images an object P from different directions, and two imaging units (third) in which the overlapping range DUP of the shooting ranges B200 and B300 is changed. The object P from the first direction imaged by the first image pickup unit 200 and the second direction imaged by the second image pickup unit 300 by the first image pickup unit 200 and the second image pickup unit 300) and the two image pickup units. A generation unit 630 that generates an image of an object P viewed from a direction different from the first direction and the second direction based on the image of the overlapping range DUP of the two captured images obtained by imaging the image, and a direction different from that of the generation unit 630. It is provided with a display unit 700 for displaying an image of the object P viewed from above. With this configuration, the display unit 700 can display an image of the object P that can be seen from any direction.

(11) The two imaging units (first imaging unit 200, second imaging unit 300) rotate around the connection unit 400 to change the overlapping range DUP of the mutual imaging ranges B200 and B300. With this configuration, the first direction in which the first image pickup unit 200 takes an image and the second direction in which the second image pickup unit 300 takes an image can be continuously and seamlessly changed.

(12) The image display device 100 as an image pickup device includes a setting unit 620 for setting different directions. With this configuration, the direction of the object P to be displayed on the display unit 700 can be set.

(13) The generation unit 630 changes the direction of the image of the object P displayed on the display unit 700 when the different directions change. With this configuration, the orientation of the displayed object P can be gradually changed as the different directions change.

The following modifications are also within the scope of the present invention, and one or more of the modifications can be combined with the above-described embodiment.
(Modification example 1)
As shown in FIGS. 9 (a) and 9 (b), the image of the object whose viewing direction is changed is shown as an example of being rotated around the vertical axis Q passing through the center of the object Pa. The axis of rotation may be at the left end of the object Pa, at the right end of the object Pa, or at any position of the object Pa. In this modification 1, the position of the rotation axis Q can be set, for example, on the menu screen.

(Modification 2)
When a shadow appears around the object, when the image of the object whose viewing direction is changed is displayed on the display unit 700, the shadow of the object displayed on the display unit 700 before and after the viewing direction is changed. On the other hand, the image correction process for changing the shape of the shadow after changing the viewing direction may be performed so that the user does not feel uncomfortable.

(Modification example 3)
In the above description, an example in which the user swipes on the display surface of the display unit 700 to change the viewing direction has been described, but another operation may be adopted instead of the swipe operation. For example, the control unit 500 detects the operation of a pointing device such as a trackball or a joystick, detects the movement of the line of sight of the user observing the display unit 700, or tilts the display surface of the display unit 700 (for example). It may be configured to detect the tilt (in the left-right direction) and move the viewing direction in the detected direction.

(Modification example 4)
The display of the image of the object whose viewing direction has been changed is not limited to the display unit 700 of the image display device 100, and may be performed by an external image display device or a display unit of a computer. When the display unit of the computer is made to perform the processing, by causing the computer to execute the program that executes the processing shown in the flowchart of FIG. 13, the same processing as that of displaying on the display unit 700 of the image display device 100 can be performed. ..
The program can be provided to an external image display device or a computer via a storage medium 900 or the like in which the program is recorded in advance. Further, the program may be provided via a network line or the like and stored in an external image display device, a memory of a computer, or the like. Further, the program can also be supplied as a computer-readable computer program product in various forms such as a data signal (carrier wave).

Further, a part of the processing of the flowchart of FIG. 13 may be performed by the imaging device, and the remaining processing may be performed by an external computer. For example, an imaging device captures a subject including an object, generates images viewed from a plurality of directions, and transmits the images to a computer. The computer generates an image for initial display based on the received image and displays it on the display screen of the computer. At this time, icons (marks 81 and 82 such as FIGS. 9A and 9B) indicating directions in which the viewing direction can be changed are also displayed. The user performs, for example, a swipe operation on the initial display screen displayed on the monitor screen connected to the computer based on the icon. The computer detects the swipe operation amount, determines an arbitrary direction, and generates an image viewed from an arbitrary direction. In this example, the computer functions as an image generator that generates an image viewed from an arbitrary direction based on an image viewed from a plurality of directions. Further, the image generation device and the monitor are image display devices viewed from an arbitrary direction.

That is, by connecting an image pickup device including the components of the image display device of the embodiment described with reference to FIG. 4 excluding the display unit 700 and the operation member 800 to a computer, an image viewed from a plurality of directions can be arbitrarily displayed. In the case of an image display device that converts an image viewed from the direction of the above, it is possible to appropriately share the processing executed by both parties.

The computer implements an image generation program for generating an image viewed from a virtually set arbitrary direction based on captured images captured from a plurality of directions. The image generation program calculates the distance to the subject in the overlapping areas based on the images of the overlapping areas of each of the plurality of captured images, and the plurality of captured images, the distances, and any of them. It is an image generation program for causing a computer to generate an image of an object viewed from an arbitrary direction based on a direction.

When the display unit is used as a monitor of a computer connected externally, the image display device of FIG. 4 is an image generation device that generates an image viewed from an arbitrary direction based on a plurality of captured images viewed from two or more directions. .. This image generator includes a distance calculation unit that calculates the distance to the subject in the overlapping areas based on the images of the overlapping areas of each of the plurality of captured images, and the plurality of captured images, distances, and the like. And a generation unit that generates an image of an object viewed from an arbitrary direction based on an arbitrary direction.

(Modification 5)
In the above-described embodiment, as an initial display before the viewing direction is set by the setting unit 620, a display image generated based on the data of the second image captured by the second imaging unit 300 (FIG. 10 (a). )), Or a display image (FIG. 10 (b)) generated based on the data of the first image captured by the first imaging unit 200 is displayed on the display unit 700. Instead, as illustrated in FIG. 14A, an object viewed from the first direction or an image viewed from the second direction is displayed on the display unit 700, and the viewing direction is changed. The image in the area where the image can be generated may be moved and displayed like a moving image.
That is, although there are three box-shaped objects selected on the menu screen among the subjects photographed by the imaging device, the user can view an image of only one of the objects P2 from an arbitrary direction. You may want to get it. Modification 5 is a modification of the image presented to the user at that time.

In the initial display of the modified example 5, in FIG. 14A, only the region of the object P2a in the overlapping range DUP where the first image and the second image overlap among the images displayed on the display unit 700. However, without swiping, the image seen from the first direction → the image seen from the direction located between the first direction and the second direction → the image seen from the second direction → the first direction and the first An image viewed from a direction located between the two directions → an image viewed from the first direction → an image viewed from a direction located between the first direction and the second direction ... (repeated thereafter) automatically Change.
According to the modification 5, it is possible to generate an image in which the viewing direction is changed in the area of the object P2a displayed on the display unit 700 as the initial display before the viewing direction is set (changed) by the setting unit 620. It is possible to visually inform the user of the fact.

(Modification 6)
Further, instead of the modified example 5, as illustrated in FIG. 14B, only the image of the region including the object for which the image viewed from an arbitrary direction is to be obtained among the images viewed from the second direction. May be configured to be displayed on the display unit 700 as a still image.
That is, although there are three box-shaped objects selected on the menu screen among the subjects photographed by the image pickup apparatus 100, the user sees only one of the objects P2 from an arbitrary direction. You may want to get. Modification 6 is a modification of the image presented to the user at that time.

In the initial display of the modified example 6, in FIG. 14B, only the region of the object P2a in the overlapping range DUP where the first image and the second image overlap among the images viewed from the second direction is displayed. It is displayed in part 700. The area of the object P2a displayed as the initial display on the display unit 700 is an area in which an image in which the viewing direction is changed can be displayed. Images of objects P1a and P3a that do not require images viewed from any direction are not displayed.
According to the modification 6, it is possible to generate an image in which the viewing direction is changed in the area of the object P2a displayed on the display unit 700 as the initial display before the viewing direction is set (changed) by the setting unit 620. This can be visually informed to the user.
As an initial display before the viewing direction is set (changed) by the setting unit 620, the display unit 700 may display the area of the object P2b in the image viewed from the first direction.

(Modification 7)
In the above description, an image display including two imaging units (first imaging unit 200 and second imaging unit 300) configured to be rotatable around an axis AX in the direction perpendicular to the ground (floor surface). The device 100 has been illustrated. The number of imaging units is not limited to two, and may be four or six.
For example, in addition to two imaging units (first imaging unit 200 and second imaging unit 300), two additional imaging units (third imaging unit and fourth imaging unit) are placed around the axis AX, respectively. It can be configured to be rotatable.
Further, two imaging units (first imaging unit 200 and second imaging unit 300) or four imaging units (first imaging unit 200, second imaging unit 300, third imaging unit, third imaging unit, first In addition to the imaging unit (4), two additional imaging units (fifth imaging unit and sixth imaging unit) may be configured to be rotatable around an axis orthogonal to the axis AX.
Increasing the number of imaging units has the advantage of increasing the accuracy of generating an image viewed from an intermediate direction between the first direction and the second direction.

Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Aspects in which the configurations shown in the embodiments and modifications are used in combination are also included in the scope of the present invention. Other aspects conceivable within the scope of the technical idea of the present invention are also included within the scope of the present invention.

100 ... Imaging device, 200 ... First imaging unit, 210, 310 ... Fisheye lens, 220, 320 ... Image sensor, 400 ... Connection unit, 500 ... Control unit, 600 ... Display control unit, 610 ... Target determination unit, 620 ... Setting unit, 630 ... Generation unit, 640 ... Synthesis unit, 650 ... Distance calculation unit, 700 ... Display unit, 800 ... Operation member, AX, GX, Q ... Axis, P, P1, P2, P3 ... Object

Claims (17)

Two imaging units that are connected by a connecting unit, image subjects from different directions, and change the overlapping range of the shooting range.
An image that generates an image of the subject viewed from a direction different from the plurality of directions based on an image of the overlapping range of a plurality of captured images obtained by imaging the subject from a plurality of directions by the imaging unit. Generator and
An imaging device including a display unit that displays an image of the subject as viewed from different directions. In the imaging device according to claim 1,
The two imaging units are imaging devices in which the overlapping range of the imaging ranges of each other is changed by rotating around the connection portion. In the imaging apparatus according to claim 1 or 2,
An imaging device including a setting unit for setting different directions. In the imaging device according to claim 3,
An imaging device including a calculation unit that calculates a direction in which different directions can be set based on the plurality of directions. In the imaging device according to claim 4,
An imaging device having a notification unit that notifies a direction in which the different directions can be set, which is calculated by the calculation unit. In the imaging device according to claim 5,
The notification unit is an imaging device that displays the calculated direction on the display unit. In the imaging apparatus according to any one of claims 1 to 6,
The image generation unit is an imaging device that changes the direction of an image of the subject displayed on the display unit when the different directions change. A distance calculation unit that calculates the distance to the subject based on a plurality of captured images obtained by imaging the subject from a plurality of different directions.
An image generation unit that generates an image of the subject viewed from a direction different from the plurality of directions based on the plurality of captured images and the distance.
An image display device including a display control unit for displaying the image on the display unit. In the image display device according to claim 8,
A setting unit for setting the different directions is provided.
The image generation unit is an image display device set by the setting unit that generates a display image of the subject viewed from the different direction. In the image display device according to claim 8 or 9.
An image display device including a calculation unit that calculates a direction in which different directions can be set based on the plurality of directions. In the image display device according to claim 10,
An image display device having a notification unit that notifies a direction in which the different directions can be set, which is calculated by the calculation unit. In the image display device according to claim 11,
The notification unit is an image display device that displays the calculated direction on the display unit. In the image display device according to any one of claims 8 to 12,
The distance calculation unit is an image display device that calculates the distance by performing stereo matching processing between images captured from the plurality of directions as the plurality of captured images. In the image display device according to claim 13,
The image generation unit is an image display device that increases the change per unit movement amount in the different directions in the image displayed on the display unit as the subject is closer. In the image display device according to any one of claims 8 to 14.
A region recognition unit that recognizes a region of a main subject or a region of a predetermined shape including the main subject based on the plurality of captured images is provided.
The image generation unit is an image display device that generates an image viewed from a different direction with respect to a region recognized by the region recognition unit. An image generator that generates an image viewed from a set direction based on a plurality of captured images viewed from two or more directions.
A distance calculation unit that calculates the distance to the subject in the overlapping areas based on the images of the overlapping areas of each of the plurality of captured images.
An image generation device including an image generation unit that generates an image of an object viewed from the set direction based on the plurality of captured images, the distance, and the set direction. An image generation program for generating an image viewed from a set direction based on a plurality of captured images viewed from two or more directions.
A process of calculating the distance to the subject in the overlapping areas based on the images of the overlapping areas of each of the plurality of captured images, and
An image generation program for causing a computer to perform a process of generating an image of an object viewed from the set direction based on the plurality of captured images, the distance, and the set direction.

Images