JP2019161391A - Display device and head-mounted display - Google Patents

Abstract

To provide a display device and a head-mounted display capable of easily grasping the position of an object to be photographed, and easily securing the safety of a user even when the user photographs the object to be photographed while moving.SOLUTION: A CPU acquires a first image (S11), and acquires a second image (S12), and acquires a first direction (S13), and acquires a second direction (S15), and calculates a first relative angle made by the first direction and the second direction (S16), and determines a range of a display image to be displayed at an HD in the second image on the basis of the relative angle (S18, S23 or S26), and segments an image equivalent to the determined display range from the second image to generate the display image (S31), and causes the HD to display the display image (S35).SELECTED DRAWING: Figure 7

Description

  The present invention relates to a display device and a head mounted display.

  A display device that displays each of images captured by a plurality of imaging optical systems is known. Patent Document 1 discloses a first image captured by a first imaging optical system that is an imaging optical system having a narrow angle of view, and a second imaging optical system that is an imaging optical system having a wider angle of view than the first imaging optical system. Disclosed is a display device that processes a captured second image and displays an image indicating the first image and the second image. Even if the photographer who is a camera user misses the subject from the angle of view of the first imaging system due to camera tilt, subject movement, etc., the subject is captured at the angle of view of the second imaging system. For example, by confirming the position of the subject in the second image, the subject can be captured again at the angle of view of the first imaging system.

JP 2012-147082 A

  The camera of Patent Document 1 assumes that a subject is imaged from a fixed position. For example, a user may take a picture of a subject while moving by using a camera or the like installed so as to be rotatable with respect to the gimbal mechanism. When the direction connecting the imaging optical system of the camera and the subject is different from the direction in which the user moves, the image displayed on the display device cannot confirm what the user is in the moving direction, and the moving user May have dangerous eyes. If the first image and the second image are displayed on a display device such as a head mounted display (HMD) worn by the user, the user secures both a captured image by the camera and a field of view in the direction in which the user moves. it can. However, since the positions of the user and the subject can move, the user may miss the subject from the angle of view of both the first imaging system and the second imaging system. In this case, there is a problem that it is difficult to capture the subject again at the angle of view of the first imaging system.

  An object of the present invention is to provide a display device and a head-mounted display that can easily grasp the position of a shooting target and easily secure the safety of the user even when the user is shooting a shooting target while moving. And

  The display device according to the first aspect of the present invention includes a display unit that displays an image, a first image acquisition unit that acquires a first image captured by the first camera, and a wider angle than the angle of view of the first camera. Second image acquisition means for acquiring a second image captured by a second camera that captures the first image, and first acquisition means for acquiring a first direction indicating a direction in which the first camera captures the first image A second acquisition unit that acquires a second direction indicating a direction in which the user's visual field is directed; a relative angle calculation unit that calculates a relative angle that is an angle formed by the first direction and the second direction; A determining unit that determines a display range that is a range to be displayed on the display unit in the second image based on the relative angle calculated by the relative angle calculating unit as a range including the first image; Before decided by decision means Comprising generation means for generating a display image is an image corresponding to the display range, and display control means for displaying the display image generated by the generating unit on the display unit.

  In the display device according to the first aspect of the present invention, the display range to be displayed on the display unit in the second image based on the relative angle formed by the first direction and the second direction is a range including the first image. decide. For this reason, even if the first direction and the second direction change, the display device can determine a display range corresponding to the change and display a display image corresponding to the display range. Therefore, by checking the display image, the user can easily capture the shooting target in the first image even if the first direction and the second direction change. Therefore, the display device according to the present invention can easily grasp the position of the shooting target even when the user is shooting the shooting target while moving, and can easily ensure the safety of the user.

  A head mounted display according to a second aspect of the present invention is provided with the display device, a mounting portion that is a member for mounting the display device on the head of the user, and the second direction. The second acquisition unit acquires the second direction detected by the detection unit, and the generation unit includes the field of view of the user based on the second direction. A display image is generated.

  The head mounted display which concerns on the 2nd aspect of this invention mounts | wears with the display apparatus which concerns on a 1st aspect to a user's head by a mounting part. Since the detection unit for detecting the second direction is provided in the mounting unit, the detection unit can accurately detect the second direction according to the direction in which the user's head is facing, or the like. The generation unit generates a display image including the user's visual field based on the second direction. Therefore, the head-mounted display can easily allow the user to recognize the position of the imaging target in the display device according to the user's visual field.

It is a perspective view of HMD1. 3 is an enlarged view of a first ball joint 6 and a second ball joint 7. FIG. It is a figure which shows HMD1 of the state by which HD2 is arrange | positioned in the 1st position or the 2nd position. It is a figure which shows HMD1 of the state by which HD2 is arrange | positioned in the 3rd position. It is a figure which shows HMD1 of the state which a 2nd direction is a downward direction. It is a block diagram which shows the electric constitution of HMD1. It is a flowchart of a main process. It is a flowchart of a main process. It is a figure showing the state where user U who wore HMD1 with left eye wearing was seen from the upper part. It is a figure showing the state where user U who wore HMD1 with left eye wearing was seen from the upper part. It is a figure which shows the part V1 equivalent to the 1st image in the excerpt scenery 521, and the part V2 equivalent to a display range. It is a figure which shows the example of the display image displayed on HD2. It is a figure which shows the state which looked at the user U who wears HMD1 by left eye wearing or right eye wearing from the upper side. It is a figure which shows the part V1 equivalent to the 1st image in the excerpt scenery 521, and the part V2 equivalent to a display range. It is a figure which shows the example of the display image displayed on HD2.

  An embodiment of the present invention will be described. As shown in FIG. 1, a head mounted display (hereinafter referred to as “HMD”) 1 is a video transmission type HMD. The HMD 1 includes a wearing tool 10, a mounting part 11, and a display device (hereinafter referred to as “HD”) 2. Hereinafter, in order to help understand the description of the figure, the upper side, lower side, left side, right side, front side, and rear side of the HMD 1 are defined. The upper side, the lower side, the left side, the right side, the front side, and the rear side of the HMD 1 correspond to, for example, the upper side, the lower side, the upper left side, the lower right side, the left side, and the right side in FIG. The upper side, the lower side, the left side, the right side, the front side, and the rear side of the HMD 1 are respectively the upper side, the lower side, the right side, the left side, the front side, and the user U wearing the wearing tool 10 (see FIG. 9). Corresponds to the rear side.

  As shown in FIG. 1, the wearing tool 10 is made of a flexible material such as resin or metal (for example, stainless steel). The wearing tool 10 includes a first portion 10A and second portions 10B and 10C. In addition, below, in order to make an understanding easy, although the wearing tool 10 is divided into 1st part 10A and 2nd part 10B, 10C and is demonstrated, the wearing tool 10 is 1st part 10A and 2nd part 10B, 10C. It is not divided into each member of these, but is an integral member as a whole.

  Each of the first portion 10A and the second portions 10B and 10C is a curved and elongated plate-like member. The first portion 10 </ b> A extends in the left-right direction between the position 102 and the position 103 in the wearing tool 10. The first portion 10A is curved in a convex shape on the front side. The position 102 is located on the left side of the center 101 in the left-right direction of the wearing tool 10. The position 103 is located on the right side of the center 101 in the left-right direction of the wearing tool 10. The second portion 10 </ b> B extends rearward from the position 102 in the wearing tool 10. The second portion 10 </ b> C extends rearward from the position 103 in the wearing tool 10. The second portions 10B and 10C each extend in a direction in which the rear ends approach each other. The wearing tool 10 is worn on the head of the user U with the first part 10A and the second parts 10B and 10C in contact with the frontal, left and right heads of the user U, respectively. The The mounting portion 11 is fixed at the position 103 of the wearing tool 10. Hereinafter, the side surrounded by the first portion 10A and the second portions 10B and 10C in the wearing tool 10 is referred to as “inner side”, and the side opposite to the inner side is referred to as “outer side”.

  The mounting portion 11 includes an arm portion 12, a fixing portion 14, a first ball joint 6, and a second ball joint 7. The fixing | fixed part 14 is fixed to the position 103 of the wearing tool 10 with the screw | thread 14A (refer FIG. 2 (A)). The arm part 12 is substantially rod-shaped. The arm portion 12 is made of resin, metal, or the like. The arm part 12 extends in the vertical direction as viewed from the front. The upper end portion of the arm portion 12 is connected to the fixed portion 14 via the first ball joint 6. The lower end portion of the arm portion 12 is connected to the HD 2 via the second ball joint 7. The arm part 12 connects the fixing part 14 and the HD 2. The arm part 12 and the fixing part 14 can hold the HD 2 on the front side of the left eye 8 of the user U in a state where the wearing tool 10 is worn on the head of the user U. The first ball joint 6 and the second ball joint 7 can adjust the position of HD2 with respect to the left eye 8 of the user U (see FIGS. 2 and 3).

  As shown in FIG. 2A, the first ball joint 6 includes a ball stud 61 and a socket 62. The ball stud 61 has a rod portion 61A and a sphere portion 61B. The spherical body portion 61B is a spherical portion provided at one end of the rod portion 61A. The socket 62 supports the sphere 61B so as to be slidable. In the following description, the direction orthogonal to the outer peripheral surface 100 of the wearing tool 10 in the position where the fixing | fixed part 14 and the socket 62 are arrange | positioned among the wearing tools 10 is called "orthogonal direction." The socket 62 has a lid portion 621 and a receiving portion 622. The receiving part 622 has a cylindrical shape and extends outward from the fixed part 14 along the orthogonal direction. The spherical portion 61B is accommodated inside the receiving portion 622. The receiving portion 622 has a thread on the outer surface. The lid 621 has a thread on the inner surface. The lid portion 621 is screwed into the receiving portion 622. The lid 621 has a circular hole 621A. The rod portion 61A of the ball stud 61 extends from the inside of the receiving portion 622 toward the hole portion 621A and passes through the hole portion 621A. The rod portion 61A extends outward along the orthogonal direction from the sphere portion 61B, bends rightward, further extends along the left-right direction, and is supported by the arm portion 12. The bending angle θ1 of the rod 61A is about 30 degrees.

  As the spherical body portion 61B slides with respect to the socket 62, the first ball joint 6 connects the fixed portion 14 and the arm portion 12 in a rotatable manner. The movable range of the arm portion 12 with respect to the fixed portion 14 is limited by the contact of the rod portion 61A with the hole portion 621A.

  As shown in FIG. 2B, the second ball joint 7 includes a ball stud 71 and a socket 72. The ball stud 71 has a rod portion 71A and a sphere portion 71B. The spherical portion 71B is a spherical portion provided at one end of the rod portion 71A. The socket 72 is provided in the HD2. The socket 72 supports the sphere 71B so as to be slidable. The socket 72 has a lid portion 721 and a receiving portion 722. The receiving part 722 is cylindrical and extends from the HD2. The spherical portion 71B is accommodated in the receiving portion 722. The receiving portion 722 has a thread on the outer surface. The lid 721 has a thread on the inner surface. The lid part 721 is screwed into the receiving part 722. The lid 721 has a circular hole 721A. The rod portion 71A of the ball stud 71 extends from the inside of the receiving portion 722 toward the hole portion 721A and passes through the hole portion 721A. The other end portion of the rod portion 71A is supported by the arm portion 12. The rod portion 71A extends from the arm portion 12 toward the left side, bends in a direction away from the wearing tool 10, and further extends, and is supported by the HD 2 via the sphere portion 71B and the socket 62. The bending angle θ2 of the rod portion 71A is about 30 degrees.

  As the spherical portion 71B slides with respect to the socket 72, the second ball joint 7 rotatably connects the HD 2 and the arm portion 12. The movable range of HD2 with respect to the arm portion 12 is limited by the rod portion 71A coming into contact with the hole portion 721A.

  As shown in FIG. 1, the HD 2 includes a housing 21. The casing 21 has a hollow box shape. A rectangular opening is provided at the rear end of the housing 21. The opening is covered with a transparent plate member. The liquid crystal panel 2 </ b> A is accommodated in the housing 21. The liquid crystal panel 2A displays a color image by controlling the output timing of each of red (R), green (G), and blue (B) light. The liquid crystal panel 2A displays an image based on the image signal received from the first camera 51 and the second camera 52 (see FIG. 6) connected via the cable 2B (see FIG. 1) in the display area. The light of the image displayed on the display area of the liquid crystal panel 2 </ b> A is emitted toward the opening of the housing 21. The light of the image passes through the opening of the housing 21 and further passes through the transparent plate member to the rear side. The image light is emitted from the housing 21 to the outside. Hereinafter, as shown in FIG. 2B, the display area of the liquid crystal panel 2 </ b> A, in other words, the portion of the liquid crystal panel 2 </ b> A from which image light is emitted is referred to as an “emission part 22”.

  In this embodiment, the 1st camera 51 (refer FIG. 6) is a camera for the user U to image a imaging | photography object with a still image or a moving image. Hereinafter, an image captured by the first camera 51 is referred to as a “first image”. The first camera 51 is electrically connected to the HMD 1. The HMD 1 is electrically connected to the first camera 51 by wire or wirelessly, and is used by the user U while being worn by the user U who operates the first camera 51.

  The first camera 51 outputs a first direction signal to the connected HMD 1. The first direction signal is a signal indicating the first direction, which is the direction in which the first camera 51 captures the first image. The first camera 51 captures a first image at a first field angle (θc1, see FIGS. 9, 10, and 13) that is a general shooting field angle. In the present embodiment, the first angle of view is 50 °. The first direction signal is generated based on the first direction acquired by a first direction detection unit (not shown) of the first camera 51. The first direction detection unit is, for example, an acceleration sensor that can detect respective accelerations in the three axial directions (X-axis direction, Y-axis direction, and Z-axis direction) that act on the first camera 51.

  The second camera 52 (see FIG. 6) is a camera that has an ultra-wide-angle lens such as a fisheye lens and can capture an image having a wider angle than the first angle of view. In the present embodiment, the second camera 52 is a so-called 360 ° camera that can capture an image of its own 360 ° orientation. The second camera 52 only needs to be able to capture an image having a wider angle than the first field angle, and does not have to be a 360 ° camera. Hereinafter, an image captured by the second camera 52 is referred to as a “second image”. The second camera 52 is electrically connected to the HMD 1. The second camera 52 is provided in order to make it easy for the user U to capture an image of the imaging target at the first angle of view. In this embodiment, the 2nd camera 52 is installed in the position of 10 A of 1st parts of the wearing tool 10 (refer FIG. 1). In addition to this, the second camera 52 may be installed at an arbitrary position such as the first camera 51, a gimbal mechanism in which the first camera 51 is installed, a hat worn by the user U, a helmet, and clothes. As the second camera 52, a plurality of directions (for example, directions from the respective positions of the first portion 10 </ b> A, the second portions 10 </ b> B and 10 </ b> C of the wearing tool 10 to the outside) are imaged. You may make it have the same function as this 360 degree camera.

  The user U fixes the wearing tool 10 to the head. The user U has the HD 2, and as shown in FIG. 3 (A), the HD 2 is arranged such that any position of the light emitting part 22 of the liquid crystal panel 2 A is arranged to face the front side of the left eye 8 of the user U. Adjust the position. At this time, since the first ball joint 6 and the second ball joint 7 freely rotate, the user U can easily move the HD 2. Hereinafter, a region including HD2 in a case where any position of the emitting unit 22 faces the front side of the left eye 8 of the user U is referred to as a “central region”.

  The center C11 of the spherical part 61B (refer FIG. 2 (A)) of the 1st ball joint 6 is defined among the fixing | fixed parts 14. FIG. The center C12 of the emission part 22 of HD2 is defined. The position of HD2 when the center C12 of the emitting part 22 faces the front side of the left eye 8 of the user U (see FIG. 3A) is referred to as “first position”. The first position is included in the central region. The distance between the centers C11 and C12 when the HD2 is disposed at the first position is referred to as a “first distance L1”.

  The liquid crystal panel 2A displays an image on the emission unit 22 based on the image signal received via the cable 2B (see FIG. 1). The light of the image is emitted to the rear side through the opening of the housing 21. When the HD 2 is arranged in the central region, the emitted image light is incident on the left eye 8 of the user U. Thereby, the user U recognizes the image.

  The user U may use the HMD 1 by arranging the HD 2 so that both the image displayed on the liquid crystal panel 2A of the HD 2 and the scenery in front of the user can be visually recognized. In this case, the user U first rotates the first ball joint 6 and the second ball joint 7 clockwise from the state in which the HD 2 is disposed at the first position (see FIG. 3A). It is rotated within the range of one predetermined angle or more and less than the second predetermined angle (see FIG. 3B). That is, for each of the first ball joint 6 and the second ball joint 7, the user U makes the first ball joint so that the rotation angle about the vertical direction is not less than the first predetermined angle and less than the second predetermined angle. 6 and the second ball joint 7 are rotated. As a result, the user U moves the HD 2 until the emission part 22 is not disposed opposite to the front side of the left eye 8. As shown in FIG. 3B, the HD 2 moves in a direction away from the wearing tool 10. The distance between the centers C11 and C12 changes to a second distance L2 that is longer than the first distance L1.

  When at least one of the first ball joint 6 and the second ball joint 7 is rotated and moved in the range from the first predetermined angle to the second predetermined angle with respect to the HD 2 arranged at the first position. The position of HD2 (see FIG. 3B) is referred to as “second position”. When HD2 is arranged at the second position, the distance between the centers C11 and C12 is the second distance L2. Of the areas excluding the central area, a predetermined area including the HD 2 arranged at the second position is referred to as a “first peripheral area”. When HD2 is arrange | positioned in the 2nd position, the user U can visually recognize the scenery in front by directing a eyes | visual_axis to the front side. In addition, the user U can visually recognize the image displayed on the emission unit 22 of the liquid crystal panel 2A by shifting the direction of the line of sight to the right side of the HMD 1 (left side as viewed from the user U).

  In this embodiment, the rotation angle when moving the HD 2 from the first position in FIG. 3A to the second position in FIG. 3B is, for example, any of the range of 10 ° to 45 °. This value may be used. In this case, the first predetermined angle that is the threshold of the rotation angle is 10 °, and the second predetermined angle is 45 °. In the example shown in FIG. 3, the first ball joint 6 rotates θ12 (= about 25 °) about the vertical direction, and the second ball joint 7 rotates θ22 (= about 25 °) about the vertical direction. As a result, the HD 2 has moved from the second position (see FIG. 3A) to the second position (see FIG. 3B). 3A and 3B, the alternate long and short dash line T11 indicates a reference line of the rotation angle of the first ball joint 6 in a state where the HD2 is disposed at the first position. An alternate long and short dash line T21 in FIG. 3B indicates a reference line of the angle of the first ball joint 6 in a state where HD2 is disposed at the second position. A dashed line T12 in FIGS. 3A and 3B indicates a reference line of the rotation angle of the second ball joint 7 in a state where the HD2 is disposed at the first position. An alternate long and short dash line T22 in FIG. 3B indicates a reference line of the angle of the second ball joint 7 in a state where HD2 is disposed at the second position. For example, when the user U wearing the HMD 1 rotates the first ball joint 6 and the second ball joint 7 to place the HD 2, the HD 2 is within the range of the user U's field of view. It may be within the included region.

  2B, the rod portion 71A of the ball stud 71 of the second ball joint 7 extends from the arm portion 12, bends away from the wearing tool 10, and is supported by the socket 72. Connected to the spherical portion 71B. For this reason, as for the position of HD2, the movable range of HD2 in the direction away from the face becomes wider and the movable range of HD2 in the direction closer to the face becomes narrower than in the case where the rod portion 71A is not bent. Therefore, the user U can easily place the HD 2 in the second position by moving it away from the face.

  The user U may use the HD2 by placing it out of the field of view so that the scenery in front of him can be seen well. In this case, the user U rotates the first ball joint 6 and the second ball joint 7 clockwise from the state in which the HD 2 is disposed at the first position (see FIG. 3A). The second predetermined angle larger than the first predetermined angle is rotated (see FIG. 4). That is, the user U rotates the first ball joint 6 so that the rotation angle about the left and right direction of the first ball joint 6 is equal to or greater than the second predetermined angle. As a result, as shown in FIG. 4, the user U moves the HD 2 until the emission unit 22 is not disposed to face the front side of the left eye 8. The distance between the centers C11 and C12 is maintained at a third distance L3 that is substantially the same as the first distance L1.

  Hereinafter, at least one of the first ball joint 6 and the second ball joint 7 is rotated and moved with respect to the HD 2 arranged at the first position by a second predetermined angle larger than the first predetermined angle. The position of HD2 in this case (see FIG. 4) is referred to as “third position”. When HD2 is arranged at the third position, the distance between the centers C11 and C12 is the third distance L3. Among the areas excluding the central area and the first peripheral area, a predetermined area including the HD 2 arranged at the third position is referred to as a “second peripheral area”. When HD2 is disposed at the third position, the user U can visually recognize the scenery in front of the eyes while suppressing the HD2 from entering the field of view.

  In the present embodiment, the rotation angle when moving the HD 2 from the first position in FIG. 3A to the third position in FIG. 4 may be an angle larger than 45 ° as an example. In this case, the second predetermined angle that is the threshold value of the rotation angle is 45 °. In the example shown in FIG. 4, the first ball joint 6 rotates θ13 (= about 120 °) about the left-right direction, and the second ball joint 7 rotates θ23 (= about 90 °) about the left-right direction. As a result, the HD 2 has moved from the first position (see FIG. 3A) to the third position (see FIG. 4). 4 indicates a reference line of the rotation angle of the first ball joint 6 in a state where the HD 2 is disposed at the first position. An alternate long and short dash line U31 indicates a reference line of the rotation angle of the first ball joint 6 in a state where HD2 is disposed at the third position. An alternate long and short dash line U12 indicates a reference line for the rotation angle of the second ball joint 7 in a state where HD2 is disposed at the first position. An alternate long and short dash line U32 indicates a reference line of the rotation angle of the second ball joint 7 in a state where HD2 is disposed at the third position. The third position is, for example, when the user U wearing the HMD 1 rotates the first ball joint 6 and the second ball joint 7 to place the HD 2 so that the HD 2 is out of the range of the user U's field of view. It may be within the included region.

  As shown in FIG. 2A, the rod portion 61A of the ball stud 61 of the first ball joint 6 extends from the spherical portion 61B along the orthogonal direction, bends rightward, and extends along the left-right direction. Supported by part 12. That is, the extending direction (left-right direction) of a part of the rod portion 61A coincides with the axial direction (left-right direction) when the first ball joint 6 rotates. For this reason, the first ball joint 6 is easier to rotate around the left-right direction as compared to the case where the rod portion 61A is not bent. Therefore, the user U can easily move the HD 2 from the first position to the third position by rotating the first ball joint 6.

  In addition, the user U fixes the wearing tool 10 to the head in a state where the vertical direction is opposite to the arrangement shown in FIGS. 3 (A) and 3 (B), and the first ball joint 6 and the second ball joint. By rotating 7 appropriately, the HD 2 can also be arranged to face the front side of the right eye 9 (see FIG. 3). The HMD 1 allows the user U to select which front side of the left eye 8 or the right eye 9 to place the HD 2 according to the ease of shooting using the first camera 51, the dominant eye of the user U, and the like. be able to. Hereinafter, the placement of HD2 on the front side of the left eye 8 of the user U is referred to as “left eye wearing”, and the placement of HD2 on the front side of the right eye 9 of the user U is referred to as “right eye wearing”.

  Thus, since the HMD 1 can hold the HD 2 on the front side of the left eye 8 or the right eye 9 of the user U, the user U can perform various directions while holding the HD 2 on the front side of the left eye 8 or the right eye 9. You can face your face. For example, as shown in FIG. 5, the user U can turn his / her face downward while wearing the HMD 1.

  With reference to FIG. 6, the electrical configuration of the wearing tool 10 and the HMD 1 will be described. The wearing tool 10 includes an acceleration sensor 41. In the present embodiment, the acceleration sensor 41 is provided inside the wearing tool 10 at the position of the first portion 10A (see FIG. 1). The acceleration sensor 41 is electrically connected to the HD2 FPGA 33 (described later). The acceleration sensor 41 detects the respective accelerations in the triaxial directions that act on the wearing tool 10. The acceleration sensor 41 outputs a signal indicating the detected acceleration to the FPGA 33.

  The HD 2 includes a CPU 31, a storage unit 32, a liquid crystal panel 2 </ b> A, FPGAs 33 to 37, and an acceleration sensor 38. The CPU 31 controls the entire HMD 1. The storage unit 32, the liquid crystal panel 2A, the speaker 2C, the vibration motor 2D, and the FPGAs 33 to 37 are electrically connected to the CPU 31. The acceleration sensor 38 is electrically connected to the FPGA 34. The signal line from which the first image signal is output from the first camera 51 is electrically connected to the FPGA 35 via the cable 2B (see FIG. 1). The first image signal is a signal indicating the first image captured by the first camera 51. The signal line from which the first direction signal is output from the first camera 51 is electrically connected to the FPGA 36 via the cable 2B. The signal line from which the second image signal is output from the second camera 52 is electrically connected to the FPGA 37 via the cable 2B. The second image signal is a signal indicating a second image captured by the second camera 52. In addition, HMD1 may have one FPGA which has all the functions of FPGA33-37 instead of FPGA33-37.

  The storage unit 32 stores a program for main processing (see FIGS. 7 and 8) executed by the CPU 31, various parameters, and the like. The liquid crystal panel 2A displays an image based on a signal output from the CPU 31.

  The FPGA 33 is a programmable logic device (PLD) having a function of detecting a second direction that is a direction in which the field of view of the user U is directed. The FPGA 33 specifies the acceleration acting on the wearing tool 10 based on the signal output from the acceleration sensor 41. The FPGA 33 detects the direction in which the first portion 10A of the wearing tool 10 faces based on the specified acceleration. Since the wearing tool 10 is worn on the head of the user U, the direction in which the first portion 10A of the wearing tool 10 faces corresponds to the second direction. The FPGA 33 outputs a signal indicating the detected second direction to the CPU 31.

  The acceleration sensor 38 is provided inside the housing 21 of the HD 2 and detects the respective accelerations in the triaxial directions that act on the HD 2. The acceleration sensor 38 outputs a signal indicating the detected acceleration to the FPGA 34. The FPGA 34 is a PLD having a function of detecting whether the HD 2 is attached to the left eye or the right eye. Hereinafter, whether the HD2 is attached to the left eye or the right eye is referred to as an HD attachment position. The FPGA 34 specifies the acceleration acting on the HD 2 based on the signal output from the acceleration sensor 38. The acceleration in the vertical direction acting on HD2 is reversed between the case where user U wears HD2 with left eye and the case where HD2 wears right eye. The FPGA 34 detects the HD mounting position based on the specified acceleration. The FPGA 33 outputs a signal indicating the detected HD mounting position to the CPU 31.

  The FPGAs 35 and 36 are interface elements that convert the format of the signal output from the first camera 51 into a format that can be detected by the CPU 31. The FPGA 35 converts the format of the first image signal output from the first camera 51 and outputs information indicating the converted first image to the CPU 31. The FPGA 36 converts the format of the first direction signal output from the first camera 51, and outputs a signal indicating the converted first direction to the CPU 31.

  The main process executed by the CPU 31 will be described with reference to FIGS. In the main process, when the output of the first image signal is started from the first camera 51 and the output of the second image signal is started from the second camera 52, the CPU 31 executes the program stored in the storage unit 32. Started by.

  As illustrated in FIG. 7, the CPU 31 obtains the first image by receiving information indicating the first image via the FPGA 35 (S11). The CPU 31 acquires the second image by receiving information indicating the second image via the FPGA 37 (S12). The CPU 31 acquires a first direction by receiving a signal indicating the first direction via the FPGA 36 (S13). CPU31 acquires a 2nd direction by receiving the signal which shows the 2nd direction output from the acceleration sensor 41 provided in the wearing tool 10 via FPGA33 (S15).

  CPU31 calculates the relative angle which is an angle which a 1st direction and a 2nd direction make (S16). Hereinafter, the relative angle calculated based on the first direction acquired in the process of S13 and the second direction acquired in the process of S15 is referred to as a first relative angle.

9, 10, and 13 show the user U who wears the HMD 1 and performs shooting using the first camera 51. It is assumed that the user U is taking a picture by installing the first camera 51 on a known gimbal mechanism (not shown) so as to be rotatable and operating the gimbal mechanism. Note that the user U may take a picture while holding the first camera 51 by hand. In the present embodiment, as shown in FIG. 9C, the reference direction of the angle indicating the first direction and the angle indicating the second direction (the direction indicated by the arrow A1, for example, the north direction) is 0 °. With the user U viewed from above, the counterclockwise direction (see arrow A2) is 0 ° to + 180 °, and the clockwise direction (see arrow A3) is 0 ° to −180 °. The same applies to FIGS. 10 and 13. Hereinafter, the angle in the first direction is θc2, and the angle in the second direction is θh. Hereinafter, an angle formed by the first direction and the second direction is referred to as a relative angle. If the relative angle is θc−h,
(Θc−h) = (θc2) − (θh)
It is calculated as a value obtained by the above calculation. In the state of FIG. 9C, it is assumed that θc2 is + 5 ° and θh is 0 °. In this case, θc−h, which is the first relative angle calculated by the process of S16 in the state of FIG. 9C, is + 5 °.

Based on the first relative angle, the CPU 31 determines a display range which is an image range to be displayed on the liquid crystal panel 2A of the HD2 among the second images (S18). Of the display range, the end on the counterclockwise direction (in the direction of arrow A2) with the position where the second camera 52 is provided on the wearing tool 10 as viewed from the upper side when the user U is viewed from above is defined as θ +, The end on the clockwise direction (arrow A3 direction) side is defined as θ−. In the present embodiment, a range in which each of the end on the arrow A2 direction side and the end on the arrow A3 direction side is expanded by the same angle as the first angle of view with respect to the first relative angle is determined as the display range. The The calculation performed for determining the display range is as follows.
(Θ +) = (θc−h) + (θc1)
(Θ −) = (θc−h) − (θc1)
Θ + derived by the above calculation in the state of FIG. 9C is + 55 °, and θ− is −45 °. Therefore, θk, which is an angle formed between θ + and θ−, is 100 °.

The CPU 31 determines whether the HD attachment position is right eye attachment (S21). This determination is made based on a signal indicating the HD mounting position received by the CPU 31 via the FPGA 34. When the HD attachment position is right eye attachment (S21: YES), the CPU 31 determines whether the display range determined in the process of S18 is within the range of the left eye visual field of the user U (S22). The left eye visual field is a horizontal viewing angle that the user U can visually recognize with the left eye 8 without moving the left eye 8. The horizontal viewing angle that the user U can visually recognize with the right eye 9 without moving the right eye 9 is referred to as a right eye visual field. In the present embodiment, each of the left eye field and the right eye field is a peripheral field by the left eye and the right eye. In the present embodiment, the left end of the left eye visual field (the end on the (arrow A2 direction) side) is θul +, and the left end of the left eye visual field (the end on the (arrow A3 direction) side) is θul +. -. Further, the left end (the end on the (arrow A2 direction) side) of the right eye visual field is θur +, and the left end (the end on the (arrow A3 direction) side) of the left eye visual field is θur−. (See FIGS. 9, 10 and 13). In this embodiment, based on the average horizontal viewing angle of humans,
(Θul +) = (θh) + 100 °
(Θul −) = (θh) −60 °
(Θur +) = (θh) + 60 °
(Θur −) = (θh) −100 °
It is said. In the state of FIG. 9C, since θh is 0 °, θul + is + 100 °, θul− is −60 °, θur + is + 60 °, and θur− is −100 °. As shown in FIGS. 9, 10, and 13, the left eye visual field is θul that is an angle formed by θul + and θul−. The right eye visual field is θur, which is an angle formed by θur + and θur−.

In S22, the CPU 31 displays the display range when θ +, which is the end of the display range in the direction of the arrow A2, and θul−, which is the end of the left eye visual field, in the direction of the arrow A3 are as follows. Is not within the range of the left eye visual field.
(Θ +) <(θul-)

CPU31 transfers a process to S31, when a display range exists in the range of the left eye visual field (S22: YES). When the display range is not within the range of the left eye visual field (S22: NO), the CPU 31 corrects the display range so that the display range includes the left eye visual field (S23). When the HD wearing position is the right eye wearing, a part of the right eye visual field of the user U is blocked by the HD 2, which may make it difficult for the user U to visually recognize the shooting target. In such a case, the CPU 31 corrects θ +, which is the end of the display range on the arrow A2 direction side, to be included in the range of θul indicating the left eye visual field. Thus, the CPU 31 can include the range of the left eye visual field of the user U in the display range. In the present embodiment, the CPU 31 corrects the display range so that the display range includes the left eye visual field by replacing θ + with the following θ + ′.
(Θ + ') = (θul-) + 10 °
Thereafter, the CPU 31 proceeds to S31.

On the other hand, when the HD wearing position is the left eye wearing (S21: NO), the CPU 31 determines whether the display range determined in the process of S18 is within the range of the right eye visual field of the user U (S25). In S25, the CPU 31 displays the display range when θ−, which is the end portion of the display range on the arrow A3 direction side, and θur +, which is the end portion of the right eye field on the arrow A3 direction side, have the following relationship. Is not within the range of the right eye field of view.
(Θ −)> (θur +)

When the display range is within the range of the right eye visual field (S25: YES), the CPU 31 shifts the process to S31. When the display range is not within the range of the right eye visual field (S25: NO), the CPU 31 corrects the display range so that the display range includes the right eye visual field (S26). When the HD wearing position is the left eye wearing, the user U's left eye visual field is blocked by the HD 2, which may make it difficult for the user U to visually recognize the shooting target. In such a case, the CPU 31 corrects θ−, which is the end of the display range on the arrow A3 direction side, to be included in the range of θur indicating the right eye visual field. In the present embodiment, the CPU 31 corrects the display range so that the display range includes the right visual field by replacing θ− with the following θ− ′.
(Θ − ′) = (θur +) − 10 °

  Thereafter, the CPU 31 proceeds to S31. The CPU 31 performs the processing from S21 to S26 so that the visual field of the user U is always included in the display range. Since the HD 2 displays a display image including the field of view of the user U, for example, even when the user U is shooting while moving, the user U can visually recognize the display image and prompt his / her safety confirmation.

  The CPU 31 cuts out an image corresponding to the display range by cutting out the image corresponding to the display range determined in the process of S18 or the display range corrected in the process of S23 or S26 from the second image. Is generated (S31). CPU31 detects the part corresponded to a 1st image from the display image produced | generated by the process of S31 (S32). CPU31 performs the process of S32 by performing the image process by the well-known pattern matching method using the 1st image acquired by the process of S11 as a template with respect to the display image produced | generated by the process of S31. .

  Based on the portion corresponding to the first image detected from the display image, the CPU 31 adds information indicating the position of the portion corresponding to the first image to the display image (S33). The CPU 31 displays a display image to which information indicating the position of the portion corresponding to the first image is added on the emission unit 22 of the liquid crystal panel 2A (S35). The CPU 31 proceeds to the determination of S41 (see FIG. 8).

  With reference to FIG. 9, FIG. 10, and FIG. 11, an example of a display image displayed on the emission unit 22 of the liquid crystal panel 2A by the processing so far will be described. As shown in FIG. 9C, the user U wears the HMD 1 with the left eye wearing, and performs shooting using the first camera 51 with the landscape shown in FIG. FIG. 11G shows an excerpt landscape 521 that is an excerpt of a landscape included in the horizontal viewing angle of the user U among the landscapes around the user U facing the north direction (0 °). The excerpt landscape 521 corresponds to a part of the second image captured by the 360 ° camera. A portion corresponding to the first image taken by the first camera 51 in the excerpt landscape 521 is referred to as a portion V1. The CPU 31 determines the display range in the range of the angle θk based on the first relative angle θc−h and the first angle of view θc−h (S18). Of the excerpt landscape 521, a portion corresponding to the determined display range is defined as a portion V2.

  In the state shown in FIG. 9C, θ− is −45 ° and θur + is + 60 °, so the relationship of (θ−) ≦ (θur +) is established. Therefore, the CPU 31 determines that the display range is within the range of the right eye visual field (S25: YES). For this reason, the display range is not corrected, and an image corresponding to the range of the part V2 is cut out from the second image acquired in the process of S12 based on the display range determined in the process of S18. An image is generated (S31). Let the generated display image be an image X1 (see FIG. 12K).

  The CPU 31 detects a portion corresponding to the portion V1 in FIG. 11G from the image X1 (S32). The CPU 31 adds a frame X2, which is information indicating a portion corresponding to the portion V1 in the display range, to the image X1 (S33). The frame X2 indicates the portion corresponding to the first image in the display image by surrounding the entire portion corresponding to the first image in the display image. The CPU 31 displays the image X1 added with the frame X2 shown in FIG. 12 (K) on the emission unit 22 of the liquid crystal panel 2A (S35).

  Returning to the description of FIG. CPU31 judges whether a 2nd direction exists in the specific range set regarding the visibility of the user U 1st direction (S41). For example, when the first direction points to the subject to be photographed substantially horizontally and the first camera 51 captures a first image corresponding to the portion V1 shown in FIG. The second direction, which is the direction, may indicate the downward direction (see FIG. 5). For example, this may be the case when the moving user U photographs while keeping the first angle of view of the first camera 51 substantially horizontal while confirming the current situation. In this case, the landscape in the first direction is difficult to be captured in the peripheral visual field of the user U. In such a case, if a landscape other than the shooting target is displayed to the user U on the emission unit 22 of the liquid crystal panel 2A, it may be difficult for the user U to visually recognize the shooting target. In such a case, it may not be necessary for the user U to visually recognize the scenery around the first image. In the present embodiment, with respect to the second direction, the visibility of the first direction is reduced in the peripheral visual field of the user U, and the scenery in the first direction becomes difficult to be captured. Set a direction within a specific range for visibility. Then, when the second direction is the direction of the specific range, the CPU 31 causes the emission unit 22 of the liquid crystal panel 2 </ b> A to display the first image that is the shooting target of the user U instead of the display image based on the display range. . Thereby, the emission part 22 of the liquid crystal panel 2 </ b> A improves the visibility of the shooting target of the user U and improves the usability of the HMD 1. In the present embodiment, the CPU 31 determines that the second direction is within a specific range set for the visibility of the user U in the first direction when the user U can identify that the user U is viewing the downward direction. It is judged that. When the CPU 31 determines that the second direction is within the specific range (S41: YES), the CPU 31 displays the first image on the emission unit 22 of the liquid crystal panel 2A instead of the display image (S42). In this case, since the first image is displayed as it is on the emission unit 22 of the liquid crystal panel 2A, the user U can confirm the situation at the feet and the like to maintain the safety of photographing and capture the first image that is difficult to capture in the peripheral visual field. Visible at the emission part 22 of the liquid crystal panel 2A. The CPU 31 returns the process to the determination in S41. CPU31 transfers a process to S43, when a 2nd direction is not in a specific range (S41: NO).

  In the shooting of the shooting target by the user U, the first direction and the second direction can change with time due to movement of the shooting target, change of the shooting target, movement of the user U, and the like. CPU31 acquires the 1st direction in the present time (S43). The CPU 31 determines whether or not the amount of change between the first direction acquired before the current time by the process of S13 and the first direction at the current time is greater than or equal to a predetermined amount (S44). In the present embodiment, as an example, the criterion for determining the amount of change in the first direction is 50 °, which is equal to the first angle of view θc1.

  When the amount of change in the first direction is less than the predetermined amount (S44: NO), the CPU 31 acquires the second direction at the current time (S45). The CPU 31 calculates a second relative angle formed by the current first direction acquired in S43 and the current second direction acquired in S45 (S46). The CPU 31 calculates a relative angle change amount that is a change amount between the value of the first relative angle calculated in the process of S16 (see FIG. 7) and the value of the second relative angle calculated in the process of S46. (S48).

  The CPU 31 determines whether the relative angle change amount is a predetermined amount or more (S51). In this embodiment, as an example, the criterion for determining the relative angle change amount is set to 25 °, which is a 50% angle of θc1 that is the first angle of view. When the relative angle change amount is less than the predetermined amount (S51: NO), the CPU 31 moves the position of the frame X2 in the display image according to the change amount in the first direction calculated in the process of S44 (S52). The CPU 31 returns the process to the determination in S41.

  On the other hand, when the amount of change in the first direction is greater than or equal to the predetermined amount (S44: YES), the CPU 31 returns the process to S11 (see FIG. 7). Note that the CPU 31 also returns the process to S11 when the relative angle change amount is equal to or greater than the predetermined amount (S51: YES).

  With reference to FIG. 10 to FIG. 12, an example of a display image displayed on the emission unit 22 of the liquid crystal panel 2A by the processing after S41 will be described. First, the case where the relationship between the first direction and the second direction has shifted from the state shown in FIG. 9C to the state shown in FIG. 10D will be described. In FIG. 10D, it is assumed that θc2 is + 5 ° and θh is 0 °. Since the amount of change between the first direction (−15 °) in FIG. 9C and the first direction (+ 5 °) in FIG. 10D is 20 °, the amount of change in the first direction Is less than a predetermined amount (S44: NO). In this case, based on the first direction and the second direction in the state of FIG. 10D, the second relative angle θc−h is calculated to be −15 ° (S46). Since the magnitude of the first relative angle is 5 ° and the magnitude of the second relative angle is 15 °, the relative angle change amount is calculated as 10 ° (S48). Since the relative angle change amount is less than the predetermined amount (S51: NO), only the position of the frame X2 in the display image is corrected according to the change amount in the first direction without changing the display range. (S52).

  In this case, as shown in FIG. 11 (H), the range of the portion V2 corresponding to the display range for the excerpt landscape 521 does not change from the state of FIG. 11 (G). In accordance with the change in the first direction between FIG. 9C and FIG. 10D, the portion V1 corresponding to the first image moves to the right from the portion V1 in FIG. Accordingly, as shown in FIG. 12K, the image X1 corresponding to the display image does not change compared to FIG. 12J, and only the position of the frame X2 moves to the right in the image X1.

  The case where the relationship between the first direction and the second direction has shifted from the state shown in FIG. 9C to the state shown in FIG. 10E through the state shown in FIG. 10D will be described. In FIG. 10E, it is assumed that θc2 is 0 ° and θh is + 20 °. Since the amount of change between the first direction (−15 °) in FIG. 9C and the first direction (+ 0 °) in FIG. 10D is 15 °, the amount of change in the first direction Is less than a predetermined amount (S44: NO). In this case, based on the first direction and the second direction in the state of FIG. 10E, θc-h, which is the second relative angle, is calculated as −20 ° (S46). Since the magnitude of the first relative angle calculated in the state shown in FIG. 9C is 5 ° and the magnitude of the second relative angle is 20 °, the relative angle change amount is calculated as 15 °. (S48). Since the relative angle change amount is less than the predetermined amount (S51: NO), only the position of the frame X2 in the display image is corrected according to the change amount in the first direction without changing the display range (S52). .

  In this case, the range of the portion V2 corresponding to the display range for the excerpt landscape 521 does not change from the states of FIGS. 11 (G) and 11 (H). In accordance with the change in the first direction between FIG. 10E and FIG. 10D, the portion V1 corresponding to the first image moves to the left than the portion V1 in FIG. Then, the state shown in FIG. That is, when the state shown in FIG. 9C is changed to the state shown in FIG. 10E through the state shown in FIG. 10D, the portion V2 corresponding to the display range in the excerpt landscape 521 and the first image are displayed. The portion V1 corresponding to is in the state of FIG. 11 (I) through FIG. 11 (G) through FIG. 11 (H). Accordingly, the image X1 of FIG. 12M is displayed in order on the emission unit 22 of the liquid crystal panel 2A through the image X1 of FIG. 12K to the image X1 of FIG.

  If the display image is updated each time the relative angle repeats a minute change, the display image of the emission unit 22 of the liquid crystal panel 2A appears to be slightly shaken, and the user U visually recognizes the display image. There is a possibility that the sex will decline. When the relative angle change amount is less than the predetermined amount, the CPU 31 does not update the display image in accordance with the change in the relative angle, and corrects only the position of the frame X2 in the display image, thereby improving the visibility of the display image. It is easy for the user U to capture the photographing object at the first angle of view while maintaining the above. When the relative angle change amount is equal to or greater than a predetermined amount (25 ° or more in this embodiment) (S51: YES), the display range and the display are based on the new first direction and the new second direction. In order to update the portion corresponding to the first image included in the range, the CPU 31 returns the process to S11. For example, when the amount of change in the first direction is small but the amount of change in the second direction is relatively large, the determination is made in S51. Accordingly, the CPU 31 can continuously display the display image corresponding to the change in the first direction and the second direction on the emission unit 22 of the liquid crystal panel 2A in a state including a portion corresponding to the first image. it can.

  A case where the relationship between the first direction and the second direction has shifted from the state shown in FIG. 9C to the state shown in FIG. 10F will be described. It is assumed that θc2, which is the angle in the first direction, changes from 0 ° to + 70 ° between FIG. 9C and FIG. In this embodiment, since the criterion for determining the amount of change in the first direction is 50 °, it is determined that the amount of change in the first direction is greater than or equal to a predetermined amount (S44: YES). In this case, the CPU 31 returns the process to S11 in order to update the display range and the portion corresponding to the first image included in the display range based on the new first direction. In this case, a portion V2 corresponding to the display range in the excerpt landscape 521 and a portion V1 corresponding to the first image are as shown in FIG. 11J, and an image corresponding to the range of the portion V2 is cut out from the second image. (S31). Accordingly, as shown in FIG. 12 (O), an image X1 with a frame X2 indicating a portion corresponding to the portion V1 in the display range is displayed on the emitting portion 22 of the liquid crystal panel 2A (S35).

  On the other hand, for example, the first direction changes greatly due to a large change in the position of the object to be photographed, but the change amount of the relative angle change amount may be very small. The user U maintains an angle between the direction in which his / her field of view (second direction) and the direction of the first camera 51 in his / her hand (first direction) are maintained (the posture holding the camera is substantially constant). This is the case when the image is looked back at a relatively large angle (for example, at an angle greater than or equal to the first angle of view). In such a case, it is necessary to re-determine the display range according to the change in the first direction regardless of the relative angle change amount. Therefore, CPU31 returns a process to S11, when a 1st direction changes a lot (S44: YES).

  With reference to FIGS. 13 to 15, an example of a display image displayed on the emission unit 22 of the liquid crystal panel 2 </ b> A when the display range is corrected by the process of S <b> 23 or S <b> 26 will be described. As shown in FIG. 13 (P), it is assumed that shooting by the user U is started in a state where θc2 which is the first direction is + 120 ° and θh which is the second direction is 0 °. In this state, the first relative angle (θc−h) is + 120 ° obtained by subtracting θh from θc2. Therefore, by the process of S18, θ + is determined to be + 170 ° and θ− is determined to be + 70 °.

  In FIG. 13 (P), since the user U wears the HMD1 (S21: NO), the process of S25 for determining whether the display range is within the range of the right eye visual field is performed. When θh is 0 °, θur + is + 60 °. Since θ− is + 70 °, the relationship (θ −)> (θur +) is established (S25: NO). Therefore, θ− is replaced with θ− ′ (= (θur +) − 10 °). Therefore, the end of the display range on the arrow A3 direction side is corrected from θ − (= + 70 °) to θ− ′ (= + 50 °) (S26). That is, the end of the display range on the arrow A3 direction side is expanded toward the arrow A3 direction as indicated by the arrow Y1. The corrected display range θk is 120 °.

  In this case, the range of the portion V2 corresponding to the display range for the excerpt landscape 521 is as shown in FIG. A two-dot chain line V21 in FIG. 14R indicates an end portion on the arrow A3 direction side of the display range before the display range is corrected by the process of S26. The end of the display range on the A3 direction side is corrected as indicated by the arrow Y1 by the process of S26. Therefore, as shown in FIG. 15 (T), an image corresponding to the range of the part V2 is cut out from the second image acquired in the process of S12 based on the display range determined in the process of S18. A display image is generated (S31). A portion corresponding to the portion V1 in FIG. 11R is detected from the image X1 corresponding to the display image (S32), and a frame X2 is added to the position corresponding to the detected portion (S33). The CPU 31 displays the image X1 added with the frame X2 shown in FIG. 15 (T) on the emission unit 22 of the liquid crystal panel 2A (S35).

  Further, as shown in FIG. 13Q, it is assumed that shooting by the user U is started in a state where θc2 which is the first direction is −120 ° and θh which is the second direction is 0 °. In this state, the first relative angle (θc−h) is −120 ° obtained by subtracting θh from θc2. Therefore, by the process of S18, θ + is determined to be −70 ° and θ− is determined to be −170 °.

  In FIG. 13M, since the user U wears the right eye of the HMD 1 (S21: YES), the process of S22 is performed to determine whether the display range is within the left eye visual field range. θul− when θh is 0 ° is −60 °. Since θ + is −70 °, the relationship (θ +) <(θul−) is established (S22: NO). Therefore, θ + is replaced with θ− ′ (= (θul −) + 10 °). Accordingly, the end of the display range on the arrow A2 direction side is corrected from θ + (= −70 °) to θ + ′ (= −50 °) (S23). That is, the end of the display range on the arrow A2 direction side is expanded toward the arrow A2 direction as indicated by the arrow Y1. The corrected display range θk is 120 °.

  In this case, the range of the portion V2 corresponding to the display range for the excerpt landscape 521 is as shown in FIG. A two-dot chain line V22 in FIG. 14S indicates an end portion on the arrow A2 direction side of the display range before the display range is corrected by the process of S23. The end of the display range on the A2 direction side is corrected as indicated by the arrow Y2 by the process of S23. Therefore, as shown in FIG. 15 (W), an image corresponding to the range of the part V2 is cut out from the second image acquired in the process of S12 based on the display range determined in the process of S18. A display image is generated (S31). A portion corresponding to the portion V1 in FIG. 11S is detected from the image X1 corresponding to the display image (S32), and a frame X2 is added to the position corresponding to the detected portion (S33). The CPU 31 displays the image X1 added with the frame X2 shown in FIG. 15 (W) on the emission unit 22 of the liquid crystal panel 2A (S35).

  As described above, when the first relative angle is relatively large and the user U cannot easily view the subject to be photographed because the left eye field or the right eye field is blocked by HD2, the CPU 31 displays the display range by the process of S23 or S26. Corrections are made to include U's right eye field or left eye field. Therefore, the user U can easily capture the photographing object accurately at the first angle of view by visually recognizing the display image including the landscape included in his field of view. In addition, since the user U can check the display image on the emission unit 22 of the liquid crystal panel 2A and can also check the situation in the second direction in which the user's field of view is facing, Easy to shoot.

  As described above, in the shooting of the shooting target by the user U, the first direction and the second direction indicate various directions due to movement of the shooting target, change of the shooting target, movement of the user U, and the like. Based on the relative angle formed by the first direction and the second direction, the CPU 31 determines a display range to be displayed on the emission unit 22 of the liquid crystal panel 2A in the second image as a range including the first image (S18). , S23 and S26). For this reason, even if the first direction and the second direction change, the CPU 31 determines a display range corresponding to the relative angle (S18), and displays a display image corresponding to the display range on the emission unit 22 of the liquid crystal panel 2A. (S35). Therefore, the user U can easily catch the shooting target in the first image even if the first direction and the second direction change according to shooting by visually recognizing the display image. Therefore, the HD 2 makes it easy for the user U to grasp the position of the shooting target and to ensure the safety of the user U even when the user U is shooting the shooting target while moving.

  The CPU 31 detects a portion corresponding to the first image included in the display image (S32). The CPU 31 attaches a frame X2 to the display image as information indicating the position of the portion corresponding to the detected first image (S33). Therefore, the HD 2 allows the user U to easily recognize the position of the portion corresponding to the first image in the display image by causing the user U to visually recognize the display image.

  The first direction and the second direction can change according to the progress of shooting using the first camera 51. When the second direction does not change and the first direction changes, the first image is not included in the display image depending on the degree of change in the first direction. When the first direction does not change and the second direction changes, the field of view of the user U changes according to the change in the second direction. When at least one of the first direction and the second direction changes and the relative angle change amount is equal to or greater than the predetermined amount (SS44: YES), the CPU 31 returns the process to S11 and based on the latest first relative angle. The display range is newly determined (S18). As a result, the HD 2 can update the display image when the first image is not included in the display image. Accordingly, the HD 2 can display a display image that allows the user U to recognize a portion corresponding to the first image even when the first direction and the second direction change according to the progress of shooting.

  For example, when the second direction is a specific direction that is neither the traveling direction of the user U or the direction from the user U toward the shooting target, the display image is not displayed on the HD 2 and the first image is displayed. If U is displayed as it is, the user U may easily recognize the shooting target. When the second direction is a direction within a specific range set for the visibility of the user U in the first direction (S41: YES), the CPU 31 outputs the first image instead of the display image to the liquid crystal panel 2A. It is displayed on the part 22 (S42). In this case, the user U can confirm the imaging target by visually recognizing the first image captured by the first camera 51. Therefore, the usability of HD2 is improved.

  The HMD 1 wears the HD 2 on the head of the user U by the wearing tool 10 and the wearing part 11. The acceleration sensor 41 is provided on the wearing tool 10. For this reason, CPU31 can acquire a 2nd direction correctly based on the signal output from the acceleration sensor 41 (S15). The CPU 31 determines a display range including the field of view of the user U based on the second direction (S18, S23, and S26), and generates a display image based on the determined display range (S31). Therefore, the HMD 1 can easily make the user U recognize the position of the shooting target in the HD 2 according to the user U's visual field.

  Depending on whether the HD attachment position is left eye attachment or right eye attachment, the left eye field or right eye field is likely to be blocked by HD 2, and it may be difficult for the user U to visually recognize the photographing target. The CPU 31 determines whether or not the display range determined by the process of S18 is included in the visual field of the user U according to the HD mounting position (S22 or S25). When the display range is not included in the visual field of the user U corresponding to the HD mounting position, the HMD 1 corrects the display range so that the visual field of the user U is effectively included in the display range (S23 or S26). Therefore, the HMD 1 can reduce the difficulty in visually recognizing the shooting target by the HD 2 by causing the HD 2 to display an appropriate display image corresponding to the HD mounting position on the HD 2.

  The emission unit 22 is an example of the “display unit” in the present invention. The CPU 31 that performs the process of S11 is an example of the “first image acquisition unit” in the present invention. The CPU 31 that performs the process of S12 is an example of the “second image acquisition unit” in the present invention. The CPU 31 that performs the process of S13 is an example of the “first acquisition means” in the present invention. The CPU 31 that performs the process of S15 is an example of the “second acquisition unit” in the present invention. The CPU 31 that performs the process of S16 is an example of the “relative angle calculation means” in the present invention. The CPU 31 that performs the processes of S18, S23, and S28 is an example of the “determination unit” in the present invention. The CPU 31 that performs the processes of S31 and S33 is an example of the “generating means” in the present invention. The CPU 31 that performs the process of S35 is an example of the “display control means” in the present invention. The CPU 31 that performs the processes of S43 and S45 is an example of the “third acquisition unit” in the present invention. The CPU 31 that performs the processing of S48 is an example of the “variation amount calculation means” in the present invention. The CPU 31 that performs the process of S51 is an example of the “change amount determination means” in the present invention. The wearing tool 10 and the mounting portion 11 are examples of the “mounting portion” in the present invention. The acceleration sensor 41 is an example of the “detection unit” in the present invention. The CPU 31 that performs the process of S21 is an example of the “mounting position determination unit” in the present invention.

  In addition, this invention is not limited to said embodiment, A various change is possible. CPU31 may judge that the 2nd direction exists in a specific range, and may perform processing of S41 and S42, when it exists in the range which can specify that the 2nd direction is viewing the up direction. The CPU 31 may determine that the second direction is within a specific range when the HD 2 is disposed at the third position. The CPU 31 calculates the difference between the acceleration detected by the acceleration sensor 41 and the acceleration detected by the acceleration sensor 38, thereby detecting the position of HD2 with respect to the wearing tool 10, thereby arranging HD2 at the third position. It may be determined whether or not.

  When the second direction is within the specific range, the CPU 31 calculates the first relative angle (S16), determines the display range (S18), corrects the display range according to the HD mounting position (S21 to S26), and the display image. It is good also as not performing at least any one of each process of display (S35). In this case, the processing load on the CPU 31 is reduced. In this case, the CPU 31 may display the first image as it is on the emission unit 22 of the liquid crystal panel 2A, or may not display an image on the emission unit 22 of the liquid crystal panel 2A.

  As the display range becomes wider by correcting the display range in S23 or S26, the size of the display image cut out from the second image becomes larger. When the first angle of view of the first camera 51 does not change, the size of the portion corresponding to the first image in the display image becomes relatively smaller as the size of the display image becomes larger. When the entire display image is displayed on the entire emission unit 22, the larger the size of the display image, the smaller the portion of the image corresponding to the first image is displayed on the emission unit 22. As the first image included in the display image becomes smaller, the user U who sees the display image becomes less likely to visually recognize the contents of the first image, and the first image is less likely to capture the shooting target. In order to avoid such inconvenience, the CPU 31 determines the display range according to the process of S18 and the display range corresponding to the HD mounting position by the processes of S21 to S26 when the display range becomes a predetermined range or more. It is good also as not performing correction. The CPU 31 may determine whether to perform the process of S18 and the processes of S21 to S26 according to the magnitude of the first relative angle. CPU31 may display the 1st image on the emission part 22 of liquid crystal panel 2A as it is, when not performing the process of S18 and the process of S21 to S26. Alternatively, the CPU 31 may appropriately cut out an image having a wider angle than the first angle of view from the second image on the basis of the first direction, and display the cut out image on the emission unit 22 of the liquid crystal panel 2A.

  The frame X2 may indicate a part corresponding to the first image in the display image by surrounding at least a part of the part corresponding to the first image in the display image. In the display image, information indicating a portion corresponding to the first image in the display image may have various modes in addition to the frame X2. In place of the frame X2, the portion corresponding to the first image in the display image is clearly displayed, and the other portion of the display image is displayed more clearly than the portion corresponding to the first image. Of these, a portion corresponding to the first image may be shown. The portion corresponding to the first image in the display image may be displayed in color, and the other portion of the display image may be displayed in black and white to indicate the portion corresponding to the first image in the display image.

  The processing from S22 to S26 may be performed regardless of the HD mounting position. For example, regardless of the HD mounting position, is θ + and θ− indicating the display range determined by the processing of S18 within a range of a predetermined angle indicating that it is within the range of the visual field of the user U? Depending on whether or not, the values of θ + and θ− may be corrected.

  In the above embodiment, both the first image and the second image are captured in real time by the first camera 51 and the second camera 52. For example, a wide-angle image captured in advance is used as the second image. Also good.

  The HMD 1 may photograph the user U's eyes with a camera (not shown). The HMD 1 may specify the eye of the user's eye as a reference point and specify the iris of the user's eye as a moving point from an image based on image data output from the camera. The FPGA 33 may specify the second direction from the iris position with respect to the specified position of the eye.

  The CPU 31 acquires the first image at this time instead of the processing of S52, and detects the position corresponding to the acquired first image from the display image by the pattern matching method or the like, thereby determining the position of the first image. The information shown may be modified.

  Instead of the emission part 22 of the liquid crystal panel 2A, a general-purpose display device such as a liquid crystal panel may be provided. This display device may be provided in the gimbal mechanism in which the first camera 51 is installed, or may be provided in the first camera 51. The display device may be provided at a position where the user U can visually recognize the hat worn by the user U, the eaves portion of the helmet, or the like. In other words, the “display device” of the present invention is not limited to the HD2 of the HMD 1 and may take various forms.

  In the above embodiment, the display range is calculated for the horizontal visual field extending in the horizontal direction around the left and right directions of the user U. However, the display range is similarly calculated for the vertical visual field extending in the vertical direction of the user U. May be.

  The second direction may be detected by the acceleration sensor 38. The HD mounting position may be detected by the acceleration sensor 41. In order to detect the first direction, the second direction, and the HD mounting position, in addition to the acceleration sensor, various known sensors and circuits such as a gyro sensor, an optical sensor, and various circuits may be used.

  The wearing tool 10 is not limited to the above configuration, and may be a configuration such as an attachment attached to a hat, a helmet, or the like, or a configuration such as a frame of glasses. Further, the HMD 1 may not have the wearing tool 10. The fixing portion 14 may be fixed to another member different from the wearing tool 10. For example, the fixing unit 14 may be fixed to a hat, a helmet, or the like worn by the user U. The acceleration sensor 41 that detects the second direction may be provided on the arm portion 12, the fixing portion 14, or the like of the mounting portion 11. The acceleration sensor 41 may be provided on a hat, clothes, or the like worn by the user U.

  The first distance L1 when HD2 is disposed at the first position may be longer than the second distance L2 when HD2 is disposed at the second position. The rotation angle of at least one of the first ball joint 6 and the second ball joint 7 when the HD 2 is moved from the first position to the second position may be smaller than the first predetermined angle. The rotation angle of at least one of the first ball joint 6 and the second ball joint 7 when the HD 2 is moved from the first position to the third position may be smaller than the second predetermined angle. The mounting portion 11 may have another adjustment mechanism, for example, a flexible joint formed of an elastic body such as rubber, instead of the first ball joint 6 and the second ball joint 7.

1 HMD
2 HD
2A Liquid crystal panel 22 Output part 10 Wearing tool 11 Mounting part 12 Arm part 14 Fixing part 22 Output part 31 CPU
38, 41 Acceleration sensor

Claims (6)

A display for displaying an image;
First image acquisition means for acquiring a first image captured by the first camera;
Second image acquisition means for acquiring a second image captured by a second camera that captures an image having a wider angle than the angle of view of the first camera;
First acquisition means for acquiring a first direction indicating a direction in which the first camera captures the first image;
Second acquisition means for acquiring a second direction indicating a direction in which the user's visual field is directed;
A relative angle calculating means for calculating a relative angle that is an angle formed by the first direction and the second direction;
A determining unit that determines, based on the relative angle calculated by the relative angle calculating unit, a display range that is a range to be displayed on the display unit in the second image as a range including the first image;
Generating means for generating a display image that is an image corresponding to the display range determined by the determining means;
And a display control unit that displays the display image generated by the generation unit on the display unit.   The display device according to claim 1, wherein the generation unit generates information by adding information indicating a portion corresponding to the first image in the display image to the display image. Third acquisition means for acquiring the first direction and the second direction at a time after the display image is displayed on the display unit by the display control means;
The first relative angle, which is the relative angle formed by the first direction acquired by the first acquisition unit and the second direction acquired by the second acquisition unit, is acquired by the third acquisition unit. A change amount calculating means for calculating a relative angle change amount that is a change amount between the second relative angle that is the relative angle formed by the first direction and the second direction;
Change amount determination means for determining whether the relative angle change amount calculated by the change amount calculation means is greater than or equal to a predetermined amount;
The display according to claim 1, wherein the generation unit newly generates the display image when the change amount determination unit determines that the relative angle change amount is equal to or greater than a predetermined amount. apparatus. A second direction determination means for determining whether the second direction acquired by the second acquisition means indicates a direction within a specific range set with respect to the visibility of the first direction of the user;
The display control unit causes the display unit to display the first image when the second direction determination unit determines that the second direction is within the specific range. Item 4. The display device according to any one of Items 1 to 3. The display device according to any one of claims 1 to 4,
A mounting portion that is a member for mounting the display device on the user's head;
A detection unit that is provided in the mounting unit and detects the second direction;
The second acquisition means acquires the second direction detected by the detection unit,
The head-mounted display, wherein the generation unit generates the display image including a visual field of the user based on the second direction. The mounting unit can mount the display unit of the display device by selecting a position with respect to either the right eye or the left eye of the user,
A mounting position determination unit that determines whether the display unit is mounted in a position corresponding to either the right eye or the left eye of the user;
6. The head mounted display according to claim 5, wherein the determining unit changes the display range in accordance with a determination result by the mounting position determining unit.

Images