JP2018067773A - Imaging device, control method thereof, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device that notifies a user of a direction of a prescribed subject detected in a video obtained by photographing the surround of the user in a wide range in a manner in which the user can intuitively recognize it.SOLUTION: An HMD 100 includes: an attachment section 101 attached to a user; a display unit 104 disposed at a position facing eyes of the user when the attachment section 101 is attached to the user; a whole circumference imaging unit 102 for taking an image of the periphery centering on the user; an angle measuring unit 103 measuring an angle of the attachment unit 101 to a horizontal direction; a video generation unit 105 for generating a panoramic video setting the angle measured by the angle measuring unit 103 from the video taken by the whole circumference imaging unit 102 as a center of a vertical direction; and a display control unit 106 maintains control to display the panoramic image generated by the video generation unit 105 on the display unit 104.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus that is worn by a user (user) and images the surroundings, a control method thereof, a program, and a storage medium.

  An imaging system that captures a wide viewing angle range and provides a user with information about the captured subject is known, and is used in the field of surveillance cameras, for example. In Patent Document 1, the imaging range frame of the zoom image is automatically changed according to the zoom position in the wide-field captured image captured by attaching a wide viewing angle lens (panoramic lens) to the upper surface of the digital camera. An imaging apparatus that can display the image has been proposed. Furthermore, Patent Document 2 describes a rear monitoring device in which an imaging unit is attached to a back of a horseshoe-shaped collar and the monitor and the imaging unit are electrically connected to each other.

JP 2013-251783 A JP 2004-180250 A

  In the technique described in Patent Document 1, the wide viewing angle lens is provided so as to protrude upward from the upper surface of the digital camera. For this reason, there is a problem in that the surrounding area of the user cannot be captured sufficiently because the rear imaging range is blocked by the user's head or the like during imaging. In addition, the technique described in Patent Document 2 has a problem that the user cannot be notified so that the user can intuitively recognize the direction of the subject.

  According to the present invention, the user can be imaged in a wide range, and the user can be notified so that the user can intuitively recognize the direction of the predetermined subject detected in the captured image. It is an object of the present invention to provide an imaging device that can be used.

  An imaging apparatus according to the present invention includes a mounting unit mounted on a user, a display unit disposed at a position facing the user's eye when the mounting unit is mounted on the user, An imaging unit that captures a predetermined range around the user, a measuring unit that measures an angle at which the mounting unit is inclined with respect to a horizontal direction, and a measurement unit that measures the image captured by the imaging unit. Generating means for generating a panoramic image with the angle set as the center in the vertical direction, and control means for controlling the panoramic image generated by the generating means to be displayed on the display unit. .

  According to the present invention, the user can be imaged in a wide range, and the user can be notified so that the user can intuitively recognize the direction of the subject detected in the captured image. it can.

It is a figure explaining schematic structure of HMD concerning a 1st embodiment of the present invention. It is a figure explaining the specific example of the usage condition of HMD. It is a flowchart of generation | occurrence | production and display control of a perimeter panoramic image in HMD. It is a figure explaining the perimeter imaging model in which the light ray which entered from real space passes through a fisheye lens, and is projected on an image sensor. It is a figure which shows typically the distorted circular image | video imaged with a fisheye lens, and the plane regular image | video produced | generated from a distorted circular image | video. It is a figure explaining schematic structure of the wearable camera which concerns on 2nd Embodiment of this invention. It is a figure explaining the usage condition of a wearable camera. It is a flowchart of the photography assistance process in the imaging system which consists of a wearable camera and a digital camera.

  Embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the following description, a spectacle-type head mounted display (hereinafter referred to as “HMD”) will be described as an example of the imaging apparatus according to the first embodiment of the present invention.

  FIG. 1A is a block diagram showing a schematic configuration of the HMD 100 according to the first embodiment of the present invention. FIG. 1B is a perspective view showing the appearance of the HMD 100. The HMD 100 includes a mounting unit 101, an all-round imaging unit 102, an angle measurement unit 103, a display unit 104, a video generation unit 105, a display control unit 106, an operation unit 107, a CPU 108, a memory 109, a power supply unit 110, a communication unit 111, and a notification. A control unit 112 and a vibration unit 113 are included. The all-round imaging unit 102 includes a first imaging unit 102a and a second imaging unit 102b. The vibration unit 113 includes a first vibrator 113a and a second vibrator 113b.

  The mounting unit 101 has a structure corresponding to a frame of glasses. By adopting the structure of glasses in the HMD 100, the user (user of the HMD 100) can easily attach and detach (attach / remove) to a predetermined position of the head, and obtain a stable wearing state without a sense of incongruity. it can. Each of the first imaging unit 102a and the second imaging unit 102b is an imaging unit having a fisheye lens and an imaging device, and in opposite directions on the left and right temple parts (vines) that are part of the mounting part 101 ( It is arranged to face the outside.

  The fisheye lens is a lens that can capture the entire sky (total field of view is 180 degrees) in an image circle (image circle) of a predetermined size. In the fisheye lens, where the full-field imaging is realized in the captured image is determined according to the position and size of the image circle with respect to the imaging element. The case where the image circle is within the short side of the image sensor is called an all-round fisheye, and the first imaging unit 102a and the second imaging unit 102b are such an all-round fisheye imaging unit. Therefore, in the HMD 100, the all-around imaging unit 102 can acquire an image of the entire periphery around the user wearing the HMD 100. Therefore, the all-round imaging unit 102 only needs to include at least the first imaging unit 102a and the second imaging unit 102b, but the number of imaging units constituting the all-around imaging unit 102 is limited to two. is not. For example, the omnidirectional imaging unit 102 may include an imaging unit that captures a front direction in which the user's face is facing, and may acquire a omnidirectional video in cooperation with other imaging units.

  The angle measurement unit 103 includes a three-axis gyro sensor or the like, and measures the angle (elevation angle, depression angle) of the HMD 100 with respect to the horizontal direction according to the inclination of the head of the user wearing the HMD 100 with respect to the horizontal direction. For example, the angle measurement unit 103 is arranged in the mounting unit 101 so as to detect that the tilt angle of the mounting unit 101 with respect to the horizontal direction is 0 ° in a state where the user's face is facing the horizontal direction.

  The display unit 104 is, for example, a transflective organic EL display or the like, and a lens unit provided on the mounting unit 101 (frame) so as to face the user's eye so that glass see-through display is possible. It is provided in the lower area. The user can see the actual scenery directly viewed and the graphics on the display unit 104 optically superimposed. Note that an imaging unit that captures the front direction of the HMD 100 is provided in the mounting unit 101 (frame), and the video acquired by the imaging unit and the graphics of the video acquired by another imaging unit are electrically superimposed and displayed. It is good also as a structure (video see-through display). In this case, the display unit 104 can be configured by a non-transmissive display such as a TFT liquid crystal display.

  The video generation unit 105 generates an all-round image from the images captured by the first imaging unit 102a and the second imaging unit 102b, and at the same time, is based on the angle information measured by the angle measurement unit 103. Is generated. A method for generating the all-round panoramic video will be described later. The display control unit 106 displays the display mode on the display unit 104, for example, an overlaid display of a real landscape directly viewed by the user and an all-around panoramic video generated by the video generation unit 105, a display position or display of a GUI (Graphical User Interface) Control timing, degree of overlay, etc. The operation unit 107 is a touch pad, a switch, or the like that receives a user instruction to the HMD 100 and is provided in the mounting unit 101. By operating the operation unit 107, it is possible to execute various processes such as setting of the operation mode of the HMD 100 and a photography process. The operation unit 107 includes an operation input function based on a captured gesture by imaging a user's finger movement (gesture) or the like using the first imaging unit 102a and the second imaging unit 102b.

  The CPU 108 performs overall control of the HMD 100 by executing the program stored in the memory 109 and controlling the operation of each unit constituting the HMD 100. The memory 109 has a volatile area and a nonvolatile area. The volatile area of the memory 109 has a work area where a program read from the non-volatile area is developed, and an area for temporarily storing video data captured by the all-round imaging unit 102. The non-volatile area of the memory 109 has an area for storing programs necessary for operation control of the HMD 100, constants and variables for system control, and an area for storing video data captured by the all-round imaging unit 102.

  The power supply unit 110 is a primary battery such as an alkaline battery or a lithium battery, or a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, and supplies necessary power to each part of the HMD 100. In addition, the power supply unit 110 includes a power switch that switches on / off the power according to a user operation or other conditions. In the HMD 100, when a secondary battery is used, the secondary battery can be charged by a non-contact charging (non-contact power transmission) method or an AC adapter connection. A sensor for detecting whether or not the HMD 100 is mounted on the user's head is provided in the mounting portion 101 (for example, a pad (nose pad) or a modern (front cell) portion) and is not mounted on the head. In some cases, the power supply may not be turned on.

  The communication unit 111 performs bidirectional communication with an external device according to control by the CPU 108. Examples of the external device include, but are not limited to, an information device such as a personal computer (PC) and a smartphone, and an electronic device such as a digital camera having a communication function. The communication method is not particularly limited, but it is desirable to use a wireless method in order to make the wearing feeling comfortable. Each of the first vibrator 113a and the second vibrator 113b included in the vibration unit 113 includes an actuator, a motor, a speaker, and the like, and the mounting unit 101 (so that a specific direction can be notified to the user by vibration or voice. The left and right temple portions are arranged. The notification control unit 112 controls the operation of the vibration unit 113.

  FIG. 2 is a diagram for explaining a specific example of the usage mode of the HMD 100. FIG. 2A is a diagram illustrating an aspect in which the face of the user wearing the HMD 100 faces in the horizontal direction. As described above, the angle measurement unit 103 is adjusted to output an output indicating that the elevation angle and the depression angle are 0 degrees in the posture of the HMD 100 when the user's face is in the horizontal direction. FIG. 2B is a diagram illustrating a display mode of an image displayed on the display unit 104 in the state of FIG. On the display unit 104, a panoramic image of the entire circumference of the user in the horizontal direction is displayed in a lower area of the actual landscape so as to be superimposed on the actual landscape in front of the user.

  FIG. 2C is a diagram illustrating a mode in which the user wearing the HMD 100 changes his face upward. FIG. 2D is a diagram illustrating a display mode of an image displayed on the display unit 104 in the state of FIG. On the display unit 104, a panoramic image of the entire circumference above the user is displayed in a lower area of the actual landscape so as to be superimposed on the actual landscape above the user. FIG. 2E is a diagram illustrating a mode in which the user wearing the HMD 100 changes his face downward. FIG. 2F is a diagram illustrating a display mode of an image displayed on the display unit 104 in the state of FIG. On the display unit 104, a panoramic image of the entire circumference of the lower part of the user is displayed in a region below the actual scenery so as to be superimposed on the actual scenery of the lower part of the user.

  A user wearing the HMD 100 can also view an all-around panoramic image with the elevation angle or depression angle as the center in the vertical direction (vertical direction) while visually recognizing the actual landscape in the direction in which the user's face is facing. At this time, the all-round panoramic image is displayed on the display unit 104 so that the right and left center of the all-round panoramic image substantially coincides with the direction in which the user's face is facing. Therefore, the user can easily grasp in which direction the subject in the all-around panoramic image is on the right side or the left side with respect to the direction in which the user's face is facing.

  For example, when a person is walking, the elevation angle and the depression angle are often oriented in a direction of 0 degrees (substantially horizontal). At that time, automobiles, motorcycles, other pedestrians and the like who should pay attention to safety generally approach from a direction where the elevation angle and depression angle are 0 degrees (substantially horizontal). If the user wears the HMD 100 in such a situation, even if the car or the like approaches the user from a direction other than the front (rear, right side, left side), the approaching car or the like is displayed in the panoramic video. Therefore, the user can pay attention to safety from an early stage. At this time, in the HMD 100, since the first vibrator 113a and the second vibrator 113b are provided in each of the left and right temple parts, different vibrations (acoustics) are generated on the left and right sides and transmitted to the user. Direction information can be notified to the user. For example, the user can be notified of the direction in which a specific subject (for example, a car) approaches by vibration or voice.

  Further, for example, when the user is observing a bird that has stopped at the front tree with the face facing upward at a predetermined angle, the bird flies away. Therefore, it is assumed that the user changes his face and follows the bird with his eyes, but has lost his sight from the forward visual field. At this time, the display unit 104 of the HMD 100 displays an all-around panoramic image that captures a landscape that is approximately the same angle as the elevation angle with the user as the center, and a bird that flies in the displayed all-around panorama image. The probability of being displayed is high. Therefore, the user can capture the bird chased by his eyes in the real scenery ahead by changing the direction of the head based on the panoramic video of the entire circumference.

  In this way, by wearing the HMD 100, the user can easily know in which direction the subject around him / her is, so that the behavior to pay attention to safety and the presence of the subject to be actually viewed are present. It is possible to take an action of turning the face in the direction to be done without waste.

  Next, the elevation angle or depression angle in the direction in which the face of the user wearing the HMD 100 is facing is detected, and an all-round panoramic image having the detected elevation angle or depression angle as the center in the vertical direction is generated and displayed on the display unit 104. A control flow will be described. FIG. 3 is a flowchart of the generation of the all-round panoramic video in the HMD 100 and the display control on the display unit 104 of the generated all-round panoramic video. 3, the CPU 108 expands the program stored in the nonvolatile area of the memory 109 to the volatile area (work area) in the memory 109 and controls the operation of each unit constituting the HMD 100. It is realized by.

  When the power switch of the HMD 100 is turned on, processing in the HMD 100 is started. In step S301, the CPU 108 controls the all-round imaging unit 102 to start imaging around the user. In step S <b> 302, the CPU 108 controls the angle measurement unit 103 to measure the depression angle and elevation angle (inclination angle with respect to the horizontal direction of the HMD 100) in the direction in which the user's face is facing. In step S303, the CPU 108 controls the video generation unit 105 to set the angle θ in the vertical direction based on the video imaged by the all-round imaging unit 102 in step S301 and the elevation angle or depression angle θ measured in step S302. A panoramic video with the entire center is generated. A method for generating the all-round panoramic video will be described later.

  In step S <b> 304, the CPU 108 controls the display control unit 106 to display the all-around panoramic image generated in step S <b> 303 on the display unit 104. In step S305, the CPU 108 determines whether or not a predetermined end condition by the user is satisfied. The predetermined end condition includes, for example, an operation of turning off a power switch. If the CPU 108 determines that the end condition is satisfied (YES in S305), the CPU 108 ends the process. If the CPU 108 determines that the end condition is not satisfied (NO in S305), the process proceeds to step S306. In step S306, the CPU 108 determines whether the depression angle and elevation angle in the direction in which the user's face is facing have changed. If CPU 108 determines that the depression angle and elevation angle have changed (YES in S306), the process returns to step S302. If CPU 108 determines that the depression angle and elevation angle have not changed (NO in S306), the process returns to step S304.

  Next, a process of generating an all-round panoramic image having a predetermined angle θ (elevation angle or depression angle) as a vertical center in step S303 will be described. FIG. 4 is a diagram for explaining an imaging model in each of the first imaging unit 102a and the second imaging unit 102b. All rays incident from the real space pass through the fisheye lens and are projected onto the two-dimensional imaging element. It is a figure explaining a circumference imaging model. FIG. 5A is a diagram schematically showing a distorted circular image captured by a fisheye lens. FIG. 5B is a diagram schematically showing a planar regular image generated from the distorted circular image in FIG.

  Here, the imaging surface of the imaging device is the XY plane, and the central axis of the fisheye lens is the Z axis. In addition, a light ray that has entered the fisheye lens from real space is parallel to the Z axis on the spherical surface H, and is projected onto the imaging surface (XY plane) of the imaging device. Therefore, an image in which the entire circumference is imaged by the fisheye lens and imaged on the XY plane is a distorted circular image as shown in FIG. A vector n shown in FIG. 4 is a line-of-sight vector with an elevation angle (or depression angle) of θ degrees, and is a planar regular image that is imaged in the range of azimuth angle φ = 0 degrees to 180 degrees on a UV plane orthogonal to the vector n. T is a panoramic image in which the elevation angle or the depression angle of the image of which the entire circumference is imaged by the fisheye lens is θ degrees. This image is a planar regular image (image without distortion) as shown in FIG.

  The process of generating a panoramic image from an image captured by the omnidirectional imaging unit 102 having a fisheye lens is a planar regular image as shown in FIG. 5B from a distorted circular image as shown in FIG. Is a process for generating That is, the pixel value of the pixel located at the point Ti (ui, vi) on the UV plane is changed from the pixel value located at the point Si (xi, yi) on the XY plane (a distorted circular image S on the two-dimensional XY orthogonal coordinate system). The pixel value is determined on the basis of the pixel value of the inner pixel). For that purpose, a correspondence relation between a point (ui, vi) on the two-dimensional UV orthogonal coordinate system defining the UV plane and a point (xi, yi) on the two-dimensional XY orthogonal coordinate system is necessary. Then, using the corresponding relational expression, an arbitrary point on the planar regular image T is associated with any point of the distorted circular image S, and the distorted circular image can be converted into a planar regular image. it can.

  Corresponding relational expressions between these two points are expressed by the following expressions 1 and 2. However, in the following formulas 1 and 2, w = m × R, R is the radius of the distorted circular image S, and m is the relationship between the number of pixels of coordinate values u and v and the number of pixels of coordinate values x and y. The number of pixels of the planar regular image T increases as m increases. Note that, as will be described later, the planar regular image T by one imaging unit is displayed on the display unit 104 by being joined with the image by the other imaging unit. Therefore, the value of m is finally determined by the number of pixels displayed on the display unit 104.

  First, the image generation unit 105 generates a planar image from the images captured by the first imaging unit 102a and the second imaging unit 102b. Subsequently, the video generation unit 105 connects the generated planar images in the horizontal direction (the U-axis direction of each planar image) to generate an all-round panoramic image around the user with the user at the center. For the joining of images, for example, a known technique such as a pattern matching technique for comparing pixels in the vicinity of the joining position of each image can be used. At that time, the panoramic image of the entire circumference needs to be substantially coincident with the direction in which the user's face faces when the left and right centers are displayed on the display unit 104. The first imaging unit 102a and the second imaging unit 102b are arranged so as to sandwich the user's head from the left and right when the HMD 100 is mounted on the user. In this case, the respective planar images are connected such that the planar image by the left first imaging unit 102a is on the left side and the planar image by the right second imaging unit 102b is on the right side.

  In addition, when an overlap area is generated when connecting each plane image that can obtain a plurality of imaging units including a fisheye lens, a joint is formed on the inner side of the left and right ends of each corresponding plane image according to the arrangement of each imaging unit. It may be set. In addition, unlike the above-described HMD 100, an HMD in which an imaging unit is arranged so as to capture the direction in which the user's face is facing (front of the user) and the opposite direction (back of the user) is assumed. In this case, the plane image generated from the imaging unit that captures the rear of the user is divided into left and right parts, the left part thereof being the right end part of the entire panoramic video, and the right part thereof being the left end of the entire panoramic picture. Connect to each part.

  As described above, the HMD 100 in which the entire periphery imaging unit 102 is provided on the mounting unit 101 mounted on the user's head is easy to attach and detach, and can be stably mounted. In addition, since the user can visually recognize the entire panoramic image generated by imaging the entire periphery of the user, the user can intuitively recognize the direction of the surrounding subject. Further, in the HMD 100, an all-round panoramic image having the elevation angle or depression angle in the direction facing the user as the center in the vertical direction is generated from the image captured by the all-round imaging unit 102, and the generated all-round panoramic image is the face of the user. It is displayed superimposed on the actual landscape in the direction you are facing. Thereby, even when the user has a subject in a direction other than the direction in which his / her face is directed, the user can quickly change the direction of the face without hesitation and quickly capture the subject in the field of view. Furthermore, the left and right center of the entire panoramic image is displayed so as to substantially match the direction in which the user's face is facing, so that the user can view the panoramic image of the entire periphery in the direction in which the user's face is facing. It is possible to easily grasp which direction of the head.

  The HMD 100 can also identify a subject by a user operation and display an index indicating at least one of the position and direction of the identified subject superimposed on the entire panoramic video displayed on the display unit 104. It is. To identify the subject, a technique called “moving object tracking” that is used in an imaging apparatus such as a digital camera and keeps the imaging setting condition for the moving subject can be used. In that case, feature information such as color difference information of the subject specified by the user's operation is stored in the memory 109, a region having the feature information is detected from the captured image, and a square frame is detected in the detected region. Or the like may be displayed as a GUI. As a result, the user can more easily recognize the position and direction of a specific subject.

  When the subject is specified, on / off of the display of the all-around panoramic video on the display unit 104 may be switched between when the specified subject is in the actual landscape and when it is not. . That is, when the specified subject is in the actual landscape, the user does not need to confirm the subject with the all-around panorama image, and therefore the display of the all-around panorama image is controlled to be off. On the other hand, the case where the user wants to confirm the panoramic video image is the case where the subject is not in the actual landscape. In this case, the display of the panoramic video image is controlled to be on. Whether or not the specified subject is in the real scene can be detected from the panoramic video of the entire circumference. That is, the direction in which the user's face is facing is specified in the entire range of the entire panoramic video. Therefore, if the subject is specified within a certain range in the direction in which it is determined that direct viewing by the user is possible in the all-round panoramic video, it can be determined that the specified subject is in the actual landscape. When the HMD 100 includes another imaging unit that can capture a real scene in front of the user, it is also possible to determine whether or not there is a specified subject in the image captured by the imaging unit. It is possible to determine whether or not the specified subject is in the actual landscape.

  When the HMD 100 includes another imaging unit that can capture a real landscape in front of the user, the panoramic video is displayed on the display unit 104 when the subject in the front real landscape is to be captured as a photograph. If this is done, the confirmation of the correct composition may be hindered. Therefore, when the HMD 100 is set to the mode for capturing a real landscape in front, it is desirable that the display of the panoramic image of the entire circumference is controlled to be off. As described above, the display of the all-round panoramic image is controlled to be turned off according to the use situation, so that an effect of reducing power consumption can be obtained.

  Next, a second embodiment of the present invention will be described. Here, as an example of the imaging apparatus according to the second embodiment, a wearable camera used on the neck is taken up. FIG. 6 is an external perspective view of wearable camera 600. The wearable camera 600 includes an arc-shaped or substantially U-shaped neck hanging portion 601 in which a part of a ring is cut off, a first imaging unit 602a, a second imaging unit 602b, and an outer peripheral surface of the neck hanging portion 601. It has an all-round imaging unit including a third imaging unit 602c. The wearable camera 600 includes an angle measurement unit 103, an image generation unit 105, a display control unit 106, an operation unit 107, a CPU 108, a memory 109, a power supply unit 110, a communication unit 111, a notification control unit 112, and a vibration unit 113. Each part is equivalent to These parts are mounted in the neck portion 601 and are not shown in FIG.

  The neck hanging portion 601 is a holding member that holds the entire circumference imaging portion, the display control portion 106, and the vibration portion 113. For example, the neck hanging portion 601 is made of an elastic resin or the like and is worn when the user hangs on the neck (see FIG. 7), and is configured to obtain a good wearing feeling. The first imaging unit 602a to the third imaging unit 602c are all-round fisheye imaging units each including a fisheye lens and an imaging element. The angle measurement unit 103 can detect the horizontal direction, and the video generation unit 105 can detect all of the surroundings of the user in the same manner as the HMD 100 according to the first embodiment, using the horizontal direction detected by the angle measurement unit 103 as a reference. A circumferential panoramic image can be generated. Note that the number of imaging units included in the all-round imaging unit is not limited to three.

  FIG. 7A shows an example of how the wearable camera 600 is used. Wearable camera 600 is worn by the user alone. The wearable camera 600 is mounted so that the neck portion 601 is hung on the user's neck, so that the wearable camera 600 can be easily attached and detached and can be stably mounted. The wearable camera 600 can image the entire horizontal direction around the user at a position close to the user's viewpoint, as in the case of being worn on the head like the HMD 100. Further, similarly to the HMD 100, the vibrator (including a speaker) arranged around the user's neck can be used to notify the user of direction information by transmitting different vibrations and sounds on the left and right. .

  While being worn by the user, the wearable camera 600 captures the entire circumference in the horizontal direction centered on the user and stores the captured image in a memory. In addition, a contact sensor (human sensor) or the like is provided on the inside of the neck portion 601 and the like is captured and the video is stored only while the contact sensor detects that the wearable camera 600 is worn by the user. You may make it do. As a result, unnecessary memory is not stored, so that the memory capacity used and power consumption can be saved. Alternatively, the video storage may be terminated after a predetermined time has elapsed since the contact sensor no longer detects that the wearable camera 600 is worn by the user. Thereby, when it becomes a non-wearing state contrary to a user's will by the cause of an accident etc., the condition at that time can be memorize | stored.

  When the CPU of the wearable camera 600 detects an object (person, thing) that meets a predetermined condition in a video obtained by imaging the entire periphery of the user, the CPU notifies the user of the direction by vibration or voice. An example of the predetermined condition is “when there is an object approaching the user”. For example, a situation where another person 701 is approaching as shown in FIG. 7A applies to this condition. Such a predetermined condition (object) can be set by a user operation and can be stored in a memory. Wearable camera 600 itself does not include a display. Therefore, the setting of the predetermined condition is performed in a state where the display is connected to the wearable camera 600. The wearable camera 600 and the display can be connected in the same manner as the connection between the wearable camera 600 and the digital camera 711 described later. In addition, it is good also as a structure which can provide a small display in the neck part 601 and can set various conditions with the display.

  The CPU analyzes the video imaged by the all-round imaging unit over several frames, and the area where the size and speed of change of the area changing in the image between the frames exceeds a predetermined threshold are changing areas. The direction corresponding to is notified to the user by vibration or sound. At that time, the magnitude of vibration or sound, the number of vibrations, the vibration pattern, and the like may be varied depending on the size and speed of the changing area.

  Another example of the predetermined condition is “when a specific object appears”. In this case, feature data of “specific object” is stored in advance in a nonvolatile area of the memory. The CPU compares the feature data of the new object with the feature data of the specific object when a new object appears in the video that captures the entire periphery of the user, and matches at a predetermined threshold (ratio) or more. It is determined that “a specific object has appeared”. The CPU notifies the user of the direction of the new object by vibration or sound. Note that feature data of a specific object may be stored in an external server or the like via a communication unit. Further, the feature data may be updated as needed by a neural network or other so-called “machine learning” methods.

  FIG. 7B is a diagram showing another example of usage of the wearable camera 600, and corresponds to FIG. 6B. A cord 710 extends from both ends of the neck portion 601 and is connected to a digital camera 711 which is another imaging device. The cord 710 can be fed from the neck hook portion 601 and stored in the neck hook portion 601 by a cord feeding / winding mechanism (not shown) provided in the neck hook portion 601. The cord feeding / winding mechanism includes a lock mechanism that locks the cord 710 when the cord 710 is pulled out, a lock release mechanism that unlocks the cord 710, and a biasing mechanism that winds the cord 710. . The wearable camera 600 can be used by switching between the usage mode shown in FIG. 7A and the usage mode shown in FIG. 7B.

  For example, the wearable camera 600 and the digital camera 711 can perform two-way communication by wireless communication, and a display or an electronic viewfinder included in the digital camera 711 can be used as a display unit of the wearable camera 600. Further, the digital camera 711 can measure the elevation angle or depression angle that the photographing optical axis is in the horizontal direction, and the wearable camera 600 can generate an all-round panoramic image with the measured elevation angle or depression angle as the center in the vertical direction. . The generated all-round panoramic image can be displayed on the display or electronic viewfinder provided in the digital camera 711 so as to be superimposed on the actual scenery, and the display mode at that time is the same as in FIG. Can do. Therefore, for example, when the subject to be imaged is out of the imaging field angle of the digital camera 711, the direction of the object is recognized from the all-around panoramic image, and the image is again placed within the angle of view (photographing). It is possible to easily perform the operation.

  Note that a connector portion that connects other electronic devices including the digital camera 711 and the wearable camera 600 to be communicable may be provided at the tip of the cord 710. In that case, since the connection for mutual communication is established at the same time that the digital camera 711 is connected to the wearable camera 600, so-called pairing procedures such as SSID and password input required when using wireless communication may be omitted. it can.

  FIG. 8 is a flowchart of imaging assistance processing by the imaging system including the wearable camera 600 and the digital camera 711. The flowchart in FIG. 8 includes processing in the digital camera 711 and processing in the wearable camera 600. The various processes in the wearable camera 600 are performed by the CPU of the wearable camera 600 expanding the program stored in the non-volatile area in the memory to the volatile area in the memory and controlling the operation of each part of the wearable camera 600. Realized. Similarly, in the various processes in the digital camera 711, the CPU of the digital camera 711 expands the program stored in the non-volatile area in the memory to the volatile area in the memory, and the operation of each part constituting the digital camera 711 This is realized by controlling Note that it is assumed that the user is viewing the subject through an electronic viewfinder provided in the digital camera 711 when shooting with the digital camera 711.

  The processing is started when the wearable camera 600 is turned on or when the contact sensor detects that the wearable camera 600 is attached to the user, or when the digital camera 711 is turned on. The In the following description, the CPU of wearable camera 600 is referred to as “CPU (W)”, and the CPU of digital camera 711 is referred to as “CPU (D)”.

  In step S <b> 801, the CPU (D) starts imaging (acquisition of a subject image) with the digital camera 711. In step S <b> 802, the CPU (D) displays the image being acquired on the electronic viewfinder of the digital camera 711. In step S <b> 803, the CPU (D) determines whether or not the subject has been specified by the user's operation on the digital camera 711 or the like. The CPU (D) repeats the determination in step S803 until the subject is identified (NO in S803). If it is determined that the subject is identified (YES in S803), the process proceeds to step S804. In step S804, the CPU (D) stores the characteristic data of the specified subject in the memory and transmits the characteristic data to the wearable camera 600. The CPU (W) stores the feature data received from the digital camera 711 in the memory.

  In step S <b> 805, the CPU (W) starts imaging by the all-around imaging unit. In step S806, the CPU (W) detects the position in the captured image of the all-round imaging unit of the subject that matches the feature data based on the feature data acquired from the digital camera 711 and stored in the memory. Here, the elevation angle and depression angle of the digital camera 711 are continuously detected by an attitude detection sensor such as a three-axis gyro sensor provided in the digital camera 711. Therefore, in step S807, the CPU (W) acquires angle information (elevation angle and depression angle θ) of the digital camera 711 from the digital camera 711. In step S808, the CPU (W) controls the video generation unit to generate an all-round panoramic video from the video whose imaging has started in step S805 so that the angle acquired in step S807 is the center in the vertical direction. In the following description, it is assumed that the generated all-round panoramic image is transmitted to the digital camera 711.

  In step S <b> 809, the CPU (D) determines whether or not the previously specified subject is in the video (imaging field angle) of the digital camera 711. If the CPU (D) determines that the specified subject is not in the video of the digital camera 711 (NO in S809), the process proceeds to step S810. On the other hand, if the CPU (D) determines that the specified subject is in the video of the digital camera 711 (YES in S809), the process proceeds to step S811.

  In step S810, the CPU (D) displays the all-round panoramic image acquired from the wearable camera 600 on the electronic viewfinder. On the other hand, in step S810, the CPU (D) deletes the all-round panoramic image acquired from the wearable camera 600 from the electronic viewfinder (the entire panoramic image is not displayed in the electronic viewfinder). This is to facilitate composition confirmation when the user attempts to release (photograph) a specific subject. In other words, when the subject is in a real scene, it is not necessary to check the subject with the panoramic video, and the fact that the panoramic video is displayed on the electronic viewfinder may interfere with composition confirmation. There is. Therefore, if there is a subject in the actual landscape, the panoramic image is not displayed, and the panoramic video is displayed to enable the subject to be tracked efficiently when the subject is out of the shooting angle of view. Display on the electronic viewfinder.

  In step S812 after steps S810 and S811, the CPU (D) determines whether or not a predetermined end condition by the user is satisfied. Examples of the predetermined end condition include an operation of turning off the power of the digital camera 711. If the CPU (D) determines that the end condition is satisfied (YES in S812), the CPU (D) ends the process. If the CPU (D) determines that the end condition is not satisfied (NO in S812), the process returns to step S806. . Note that the CPU (W) can be configured to end the generation of the all-round panoramic video in response to the CPU (D) ending this processing.

  In addition, you may perform the process of said step S808-S811 as follows. In other words, in the above description, it is assumed that the entire panoramic video generated by the wearable camera 600 is transmitted to the digital camera 711. On the other hand, the wearable camera is configured so that the CPU (W) transmits the all-round panoramic image to the digital camera 711 generated in step S808 only when the transmission request for the all-round panoramic image is received from the digital camera 711. 600 is controlled. When the CPU (W) receives a request to stop transmission of all-round panoramic video from the digital camera 711, the CPU (W) controls the wearable camera 600 to stop transmission of all-round panoramic video to the digital camera 711. If the determination in step S809 is NO, the CPU (D) transmits a transmission request for the all-around panoramic video to the wearable camera 600. The CPU (W) that has received this transmission request transmits the panoramic image of the entire circumference to the digital camera 711, whereby the processing in step S810 can be executed. On the other hand, if the determination in step S809 is YES, the CPU (D) transmits to the wearable camera 600 a transmission stop request for the all-around panoramic video. The CPU (W) that has received this transmission stop request stops the transmission of the all-round panoramic video to the digital camera 711, whereby the process of step S811 can be executed.

  As described above, wearable camera 600 generates an all-round panoramic image around the user centered on the user, detects a subject that meets a predetermined condition from the all-round panoramic image, and detects the detected subject. The direction can be notified to the user by vibration or voice. As a result, the user can intuitively recognize the subject recognized by the wearable camera 600. Further, the wearable camera 600 is combined with an imaging device such as a digital camera, and the panoramic image generated by the wearable camera 600 is displayed on the display unit of the imaging device as needed, thereby assisting the imaging operation in the imaging device. be able to. At that time, the wearable camera 600 can be used as a neck strap of an imaging device or the like.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. For example, in the flowchart of FIG. 3, the operation main component of each process has been described as the CPU 108, but each process may be performed by one hardware according to the hardware configuration of the HMD 100, or a plurality of hardware may share the process. You may do it. The same applies to the processing of the flowchart of FIG. Moreover, although the example which applied this invention to HMD was demonstrated in the said description, this invention is not limited to this, The application to the use for which imaging of a wide viewing angle is desired is possible. Surveillance cameras and vehicle-mounted cameras are examples of applications used in the form of displaying the entire panoramic video generated like the HMD 100 on a display device.

  The configuration of the wearable camera 600 that does not display an image can be applied to a neckband type portable music player, a headband, or headphones having a neckband. In this case, the shape of the holding member corresponding to the neck hanging portion 601 can be a U-shape like the neck hanging portion 601. The configuration of the wearable camera 600 can also be applied to decorative items such as headbands. In this case, it is desirable to apply design and decoration to the holding member corresponding to the neck portion 601. Furthermore, the configuration of the wearable camera 600 can be applied to various helmets for motorcycles, bicycles, construction sites, etc. In this case, the shape of the holding member corresponding to the neck portion 601 is a dome shape (hemisphere). Body shape).

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 HMD (head mounted display)
DESCRIPTION OF SYMBOLS 101 Mounting part 102 Whole circumference imaging part 103 Angle measurement part 104 Display part 105 Image | video production | generation part 106 Display control part 108 CPU
113 Vibration part

Claims (17)

A mounting part to be worn by the user;
A display unit disposed at a position facing the user's eye when the mounting unit is mounted on the user;
An imaging unit that images a predetermined range around the user;
Measuring means for measuring an angle at which the mounting portion is inclined with respect to a horizontal direction;
Generating means for generating a panoramic image with the angle measured by the measuring means as the center in the vertical direction from the image captured by the imaging unit;
An imaging apparatus comprising: control means for controlling the panoramic video generated by the generating means to be displayed on the display unit.   The imaging apparatus according to claim 1, wherein the imaging unit includes a plurality of all-round fisheye imaging units, and acquires images of the entire circumference of the user by the plurality of imaging units.   The generation unit generates an all-around panoramic image from the entire surrounding image acquired by the imaging unit, and generates an all-round panoramic image as the panoramic image from the generated all-round image. 2. The imaging device according to 2.   The said control means is displayed on the said display part so that the left-right center of the said panoramic image may correspond with the direction to which the said user's face is facing substantially, The display part of Claim 1 characterized by the above-mentioned. The imaging device described. Further comprising a specifying means for specifying a predetermined subject;
The control unit superimposes an index indicating at least one of a position and a direction of the identified subject on the panoramic image displayed on the display unit when the subject identified by the identifying unit is reflected in the panoramic image. The image pickup apparatus according to claim 1, wherein the image pickup apparatus displays the image.   When the control means determines that the specified subject is within a certain range in a direction in which it is determined that direct viewing by the user is possible among the entire range of the panoramic video, the display unit of the panoramic video 6. The imaging apparatus according to claim 5, wherein the display of the image is terminated and the panoramic image is displayed on the display unit when it is determined that the specified subject does not exist within the certain range.   7. The apparatus according to claim 5, further comprising at least one of a vibration unit that is provided in the mounting portion and vibrates a direction in which the specific subject exists, or an acoustic unit that emits a sound that conveys the direction to the user. The imaging device described. An operation means for receiving the user's instruction;
The imaging apparatus according to claim 1, wherein the control unit hides the panoramic video on the display unit in accordance with an instruction input through the operation unit. The mounting portion has a structure corresponding to a frame of glasses,
The imaging apparatus according to claim 1, wherein the display unit is provided in a lower region of the lens unit provided in the mounting unit. A holding member to be worn by the user;
A detecting means provided on the holding member for detecting a horizontal direction;
An image pickup apparatus comprising: an image pickup unit provided on the holding member and picking up an image of the entire horizontal direction around the holding member. Detecting means for detecting a predetermined subject from the video imaged by the imaging unit;
It is further provided with at least one of vibration means provided on the holding member for vibrating the direction in which the subject detected by the detection means is present, or acoustic means for emitting a sound for transmitting the direction to the user. The imaging device according to claim 10.   The holding member has a substantially U shape that can be attached to the head or neck of the user, or a dome shape that can be attached to the head of the user. The imaging device according to 10 or 11.   The imaging device according to any one of claims 10 to 12, wherein the holding member includes a cord for connecting another device and means for feeding and winding the cord. .   The image pickup apparatus according to claim 13, wherein the cord includes a connector unit that connects the image pickup apparatus and the another device so as to communicate with each other. A method for controlling an image pickup apparatus having a mounting portion mounted on a user,
Capturing an image of a predetermined range around the user by the imaging unit in a state where the mounting unit is mounted on the user;
Measuring the angle at which the mounting portion is inclined with respect to the horizontal direction;
Generating a panoramic image having the measured angle as a center in the vertical direction from the image captured by the imaging unit;
And a step of displaying the panoramic image on a display unit arranged at a position facing the user's eye.   The program for functioning a computer as each means of the imaging device described in any one of Claims 1 thru | or 14. A computer-readable storage medium storing a program for causing a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 14.